Technical Report No. VSR-04.02
End-of-Year Technical Report
For Project
Digital Human Modeling and Virtual Reality for FCS
By
The Virtual Soldier Research (VSR) Program
Center for Computer-Aided Design
College of Engineering
The University of Iowa
116 Engineering Research Facility
Iowa City, IA 52242-1000
VSR TEAM
K. Abdel-Malek, J. Arora, S. Beck, M. Bhatti, J. Carroll (Clarkson University), T. Cook, S. Dasgupta, N. Grosland, R. Han, H. Kim, J. Lu, C. Swan, A. Williams, J. Yang
K. Farrell, R. Vignes, T. Sinokrot, A. Mathai, T. Marler, J. Muhs, Q. Wang, X. Zhou, J. Lee, J. Kim, X. Man, S. Rahmatala, S. Dandach, R. Fetter, E. Horn, A. Patrick, Z. Mi,
Dated: October 25, 2004
CONTRACT/PR NO. DAAE07-03-D-L003/0001
TABLE OF CONTENTS
13.4.1. Vicon Motion Capture Components
13.4.3. Motion Capture Process
13.4.4. Labeling and Calibration
13.5. Motion Capture of Multiple Subjects
13.6. Research at VSR and Major Challenges
13.6.1. Planned Usage of the Motion Capture System
13.6.1.1. Santos Mimicking a User
13.6.1.2. Validating Posture and Motion Prediction
13.6.1.3. Interacting with Santos in Real Time
13.6.1.4. Motion Capture of Cloth
13.1. Abstract
Motion tracking of moving subjects in specified spaces is now a well-defined procedure and has been implemented in different fields such as gait analysis, ergonomics, anthropometry, robotics simulation, gaming and entertainment industries, motion predictions, and animation of virtual actors in virtual environments in real time. Research in this area is considerably rich and many institutions and companies are now using this technology due to its accuracy, consistency, and robustness.
This report presents the process of utilizing Vicon eight cameras optical motion capture system as one of the essential tools in verifying the analysis and algorithms in Virtual Soldier Research (VSR) project at the University of Iowa. This includes the basic steps in using this system, the current work, and the future work.
13.2. Introduction
Motion capture systems which involves the process of recording a live motion event and translating it into usable mathematical forms are now being widely used in many applications such as medicine (1-5), sports (6, 7), entertainment industry, and in the study of human factors (8). There has been tremendous work in these areas toward achieving realistic data regarding human motion, stability, and the way human interfere with their environment. For example in the motor vehicle industry, motion capture systems have found considerable attentions in helping the designers in modifying their design to accommodate with the users requirements and comfort (9). In addition to these applications, motion capture can play a considerable role as an experimental reliable source toward verifying the analysis models for many applications. There are now many techniques available for motion capturing such as electromagnetic trackers which are being used by a number of institutions; however, our focus in this report is on optical motion capture systems.
Optical motion capture is a very accurate measuring tool in tracking the motion of moving subjects (10, 11). Today, such process can be even conducted in real time and the resulting information can be shared through the networks to different remote locations in real time. In addition, the captured data can flow in real time into different animation engines such as Maya and Kaydara. The optical motion capture has some advantages and disadvantages when compared with electromagnetic trackers (10). The advantages include extremely accurate data acquisition, flexibility; for example, it is possible to use a large number of markers on the subject, or to change the configuration of the markers, and it is possible to use a group of markers to find approximate location of the joint axis. In addition the actor can perform without the hassle of a set of cables attached to his/her body. Moreover, utilizing higher frequency cameras can increase the number of frame captured per second. The main disadvantages of the optical motion capture are occlusion (where a marker is not being seen by at least two cameras) and the cross over (where the system confuses one marker for another) between adjacent markers and the necessity of having a capture space that has minimal noise resulting from reflected and shiny spots.
In this report, we present the complete history of our Vicon motion capture system from the time we bought the system until the time of this report. At the beginning of this report we will give some background about the basics operating steps of our motion capture systems and its capabilities. Then we will introduce what have been accomplished in this area, the current work, and the plans for near future work
13.3. Overview
13.3.1. Setup at VSR
During this period, extensive research was conducted to investigate various motion capture systems available in the market. Upon completion of this study, an 8-camera VICON 8i system was purchased. Shipment of this system was received at University of Iowa in late-March 2004, and training occurred April 6-8th. Four VSR team members were trained in setting up the system, and usage of both the VICON Workstation Software and IQ software. These team members begin to gain experience and proficiency with the intricacies of motion capture in general, and the VICON system in particular. Over the coming months the motion capture system is expected to become a vital tool in this VSR effort.
During the early stage of training, Vicon motion capture instruments were installed at the engineering building, the University of Iowa. At that time the cameras were erected around the motion capture space using eight Vicon tripods. After the training period, the equipments were moved to its permanent space at the Engineering Research Facility building, Center for Computer Aided-Design (CCAD), the University of Iowa. At this permanent space the cameras are mounted on a structure attached to the roof of the laboratory.
13.3.2. Background
Motion Capture is the process of recording a live motion event and translating it into usable mathematical terms by tracking a number of key points in space over time and combining them to obtain a single 3D representation of the performance [10]. This process could be the motion of a single subject with limited degrees of freedom, a motion of skeleton structure with many degrees of freedom, or a complex motion of a collection of a number of subjects. Human motion capture, which includes tracking the movement of the head, torso, and limbs, is a very realistic example of a skeleton structure with many joints and degrees of freedom. While it is common to use motion capture system to track the motion of large skeleton structures such as the human body, these systems are also capable of tracking the motion of small skeleton structures such as hand gesture and facial expressions. Motion capture systems have now many applications including anthropometry, ergonomics, gait analysis, motion of flexible geometry, etc.
Human motion analysis is receiving increasing attention from computer vision researchers. This interest is motivated by a wide spectrum of applications, such as athletic performance analysis, patients gait analysis, surveillance of people, human machine interaction, and video conferencing (14, 49). In this report we define motion capture of human body parts as a process that involves the segmentation of the human body into links connected by joints, and then recovers the 3D structure of the human body using its 2D projections over a sequence of frames coming from eight cameras.
13.3.3. Objective
Our objective in the VSR project is to use the motion capture system as a realistic experimental tool to verify the analysis and the optimization algorithms that are used to predict the motion and the performance of our virtual human model Santos™. Other goals include using the system as an analysis measuring tool to study gait in patients with walking impairments, ergonomics studies, anthropometry, and finally and most importantly for capturing and animating of flexible geometries such as skin and clothes.
In the following section we will explain the basic components of our Vicon motion capture system (12) and the general steps in conducting motion tracking of moving subjects.
13.4. Basics
13.4.1. Vicon Motion Capture Components
In its permanent location at the CCAD, the University of Iowa, our Vicon motion capture system comprises a motion capture volume (15ft x 15ft x 10ft) surrounded by eight high-resolution cameras mounted on a structure attached to the ceiling of the volume Fig (1). Nevertheless, our motion capture system is portable and can be easily moved and erected in remote areas using eight tripods to hold the cameras (Fig.2).
The subject, whose motion is to be captured, wears a special none reflected suit with a number of reflective markers attached to his/her body in well defined positions to identify joints and segments locations (Fig. 3). Each camera in this system has a ring of LED strobe lights fixed around the lens. As the subject moves inside the capture volume, light from the strobe of the cameras is reflected back into the camera lens and strikes a light sensitive plate creating a video signal. The signal then passes through a filter that allows only light of the same wavelength through to be focused on the camera’s sensitive plate Our cameras can capture 200 frames per second and the cameras are separated from their strobe units. On the cameras the strobe ring fits around the lens, being held in place by a magnetic strip. The strobe connects electrically via a short flying lead. The signals are then collected by the data-station, which also controls the cameras and the strobes along with any other recorded data (sound or analogue signals from force plates for gait analysis). The data-station is then passes the information to a computer on which the Vicon software suite is installed.
13.4.2. Calibration
Calibration is an essential first step in any motion capture operation and plays a considerable role in minimizing the error in the resulting data due to the effect of the surrounding environment or instrument deficiency. Calibration is the process that measures the position and orientation of the capture volume and the location of each camera relative to the others. The calibration data is used in the reconstruction process of 2D data to create the virtual three-dimensional motion. There are two calibration steps in our Vicon system, which will be explained in the following.
13.4.2.1. Static Calibration
Static calibration is used to set and locate the origin and the directions of the global axes of the motion capture space. In our Vicon system we used the LFrame (Fig.4a) to conduct the static calibration for full body capture. The LFrame has an L shaped arrangement that holds four reflective markers, three on one arm and one on the other. Two alignment gauges are also mounted on the LFrame so that the frame can be aligned accurately with force plates positioned on the floor or when used in remote area. The location of the LFrame in the motion capture space during the static calibration process is shown in Figure 4b.
13.4.2.2. Dynamic Calibration
The second step in the calibration process is the dynamic calibration, which is used to calculate the relative positions and orientation of the cameras in the captured space. In this process, we used a dynamic calibration T-shape object called the wand (Fig 5a). For medium and large volumes this is a rod with two 50mm reflective balls placed 500mm apart. For small volumes the rod has 14mm spheres 100mm apart. In our system, we use a 390 mm wand with three 25mm markers. The process of dynamic calibration involves moving throughout the capture volume and waving with the wand so that it passes through and covers as much of the capture volume as possible allowing each camera to record the wand in several orientations as shown in Fig. 5b.
13.4.3. Motion Capture Process
After the calibration process, the real subject, whose motion is to be captured will enter the motion capture space with a number of reflective markers attached to it in well defined positions (see Fig. 3 for an example of a human subject) to identify joints and segments locations in time and space. The locations of the markers on the subject body are chosen according to a special template (Fig. 6), which represents a generic representation of the real subject.
Figure 6: A generic human template
In the motion capturing process, the real subject will be asked to move and conduct a set of movement in the capture space (Fig. 7). During this time, the eight cameras are used to track and record the motion of the subject. This information will be then send to the workstation which is the central application of the Vicon software used to collect and process the raw 2D video data. It takes the two-dimensional data from each camera (Fig.8) and then combines them with the camera coordinates and other cameras views to obtain three-dimensional (x, y, and z) coordinate of each marker in the list per frame in time and space (Fig. 9). The positions calculated frame by frame are then combined to create a complete set of trajectories of the markers positions throughout the time span of the trial. This can be demonstrated by graphs showing the trajectory history of each marker in x, y, and z directions.
In an ideal situation, the reconstruction of the data from 2D to 3D should result in smooth and continuous trajectories that define the path of each marker for the duration of the trial through the capture volume. Sometimes, however, the captured data is incomplete and this happens in situations where some markers are not being seen by at least two cameras, which is the minimum requirement to calculate the position of the markers for each frame. This phenomenon is called occlusion, which results in broken trajectories, and the resulting motion will be discontinuous. In some other cases the markers are so close to each other in the view such that the calculation algorithm confuses one for another, which results in another phenomenon known as a crossover . In other cases, the software produces trajectories for markers that do not really exist in the capture space but may results from reflection of spotlight or flashing material in the motion capture space, which are known as ghost markers. Nevertheless, our Vicon motion capture system has powerful post-processing capabilities that can deal and repair such problems as will be explained in the coming sections.
13.4.4. Labeling and Calibration
Reconstructed 3D data consists of three-dimensional trajectories, which represent the locations of the markers that are attached to the subject in time and space. A trajectory is the path of a marker in space during the trial. Workstation calculates the trajectories by joining the positions of each marker from frame to frame. At this stage it is possible to review the motion of the captured subject as a number of unlabelled markers (white dots in Fig. 9) that move in space. However, the software does not have the knowledge to distinguish between one trajectory from another. The process of labeling allows the software to make this distinction. This recognition operation is a two-step process. The first step is a manual labeling which comprises the manual labeling of the marker set at the first frame or at number of frames according to a specified labeling template represented by a generic skeleton structure shown in the lower part of Figure 10. This process gives the auto labeler (as a second step) the initial information toward labeling the subsequent frames automatically by looking for consistent patterns amongst markers. The upper part of Figure 10 shows the markers after the labeling process where each marker has its own color and name to recognize it from other markers. At this stage, the name and color of each marker should be consistent with its name and location in the generic subject template shown at the lower part of Figure 10.
13.4.4.1. Subject Calibration
In this process Vicon software will seek the relation between the labeled markers set shown in the upper part of Figure 10 and the markers on to the generic skeleton structure in the lower part of Figure 10. The software then uses optimization techniques to map the markers set onto the generic skeleton structure. The result will be a skeleton structure that resembles the real subject in terms of segment dimensions and joints locations.
13.4.4.2. Kinematic Fitting
In the kinematic fitting process, Vicon software will try to seek retargeting between the captured subject and the generic character skeleton (Fig. 11). If there is a lot of disproportion between the two then the software may have problems to conduct the fitting process adequately. It is more likely to have better retargeting solution if the scale of the real subject is close to the generic skeleton structure.
13.4.5. Editing
If the resulting 3D data are continuous and smooth without any noise, then it is possible to go on to the next step of visualizing, modeling, analyzing, or applying the data to an animated character. If however the data is not exactly perfect and have some anomalies at this point then the data need to be edited and cleaned before moving on to the next steps. The general steps for cleaning and editing data depend on the quality of the resulting data. The latter is a large spectrum of possibilities ranging from minor editing of small noise using filters to non-continuous and missing trajectories that need a major editing process, which can be conducted in Vicon post processing operations. The small discontinuity in the trajectories of each marker can also be filled using Vicon editing tools. One example is shown in Figure 12, which shows a number of gaps in the trajectories of one marker (grey regions). Such gaps can be filled using different interpolation schemes such as linear, quadratic, spline, etc. Figure 13 shows Figure 12 with the gaps being filled using spline interpolation. It should be noted here that this modification process is very crucial and has considerable effect on the final results.
13.5. Motion Capture of Multiple Subjects
There are many examples where the study of multiple subjects is important, for example, the movement of group of soldiers while they hold a number of equipments and conduct a certain mission, or the interaction in sports scenarios between the players and the tools they are using. In Vicon motion capture system, objects such as furniture, walking aids, sports equipments can be considered as subjects (props). These subjects are treated in a similar way in the standard motion capture process. The main differences are to decide on how and where to place markers on their bodies.
With a single marker we can capture the prop position only. While, with two markers can capture the position and orientation of the prop. Then with three markers it is possible to capture the position, orientation and rotation of the prop. Many tests have been conducted using our motion capture toward capturing multiple subjects interfering with each other or with a prop. In one example ( Fig. 14), we demonstrate the process of having a person walking toward a prop on the floor of the motion capture space.
13.6. Research at VSR and Major Challenges
13.6.1. Planned Usage of the Motion Capture System
Motion Capture is the process of recording a live motion event and translating it into usable mathematical terms by tracking a number of key points in space over time and combining them to obtain a single 3D representation of the performance [1, 47, 48]. There are wide ranges of productive ways to apply this technology to facilitate the VSR effort here at University of Iowa. Some of these are listed here:
1. The motion of real humans assigned to perform specific tasks can be captured as they carry out such tasks. The captured human motion can then be compared with the motions predicted by the virtual human model when performing the same task.
2. The virtual human models are being developed to work on principles of optimization, namely perform assigned tasks while minimizing such quantities as effort expended, discomfort, and injury probability. To see how these quantities vary in the virtual human model when performing tasks, measured human motion can be used to drive the virtual human model as he/she goes through the prescribed motions, the quantities such as effort expended, discomfort, and injury probability can be monitored to study which, if any, are actually minimized.
3. Animating the virtual human model in real time is one major task in the VSR project. Motion capture can be utilized to detect the motion of real subject in real time. The resulting data can be forwarded through the network to various animation engines such as Kaydara and Real-Time Simulation code. This issue is very important when conducting tests and exchanging information between different remote areas in real time.
4. The impact of different types of clothing on human motion can be assessed. One of the simplest places to begin is to measure changes in joint ranges of motion when certain types of clothing are worn. For example, how does the range of motion for the elbow change when the arm is ensleeved in a stiff, bulky fabric? This is just a starting point, but studies of this type will be conducted over the coming months and integrated with the virtual human model.
5. Motion capture can also be used to study the effective properties of fabrics based on how they interact with moving humans, as measured with motion capture systems. (The literature in section 6.1.4.5 describes how this might be accomplished).
6. The effect of walking impairment on the stability of patients can be addressed using motion capture. This is one major area in the study of gait analysis. Motion capture can be used in this regard to obtain various gait parameters such as stride length, cadence, joints and hip angles and then compare these parameters with normal people. Motion capture data can be also used as a reliable source to verify the dynamic stability analysis of our virtual human model Santos™. The work in this regard is started at the VSR project and a lot of effort and resources are devoted toward taking this work to the level of the state of the art in this field.
7. Ergonomics and anthropometric studies are other fields for motion capture. Studies in these fields are considerably significant and many institutions are using motion capture toward this end. In the VSR project we have conducted some preliminary works and we are planning to use our motion capture system in a number of projects for the near future.
13.6.1.1. Santos Mimicking a User
At this time, many motion capture tests have been conducted toward animating our virtual human model Santos in Real-Time Simulation code environment. In order to achieve such goal we built a skeleton structure inside Vicon (Fig. 15) that looks similar to Santos skeleton (Fig. 16) in terms of the number of segments and joint types.
Our Vicon motion capture system has the capability of obtaining local joint angles between adjacent segments using one of its pipe line post processing operations. Due to the differences between Vicon and Real-Time Simulation code coordinate systems, however, an algorithm to map the data from Vicon environment to Real-Time Simulation code engine is obtained by Vicon motion capture team. Motion capture data have been used in many tests to animate Santos in Real-Time Simulation code. One example is illustrated in Figure17. Part (a) of the latter figure depicts a Vicon skeleton structure conducting a posture inside Vicon environment (Fig 17a) ; while Figure 17b shows Santos performing the same posture inside Real-Time Simulation code environment animated by the same set of motion capture data.
13.6.1.2. Validating Posture and Motion Prediction
In the VSR project our colleagues used optimization algorithms and techniques to predict various performance of our virtual human model Santos™. The hypothesis behind optimization-based posture prediction for example, is that human motion is characterized by a performance measure or objective function, such as discomfort, potential energy, effort, etc, which can be used to predict human posture. This hypothesis can be extended to suggest that motion is governed by multiple of objective functions that can be combined simultaneously to predict human motion. Basically, the human body is modeled as a kinematics system represented by a series of segments connected by joints which represent musculoskeletal joints such as the wrist, elbow, shoulder, clavicle, pelvis, etc. (Fig. 18) Optimization tools are used to determine the rotation at each degree of freedom of each joint that minimize a performance measure such as discomfort.
In one study the VSR group use two objective functions such as discomfort and potential energy are considered.These objectives are compared independently. Then, by evaluating Pareto optimal sets for various target points (points in space that the human model must reach or contact), they determine if the final posture is more sensitive to one objective.In addition, they determine how dependent the performance of the objective functions is on the nature of the target point.
Currently, we are using motion capture system toward verifying such posture prediction analysis and optimization algorithms. In this regard, we built a skeleton structure in Vicon, which resembles the skeleton structure of Santos as shown in Figure 19. A number of preliminary experiments have been conducted using motion capture system of a real subject to conduct a set of postures and then compare the results with the analysis results. One example is depicted in Figure 20 where a subject is trying to conduct a number of postures to reach and touch different targets in space. On the other hand, Figure 21 shows a posture of Santos predicted by the analysis results using a set of objective functions.
There are many challenges in this field toward using motion capture as a reasonable and expectable verification tool for comparison purposes.
1. The process should be statically acceptable. In this regard, the motion capture test should include a range of people with specified age, gender, and body height and weight, with each of them conducting many reach out processes.
2. The comparison environment is Real-Time Simulation code. As can be seen in Figure 21 that Real-Time Simulation code is the animation engine for the analysis. Therefore, we need to find the relation between Vicon motion capture coordinate system and Real-Time Simulation code coordinate system for the various joints before using the data for verifications purposes.
3. Anthropometric studies of human body parts dimensions for the participants using motion capture system should be included in this work. This is a very important factor and can play a considerable rule toward achieving meaningful comparison between the analysis and the experiments.
(a) 
(b) 
Figure 20: The real subject conducting various postures to reach a
number of targets.
Figure 21: Santos conducting a posture to reach and touch a target.
13.6.1.3. Interacting with Santos in Real Time
Using motion capture systems to animate a virtual human in real time is an active research topic in the virtual reality community. However, techniques for capturing fast motion in real time, which can be found in a wide range of applications from virtual theatre to virtual sport, remain a challenge (13). Our Vicon optical motion capture system has the real time engine Tarsus that has the ability to record data, show the flow of data, and broadcast the 3D coordinates of the markers on the network for clients to receive in real time. While these real time properties are useful and considerably important for remote data sharing, it is still very useful to flow this data to software like Real-Time Simulation code to animate a virtual player in real time. This issue of animating Santos inside Real-Time Simulation code in real time using motion capture data is one major goal in the VSR project. This process can be explained by referring to Figure 22. Our goal is to have a subject with a number of markers attached to his/her body, conducting a motion inside the motion capture space. At the same time, we need the resulting data to flow in real time to Real-Time Simulation code engine to animate Santos in real time. Such process is very important in sharing our motion capture data through different networks in different remote locations. The work toward this end is now in it final stages and we hope to achieve this target in the very near future.
13.6.1.4. Motion Capture of Cloth
Reliable geometric data collections for non-rigidly moving surfaces such as skin and cloth can play significant roles for applications in character animation and ergonomic design [14]. One important issue in tracking motion using reflected markers is the tendencies of skin or cloth to move during the motion capture process. While most motion capture post processing systems rely upon data that reflect the relative positions of the markers to rigid segments (assuming rigid body and constant distance between the marker and the segment), skin and cloth or any deformable geometry will have variable distance between the markers in time and space. In our current work at VSR lab we are working on using our motion capture system to track the location of a set of markers attached to a piece of a flexible geometry such as cloth in 3D space. One advantage of this process is to demonstrate the deformed shape of the geometry and possibly showing the resulting wrinkles. The latter process will have significant role when comparing the resulting shape of the deformed cloth with the analysis-clothing model introduced by our colleague in the VSR project. One example utilizing this approach is depicted in Figure 23 where we tracked the motion of a thick curtain by placing a set of markers on its surface and used motion capture system to find the locations of these markers in space.
13.6.1.4.1. Motion Capture in Clothing Simulations
There have been many attempts in the literature to model the static and dynamic cloth behavior in real time utilizing physical models and motion capture techniques. Although a substantial amount of literature exists on fast cloth simulation, very little work has been performed in the area of true real time simulation. Many of the approaches applied toward modeling cloth using physical models have a good degree of realism simulating the cloth, but their common drawback is low speed. Vassilev et al [15] and Kang et al [16] described a fast technique for animating clothing on walking humans. The system is based on an improved mass- spring model of cloth and a fast new algorithm for cloth-body collision detection. By utilizing such approach it is possible to model cloth in real time in static and dynamics situations. One of the challenges in motion capture of cloth, as a flexible geometry is the difficulty in their tracking process. Toward this end, Guskov and Zhukov [17] introduced a robust tracking procedure for a regular pattern marked on a flexible moving surface, such as cloth, using motion capture. They were able to deduct the structure of the grid marked on the surface and successfully track it through long periods of time.
Another significant issue in cloth modeling for virtual human is the contact and collision between the skin and the cloth. Pelechano et al [18] addressed the problem of performing collision detection between two flexible objects, the skin and the cloth in real time animation of virtual humans. In another work, Pritchard and Heidrich [19] presented a system for the capture of deformable surfaces, most notably moving cloth, including both geometry and parameterization. Using motion capture techniques with three cameras they can therefore only capture one side of the cloth geometry, limiting it to applications such as moving curtains, flags or draped cloth. Using a larger number of calibrated cameras (traditional motion capture systems often use 8 or more), their method could be extended to applications such as clothing.
Motion capture data, which represents the experimental basis for most motion analysis, can also be used as a source for deriving many algorithms. Bhat et al [20] presented an algorithm for estimating the parameters of a cloth simulation from video data of real fabric. A perceptually motivated metric based on matching between folds is used to compare video of real cloth with simulation. This metric compares two video sequences of cloth and returns a number that measures the differences in their folds.
Simulated annealing is used to minimize the frame-by-frame error between the metric for a given simulation and the real-world footage. To estimate all the cloth parameters, they identified static and dynamic calibration experiments that use small swatches of the fabric. They reported the results of simulation parameters obtained using their technique applied to four fabrics: linen, fleece, satin and knit, and measured the mass and dimensions of the fabrics and accurately measured the position of the two top corners of the hanging fabric using a >Vicon motion capture system.
Motion capture data can also be used to enhance the animation. Pullen and Bregler [21] discussed a method for creating animations that allow the animator to sketch an animation by setting a small number of key frames on a fraction of the possible degrees of freedom. Motion capture data is then used to enhance the animation. They proposed a method for combining the strengths of key frame animation with those of using motion capture data.
In this report, it is tempting to conclude that the capturing process of cloth in real time considering static, dynamics, skin contact, and resulting forces is an achievable goal Using the spring-mass approach cited in many works (see for example ref. 15&16) which appears to be the most realistic way for the time being to be used in this work.
In summary, one way to achieve our objectives for cloth modeling in real time is to formulate an inverse problem by using some existing physical models such as the mass spring systems, which have the ability to model the static, dynamics, and contact situation in real time, and the motion capture technique. By having the motion capture of any type of clothing, it is possible to find the cloth configuration and the effect of the cloth on the human joints angles for each time frame. These data can be forwarded to a mass-spring model to find the resulting forces. The resulting forces and the range of angles for each joint can then be used by other VSR groups in their analyses. The significance of using motion capture of flexible geometry will become considerably more important in the VSR project when we include the modifications of deformable muscles and skin on our virtual human model Santos™. Then, the study of the interaction between these geometries will be achievable.
In the following we present an example of using motion capture at the VSR project to study the effect of clothing on various gait parameters. In this regard, we capture the motion of person with heavy clothing wearing a leather suit, which imposes a significant constraint on her postures and motion (Fig. 24-26). In the preliminary stages, we compare the resulting joint angles when the actor wears these heavy clothing and when she wears normal clothing. The final goal of this project is to study the effect of heavy clothing on gait and balance of soldiers conducting emergency tasks and wearing their protective suits in chemical, biological, or nuclear environments, by utilizing motion capture and dynamic stability criterions. We believe that the results of this study will add considerable advantages to the safety and performance of these soldiers when they conduct crucial tasks in hostile environments.
Figure 24: The effect of heavy cloth on hand posture.
Figure 25: The effect of heavy clothing when reaching the floor.
Figure 26: The actor wearing the heavy cloth underneath the black suit and trying to conduct a motion.
13.6.1.5. Gait Analysis
Motion capture systems have been well established in the field of gait analysis and there have been many applications in which this technology has been implemented. Applications include analyzing spinal and hip motion (6, 22, 23, 24), walking and falling (25, 26), anthropometric factors (27, 28), ergonomic design (29, 30, 31), athletics studies (32, 33), and a potential usage in psychiatry (1). Many people in different field are now interested in using motion capture systems because motion capture systems have been shown to be accurate, reproducible, and consistence (34).
Previous studies have shown that precise gait measurements are essential tools toward finding subtle changes in gait parameters (35, 36). These subtle changes can have significant roles in understanding the mechanism of imbalance and in shaping the course of treatment. For example, precise gait analysis was able to detect subtle alternations in balance in patients with traumatic brain injury at the time when clinical exams and scales (Tinetti Balance Scale) did not demonstrate a significant difference between the patient population and normal control subjects (35, 36). Modern motion capture systems based on infrared cameras and reflecting markers attached on the patient’s body can be used to provide extensive and precise data of gait parameters for balance analysis.
Figure 27 illustrates the planned procedure of using the motion capture system at the VSR project toward this end. First, gait parameters such as stride length, cadence, hip and joint angles, base width, and gait velocity will be identified for each subject using motion capture system to acquire precise measurements (Fig. 27a,b). The range of these parameters within a certain group of subjects will be quantified. An attempt will be made to identify those parameters differences that are most profound. Second, the effect of the various gait parameters on the subject stability will be identified using our dynamics stability model (Fig. 27c).
13.6.1.5.1. Dynamics Stability Model
The main objective in the VSR project is to build a virtual human model (Santos) that looks like and behaves like a human. In order to achieve such goal, the virtual human model must acquire the necessary physical tools that make him able to sense and be aware of the surrounding environment. One of these tools is the process of making Santos capable of exerting and reacting to the external loads in his living environment. In this section we will present the role of our motion capture system toward this end and the ongoing research in this field.
Our dynamics stability skeleton model at the VSR project at this time has 17 links and 25 degrees of freedom in three-dimensional space (Fig. 28). The model considers the internal forces represented by joints forces and muscle torques. The internal forces react to the external forces to stabilize body motion or to perform an intended motion. The human body is considered as an indeterminate system, because there is more than one solution that satisfies the equation of motion for the human body. For this reason, gradient based optimization methods have been successfully applied to solve for the motion of such indeterminate system (37-41). The optimal motion is obtained as a variant of the physical attributes (mass distribution, dimensions), motion goal (pick up, deliver), environmental factors (floor conditions), etc. The objective of the optimization problem can include maximum stability, minimum energy consumption, and minimum torque condition. The human dynamics stability model is based on a number of stability parameters that need to be satisfied. The stability parameters measure the balance of the body under instantaneous disturbances (force/moment). The stability parameters include,
(a) Zero Moment Point (ZMP), which is the conceptual extension of a static stability parameter Geometrical Center of Mass (GCoM) to a dynamic case. The dynamic equilibrium is satisfied as long as the ZMP stays within the ground-contact footstep. The stability of a given motion can be increased by forcing the ZMP to be as close as possible to the center of footstep.
(b) Torque Margin, which measures the margin of the actuating torque to the maximum torque that a person can exert at each joint in the human body. The stability for a given motion will be higher with higher torque margin.
A number of preliminary tests have been conducted in the motion capture research in the VSR project to verify the dynamic stability formulas for the virtual human model. In these tests we are working to obtain the above stability parameters using a realistic human motion capture data and compare these data with those predicted by the analysis and optimization tools. In one of these tests, we tracked the motion of a person holding heavy loads while walking. In another test, we capture the motion of a person pushing a heavy load as shown in Figure 29a. In this regard, we use the resulting gait parameters from the motion capture system as input to our dynamics stability criterion to find the ZMP, then compare these results with that predicted by the stability analysis model (Fig. 29b).
13.6.1.6. Ergonomics Studies
There has been a significant amount of research performed toward studying the effect of ergonomics factors on the performance of people using motion capture. For example, in vehicles design process, the biomechanical study of the driver&# 146;s posture and movements are the most imperative aspects for the ergonomic design process of the entire vehicle. The objective of these studies is to design efficient and acceptable driving environments for the occupants of the vehicle such as how a driver’s body interacts with the cushion and backrest of the car seat as the person changes his or her posture to complete various tasks. When using motion capture in such studies, a large number of markers are required to identify the vehicle interior and the car driver (26). Another example of using motion capture for ergonomics study is a sports related injury. In most sports the players move and rotate their limbs abruptly during the game, which makes their joints vulnerable to injury. Motion capture systems have been used in such performance to analyze the motion of joints, such as the shoulder when throwing a baseball or the knee while running (29, 43- 46).
Motion capture allows the designer to obtain realistic data that is directly related to the type of postures and movements a person makes while he/she interacts with the equipments in their surrounding environments. These data will then be considered as significant source of information to verify the analysis and optimization tools. In the preliminary tests at the motion capture lab at the VSR project, we are working to analyze the effect of interaction between human subjects and their working environment. One example (Fig. 30) is devoted to study the effect of an office chair height on the comfort level of the user. A number of markers have been installed in predefined locations to identify the chair and user parameters. During this test, the user is asked a number of questions about her level of comfort while we change the height of the chair.
At our motion capture lab we are planning in the near future to use our system extensively in a number of ergonomics tasks that include the interaction between the workers and their working environment. Our goal in these studies is to increase workers safety and production level.
Figure 30: example of the ergonomic studies of how the chair is comfortable
13.6.1.7. Anthropometry
Anthropometry is the study of human body measurements such as Leg length, Knee width, Ankle width, etc (27). In most virtual reality system it is crucial to impose realistic anthropometric characteristics that accurately correspond to the user. In this project, we are planning to obtain algorithms that are capable of measuring the dimensions of specific anatomic landmarks of a subject (real human) and reproducing the corresponding virtual human representation of the subject anthropometry in the resulting virtual human. In another word, we can accurately relate the resulting virtual human model to a specific human actor.
The following example demonstrates a simple algorithm to find the equation and location of surfaces in space and the sizes of a parallelogram using motion capture 3D data.
13.6.1.7.1. Example
In this example, we demonstrate a simple procedure of how to use the resulting 3D markers location in 3D space to find the dimension of a parallelogram (Fig. 31a). In this process, we attached a number of markers on each side of the box and conduct a classical motion capture trial to find the locations of the markers in space with respect to the global coordinate system (Fig. 31b). Then, we used this information in our analysis, for instance to find the distance between the top surface and the lower surface of the box we did the following:
First we found the vectors >
and >
on the top surface in the global coordinate system. Then we normalized and
and conducted a cross product between
them to find the equation of a unit normal >
on the top surface.
To
find the distance between the top surface and the lower surface, we choose a marker at the
lower surface like marker c and found the vector 
between point 
and point c (Fig. 31a). The distance (d) in this
case will be the projection of 
onto the normal .
In general we can use similar approaches to find the dimensions of the different parts of the human body. It should be noted here that our Vicon motion capture system has the facility of providing the distance between the markers and the center of the joint to which these markers are attached.
13.7. Executive Summary
In this report, we have presented the complete history of our Vicon motion capture system from the time we bought the system until the time of this report. At the beginning of this report we have given some background about the basic operation steps in our motion capture systems and its capabilities. Then we introduced what have been accomplished in this area, our current work, and the plans for near future work.
As have been shown in this report that there are wide ranges of applications that have been applied using the motion capture technology to facilitate the VSR effort here at University of Iowa. Some of these are listed here:
1. The motion of real humans assigned to perform specific tasks has been captured as they carry out such tasks. The captured human motion is then compared with the motions predicted by the virtual human model when performing the same task using animating environments such as Real-Time Simulation code.
2. The virtual human models are being developed to work on principles of optimization, namely perform assigned tasks while minimizing such quantities as effort expended, discomfort, and injury probability. Motion capture has been used to provide experimental realistic data to verify some of these approaches. In the near future, we believe that motion capture will play a considerable role in this field.
3. Animating the virtual human model in real time is one major task in the VSR project. Motion capture has been utilized to detect the motion of real subject in real time. The resulting data can be forwarded through the network to various animation engines such as Kaydara and Real-Time Simulation code. This issue is very important when conducting tests and exchanging information between different remote areas in real time. The work toward this end is in its final stages, and we are collaborating with Vicon motion System to achieve this goal in the near future.
4. The impact of some types of clothing on human motion has been assessed. In the preliminary stages, we compare the resulting joint angles when the actor wears heavy clothing (leather) and when she wears normal clothing. In addition, in our current work at VSR lab we are using our motion capture system to track the location of a set of markers attached to a piece of a flexible geometry such as cloth in 3D space. One advantage of this process is to demonstrate the deformed shape of the geometry and possibly showing the resulting wrinkles. The latter process will have significant role when comparing the resulting shape of the deformed cloth with the analysis-clothing model introduced by our colleague in the VSR project. The final goal of this project is to study the effect of heavy clothing on gait and balance of soldiers conducting emergency tasks and wearing their protective suits in chemical, biological, or nuclear environments, by utilizing motion capture and dynamic stability criterions. We believe that the results of this study will add considerable advantages to the safety and performance of these soldiers when they conduct crucial tasks in hostile environments.
5. Motion capture can also be used to study the effective properties of fabrics based on how they interact with moving humans, as measured with motion capture systems. Algorithms to ward this end will be presented in future work.
6. The effect of walking impairment on the stability of patients using motion capture system has been started at the VSR project. This will be used besides the dynamic stability analysis of our virtual human model Santos to understand the mechanisms of these impairments.
7. Ergonomics and anthropometric studies are other fields for motion capture. In the VSR project we have conducted some preliminary works and we are planning to use our motion capture system in a number of projects for the near future.
As have been indicated in this report that motion capture is a field with enormous applications and many institutions are devoting huge effort in using such technology in various applications. At the VSR project, we believe that we have now a good experience in using our motion capture system and we are planning to achieve our goals at the VSR project in the near future and contribute in other related projects.
13.8. References
[1] Alessi, N. E.; Haung, M. P., 2001: Digital motion phenomenology of depression, Studies in Health Technology & Information. 81, 30-37.
[2] Komura, T.; Shinagawa, Y., 2001: Attaching physiological effects to motion captured data.Preceedings Graphics Interface, Otawa, Ont, Canadian Information Processing Socity.
[3] Brouwer, B.; Musselman, K; Culham, E. 2004: Physical function and health status among seniors with and without a fear of falling, Gerontology, 50(3), 135-141.
[4] Kejonen, P.; Kauanen, K.; Vanharanta, H. 2003: The relationship between anthropometric factors and body-balancing movements in postural balance. Archives of Physical Medicine & Rehabilation, 84(1), 17-22.
[5] Brady, RA.; Pavol, MJ.; Owing, TM.; Grabiner, MD. 2000: Foot displacement but not velocity predicts the outcome of a slip induced in young subjects while walking. Journal of Biomechanics, 33(7), 803-808.
[6] Zhang X.; Xiong, J.; Bishop, AM., 2003: Effects of load and speed on lumber vertebral kinematics during lifting motions. Human Factors, 45(2), 296-305.
[7] Guskiewicz KM.; Riemann BL.; Perrin DH.; Nashner LM., 1997: Alternative approaches to the assessment of mild head injury in athletes. Medicine & Science in Sports & Exercise. 29(7 Suppl):S213-21.
[8] Zakotink, J; Matheson, T.; Durr, V. 2004: A posture optimization algorithm for model-based motion capture of movement. Journal of Neuroscience Methods, 135(1-2), 43-54.
[9] Andreoni,G.; Santambrogio, G.; Rabuffetti, M.; Pedotti, A., 2002: Method of the analysis of posture and interface pressure of car drivers. Applied Ergonomics, 33 511-522.
[10] Menache, A. 1999: Understanding motion capture for computer animation and video games, Morgan Kaufmann.
[11] Miller, C.; Mulavara, A.; Bloomberg, J., 2002: A quasi-static method for determining the characteristic of motion capture camera system in a “split-volume” configuration. Gait & Posture. 16(3), 283-287.
[12] Vicon motion system manual and documentations.
[13] Tang, E.; Cavazza, M.; Mountain, D.; Earnshaw, R. 1999: A constrained inverse kinematics technique for real-time motion capture. The Visual Computer 15, 413–425
[14] Aggarwal, J.K.; Cai, Q. 1999: Human Motion Analysis: A Review”, title = "Human Motion Analysis: {A} Review", Computer Vision and Image Understanding: CVIU, 73, 3, 428-440.
[15] T. Vassilev, B. S.; Chrysanthou, Y. 2001: Fast cloth animation on walking avatars, EUROGRAPHICS 2001/ A. Chalmers and T.-M. Rhyne, 20.
[16] Kang, Y. M.; Choi, J.H.; Cho, H.G.; Lee, D.H. “An efficient animation of wrinkled cloth with approximate implicit integration”, Department of Computer Science, Pusan National University.
[17] Guskov, I.; Zhukov, L. “Direct pattern tracking on flexible geometry”.
[18] Pelechano, N.; Bull, L.; Slater, M. “Fast collision detection between cloth and a deformable human body”, Department of Computer Science, University College
London, UK.
[19]Pritchard, D.; Heidrich, W. 2003: Cloth motion capture, EUROGRAPHICS 2003 /
P. Brunet and D. Fellner, 22.
[20] Bhat,K.,S.; Twigg,C.D.; Hodgins, J.K.; Khosla, P.K.; Popovi, Z.; Seitz, S.M. 2003:
Estimating cloth simulation parameters from video, EUROGRAPHICS/ SIGGRAPH Symposium on Computer Animation. (Eds.) D. Breen, M. Lin.
[21] K. Pullen, and C. Bregler “Capture assisted animation: texturing and synthesis”,
Stanford University
[22] Hagio, K.; Sugano, N.; Nishii, T.; Miki, H.; Otake, Y.; Hattori, A.; Suzuki, N.; Yonenobu, K.; Yoshikawa, H.; Ochi, T. 2004: A novel system of four-dimensional motion analysis after otal hip athroplasty. Journal of orthopaedic Reaserch, 22(3), 665-670.
[23] Reft, J.; Hasan, Z. 2002: Trajectories of target reaching arm movements in individuals with spinal cord injury: effect of external trunk support, Spinal Cord, 40(4), 186-191.
[24] Manal, K. McClay, D., Galinat, B., and Stanhope, S. 2003: The accuracy of estimating proximal tibial translation during natural cadence walking: bone vs. skin mounted targets. Clinical Biomechanics, 18(2), 126-131.
[25] Rachel, A., Michael, J., Tammy, M., Mark, D. 2000: Foot displacement but not velocity predicts the outcome of a slip induced in young subjects while walking. Journal of Biomechanics, 33, 803-808.
[26] Khoo, B.C, Goh, J.C, Bode, K. 1995: Biomechanical model to determine lumbosacral loads during single stance phase in normal gait. Medical Engineering & Physics, 17 (1), 27-35.
[27] Kejonen, P.; Kauanen, K.; Vanharanta, H. 2003: The relationship between anthropometric factors and body-balancing movements in postural balance. Archives of Physical Medicine & Rehabilation, 84(1), 17-22.
[28]Kang, J.; Chaloupka, EC, Mastrangelo, MA.; Hoffman, JR. 2002: Physiological and biomechanical analysis of treadmill walking up various gradients in men and women,European Journal of Applied Physiology, 86(6), 503-508.
[29] Andreoni, G.; Santambrogio, GC.; Rabuffetti, M.; Pedotti, A. 2002: Method for the analysis of posture and interface pressure of car drivers, Applied Ergonomics, 33(6), 511-522.
[30] Chaffin, D. 2002: On simulating human reach motions for ergonomics analyses. Human Factors and Ergonomics in Manufacturing, 12 (3), 235-247.
[31] Maiteh, B. 2003: An application of digital human modeling and ergonomics analysis in workplace design. American Society of Mechanical Engineers, Manufacturing Engineering Division, MED, 14, 731-735.
[32] Nakamura, Y., Otsubo, N., Hayashi, T., Nakamura, M., Nishikawa, H., Tachibana, T., Nobuhara, K. 2002: Preliminary study of 6 degrees of freedom measurement of scapula movement using a motion capture system. Annual International Conference of the IEEE Engineering in Medicine and Biology - Proceedings, 3, 2564.
[33] Ashby, BM.; Heegaard, JH. 2002: Role of arm motion in the standing long jump, Journal of Biomechanics, 35(12), 1631-1637.
[34]Miller, C.; Mulavara, A.; Bloomberg, J., 2002: A quasi-static method for detemining the characteristic of motion capture camera system in a “split-volume” configuration. Gait & Posture. 16(3), 283-287.
[35] Basford, JR.; Chou, LS.; Kaufman, KR.; Brey, RH.; Walker, A.; Malec, JF.; Moessner, AM.; Brown, AW.2003: An assessment of gait and balance deficits after traumatic brain injury. Archives of Physical Medicine & Rehabilitation. 84(3), 343-349.
[36]Chou, L., Kaufman, K.R., Brey, R.H. 2000: Stepping over an obstacle challenges dynamics stability in the elderly adults. 24th Annual Meeting of the American Society of Biomechanics, University of Illinois at Chicago, July 19-22.
[37] Mostafa, R.; Bessonnet, G. 2001: Sagittal gait of a biped robot during the single support phase. Part 2: optimal motion, Robotica , 19, 241-253.
[38] Saidouni ,T.; Bessonnet, G.; 2003: Generating globally optimized sagittal gait cycles of a biped robot, Robotica, 21, 199-210.
[39] Mu, X.; Wu, Q. 2003: Synthesis of a complete sagittal gait cycle for a five-link biped robot. Robotica, 21, 581-587.
[40]Chevallereau, C.; Aoustin, Y. 2001: Optimal reference trajectories for walking and running of a biped robot, Robotica , 19, 557-569.
[41] Asano, F.; Yamakita, M. 2001: Virtual gravity and coupling control for robotic gait synthesis, IEEE transactions on systems, man, and cynbernetics- Part A: Systems and Humans, 31( 6).
[43] Nakamura, Y.; Otsubo, N.; Hayashi, T.; Nakamura, M.; Nishikawa, H.; Tachibana, T.; Nobuhara, K. 2002: Priliminary Study of 6 Degrees of Freedom Measurement of Scapula Movement Using a Motion Capture System. Second Joint EMBS/BMES Conference, Houston, TX.
[44]Nakamura, M.; Nakamura, Y.; Kato, N.; Hayashi, T.; Fukuda, N.; Nobuhara, K.; 2002: Differences of Joint Movements Between the Shoulder and the Elbow Used in Fastball/Curveball Pitches a Dynamic Analysis. Second Joint EMBS/BMES Conference. Houston, TX.
[45] Gatt, C Jr.; Pavol, M.; Pavol, M.; Parker, R.; Grabiner, M. 1998: Three-dimensional knee joints kinetics during golf swing. Influences of skill level and footwear. The American Journal of Sports Medicine. 26, 285-294.
[46] Szeto, G.; Straker, L.; Raine, S. 2002: A field comparison of neck and shoulder postures in symptomatic and asymptomatic office workers. Applied Ergonomics. 33, 75-84.
[47] T.B. Moeslund, E. Granum “A Survey of computer vision-based human motion capture”, Computer Vision and Image Understanding, 81, pp. 231-268 (2001).
[48] L.P. Nedel, D. Thalmann “Animation of Virtual Human Bodies Using Motion Capture Devices”.
[49] Moeslund, T.B.; Granum, E. 2001: A Survey of Computer Vision-Based Human Motion Capture, Computer Vision and Image Understanding, 81, 231-268.