Although much work has been completed with robotic and human motion prediction, human posture prediction has received relatively little attention. Yet, the ability to predict human postures accurately, promises to improve the capabilities of virtual humans that ultimately save time and money. We develop and demonstrate a new optimization-based approach to human posture prediction. This approach provides highly accurate human postures for a realistic avatar with a high number of (DOFs), and does so in real-time. This approach also incorporates the ability to consider simultaneously multiple end-effectors with shared DOFs while considering a single objective. In addition, a new method for governing the orientation of any component of a human model is developed. A relatively simple constraint is added to an optimization formulation, allowing one to stipulate a global direction for either one or two local reference-frame-axes. Multi-objective optimization (MOO) is used in two capacities: 1) to develop new superior human performance measures, and 2) to aggregate these measures. First, a new potential energy function is developed, and it is shown that human posture has little dependence on potential energy. However, minimizing delta-potential-enery does result in natural postures when the target point is behind the avatar. Consequently, we propose a modification to the idea of task-based human motion prediction: the location of a task relative to the avatar, in addition to the nature of the task, factors into human postures, and different zones around the avatar necessitate different performance measures. Next, an advanced discomfort function is created that incorporates three crucial components of musculoskeletal discomfort: 1) the tendency to move towards a comfortable position, the tendency to move only certain segments of the body sequentially, and 3) the discomfort associated with joints that are articulated near their limits. Using MOO to combine these two new performance measures, we show that the most accurate results (postures) are obtained when delta-potential-energy is coupled with discomfort but only to a small degree. We find that the weighted global criterion method is most effective in depicting the complete Pareto optimal set for these functions. Analysis of various Pareto optimal sets reveals that most of the Pareto optimal solution points represent high trade-offs between the objectives. Thus, most points are irrelevant; there is no advantage in considering multiple sets of preferences with regards to the objectives

2.1 Introduction

The advantages of computer simulation are well understood and leveraged in engineering. However, most products require human interaction, and the simulation of a prototype is complete only when coupled with a user. Consequently, development of accurate, robust human models and the ability to use these models in computer simulations promises to contribute significantly to a wide variety of design efforts, providing a key element in the design and analysis of any product that involves human interaction. In addition, such virtual humans can provide vital understanding of how musculoskeletal joints and extremities function.

Although much work has been completed with virtual humans and posture prediction, no model combines realistic appearance, real-time analysis, a high number of degrees of freedom (DOFs), and extensive autonomy. However, we present a new optimization-based approach to posture prediction, and this approach addresses the current issues with human modeling. Objective functions form the basis of this approach and characterize human performance measures, such as discomfort, potential energy, effort, etc. These objective functions then govern the posture of a skeletal system. This approach allows the model to represent realistically a human in infinitely many scenarios, rather than just a finite number of pre-recorded situations. In addition, it provides the foundation for real-time analysis of human models with significantly more DOFs than most currently available virtual humans, as discussed in Chapter 1.

Although human models have been used in engineering with varying degrees of functionality, the primary concern tends to be analysis of gross body motion and posture. Thus, the only stipulation when trying to duplicate human motion concerns the position of some part of the body, typically a finger. A body part is usually required to touch a specific target point or move between various points. In turn, the human model returns a realistic posture or motion. However, if a model is required to interact with its surroundings, then the orientation of different body parts becomes critical. For instance, in such cases it is not sufficient for a finger simply to contact a point; it must contact a point with the hand in a specific orientation necessary to accomplish a specific task such as picking up a glass or pushing a button. Thus, we incorporate in our posture-prediction approach the means to dictate the orientation of the virtual human’s extremities.

The hypothesis behind optimization-based posture prediction is that human motion concerning different tasks is governed by the different performance measures. This hypothesis can be extended to suggest that motion is governed by multiple performance measures simultaneously. Then, one must aggregate these different measures using multi-objective optimization (MOO). Thus, a thorough study of MOO within the context of posture prediction can provide insight into virtual-human models. However, although human posture prediction has been studied, methods and concepts associated with MOO have not yet been fully exploited. In fact, currently, only a basic weighted sum has been used to combine multiple performance measures, and there is little motivation for weight-selection other than trial-and-error results. Furthermore, such an approach has only been used to yield a single solution point. Studying additional methods and evaluating sets of solutions determined using a series of weighting values (determining the Pareto optimal set) can help improve posture prediction models as well as provide a practical study of optimization methods. In addition, an in-depth MOO study can help determine which function or combination of functions most realistically predicts human motion.

In addition to amalgamating multiple human performance measures, MOO can be used to create new performance measures. These measures are metrics that govern how and why a human model moves, given a particular scenario, and different measures can result in slightly different postures. However, currently, only a limited number of performance measures are available, and different components of these measures have not been viewed in terms of MOO.

In this report, we develop a new approach for accurate real-time posture prediction. A novel constraint for controlling orientation of a body part is developed. We incorporate MOO into the posture prediction problem, in two ways: 1) the development of new human performance measures, and 2) the combination of different performance measures. Concurrently, we analyze the relative performance of the MOO methods. In general, we demonstrate the advantages of incorporating MOO in posture prediction, which has never been done before to any significant degree.

2.1.1 Review of the Literature and Motivation

Generally, posture prediction methods can be classified into four categories: experimental, algebraic, geometric, and iterative inverse kinematics (IK) solutions. The experimental approach is based primarily on statistical regression equations developed from a large number of measured postures (Beck and Chaffin, 1992; Verriest et al., 1994). The advantage of such methods is that numerical iterations are not needed and joint limits are automatically satisfied. However, application is limited by the database. Furthermore, in order to produce accurate predictions, many experiments are needed based on people of different sizes. Faraway et al. (1999) uses a pseudo-inverse Jacobian to modify the posture obtained from an experimental database, to satisfy position constraints for the end-effector. This method is limited by the database of human postures, and it becomes inconvenient or even impossible for general task-based posture prediction.

Algebraic solutions are significantly faster than iterative IK solutions. However, the problem with the algebraic method is that not all kinematic chains have a closed-form solution. This is especially true for kinematic chains with more than 6 DOFs (McKerrow, 1991). Furthermore, the derivation of the closed-form equations is a lengthy process, and it is impossible to apply the algebraic method to general cases.

A geometric algorithm of inverse kinematics was proposed to predict the arm reach posture (Wang and Verriest, 1998; Wang, 1999). The main advantage of the geometric method is that the non-linear nature of the shoulder joint limit can be handled in a direct and easy way; matrix inverse calculation is avoided. However, this geometric algorithm is limited to relatively basic arm models. It is not suitable for general cases when a more complex shoulder is involved or when larger number of DOFs is used.

Generalized iterative IK methods enable any structure with an arbitrary number of DOFs to be animated with minimal human intervention. Because of this advantage, iterative IK methods have become the main approach for posture prediction. As a result, numerous reports have appeared that use this approach, but that are limited in breadth generality. For example, Goldenberg et al. (1985) reported a generalized solution to the IK problem in robotic manipulators.

The most popular iterative approach to posture prediction is based on the pseudo-inverse method used for redundant manipulator control in robotics (e.g., Nakamura, 1991; Chiaverini and Siciliano, 1991). The fundamental idea of this approach is to consider any desired joint space motion as a reference model inserted into the secondary task of an inverse kinematic control scheme. Mi (2004) briefly discusses the drawbacks such pseudo-inverse methods.

Despite the significant amount of literature concerning posture prediction, there is little development of optimization-based methods and no research concerning MOO in this capacity. In addition, optimization-based posture prediction readily lends itself to the incorporation of additional constraints that can model various features such as orientation stipulation.

2.1.1.1 Optimization-Based Posture Prediction

The literature concerning optimization-based posture prediction is relatively limited. One of the earliest and most extensive works with human posture prediction is provided by Riffard and Chedmail (1996). The DH-method is used to model a seven-DOF upper torso of a human. The optimum placement of the torso and the optimum posture of an arm are determined such that the hand reaches a given target, there are no collisions between the operator and objects in the environment, and the joint angles are within specified limits. In addition, coupling between particular joint angles and variable joint limits is modeled. This can be especially crucial with the shoulder joint. The approach satisfies the requirement that the avatar sees the target point and the requirement that the avatar is properly oriented relative to the target point. Finally, the torques resulting from an externally applied load, are considered. Equations for each of the above-mentioned components (target contact, collision, vision, orientation, and torque) are essentially combined in a weighted sum, and the weights are used as scaling factors for the different components. The final unconstrained problem is solved using simulated annealing, which is a global optimization technique. Consequently, the solution process is relatively slow (approximately two to six minutes on an Indigo II SGI).

Yu (2001) uses the same fundamental approach but with joint displacement and potential energy as objective functions for a three-DOF arm. The problem is solved using a genetic algorithm, which is also a relatively slow global optimization technique.

Mi et al. (2002b) and Mi (2004) extend the work of Yu (2001) to a 15-DOF model with similar objective functions. A real-time optimization algorithm is developed that combines genetic-algorithm results from a library of off-line computations, with results from an unconstrained gradient-based BFGS algorithm. In addition, the results are validated with the results from IKAN (Tolani et al., 2000) using a seven-DOF torso. Results are also validated using HUMOSIM (Human Motion Simulation ate the Center for Ergonomics of University of Michigan).

Although a significant amount of work has been completed concerning the concept of optimization-based posture prediction, this approach depends heavily on the objective function, and relatively little work has been conducted which implements multiple and varied human performance measures. Thus, we investigate the development and improvement of these objective functions. In addition, we provide a thorough study of how to combine some of these functions by using MOO.

2.1.1.2 Multi-objective Optimization

Despite the current work with optimization-based motion prediction and posture prediction, MOO has not been exploited thoroughly. In fact, use of MOO with human modeling in general is minimal. It involves only the weighted sum method and with no indication of weight values, no discussion of normalization even with objective functions that have significantly different orders of magnitude and physical significance, and no analysis of the results in terms of MOO theory.

With respect to robotic motion prediction, Saramago and Steffen (1998) essentially use the weighted sum method to minimize the travel time for a robot and the mechanical energy of robotic actuators. Saramago and Steffen (2000) use the weighted sum method to minimize travel time, mechanical energy, and a penalty parameter for collision avoidance, but the values for the weights are not indicated.

With respect to human posture prediction, Riffard and Chedmail (1996) string together a variety of criteria (discussed above) in a weighted sum. The weights are used as scaling factors to reduce the difference in magnitude of the different criteria. As discussed in Chapter 5, this practice does not necessarily constitute a reasonable approach to normalization. Mi et al. (2002b) mention MOO in the development of an optimization algorithm. However, although many objective functions are described, only one is used in the final formulation. Using a weighted sum, the objective function is combined with the primary constraint, not with another objective. Thus, conceptually, the result is an unconstrained optimization problem representing a penalty function approach to constrained optimization. Mi (2004) briefly mentions the use of MOO to combine discomfort, effort, and potential energy as human performance measures. However, there is no indication as to which MOO method is used.

Marler and Yang (2004), and Yang et al. (2004) present the first thorough studies of MOO for use with human posture prediction, and this study is an extension of these works.

2.1.2 Objectives of the Report

Based on the deficiencies in the current literature, the following specific objectives are pursued in this study:

1) Present an accurate real-time optimization-based approach to human posture prediction.

2) Develop a method for constraining body-segment orientation during posture prediction.

3) Use MOO to develop new human performance measures pertaining to discomfort and delta-potential-energy.

4) Use MOO to combine different human performance measures and to provide insight into the posture prediction problem. Determine to what extent delta-potential-energy should be integrated with discomfort.

5) Determine the form of the Pareto optimal set and study the trade-offs between the two performance measures.

6) Analyze the relative performance of MOO methods in terms of computational requirements and solution characteristics.

7) Evaluate the concept of basing human motion on multiple performance measures simultaneously.

2.1.3 Overview of the Report

A brief overview of the human model presented in Chapter 1 is described with an extended explanation of how the Denavit-Hartenberg method works, which is the core of the kinematic model used for representing humans. A brief overview of the fundamental concepts for MOO is presented. Then, details concerning the MOO problem formulation are described, including the development of new human performance measures. An orientation constraint is developed and incorporated in the optimization formulation. Results are given in terms of objective-function values and posture illustrations for multiple performance measures independently. Finally, MOO is used to combine the performance measures and to depict the Pareto optimal set when two of the performance measures are considered. The Pareto optimal sets are presented for four different target points and are determined using the weighted sum method, the weighted global criterion method, the weighted min-max method, and the -constraint method.

2.2 OVERVIEW OF HUMAN MODEL

The work in this chapter is based on the model for Santos^TM, which is discussed in detail, in Chapter 1. In this section, we simply provide a brief overview of that model, touching on the aspects that are relevant to this particular study.

We depict the human skeleton as a kinematic system, a series of links connected by revolute joints that represent musculoskeletal joints. In general, our approach to posture prediction entails finding the rotational displacement for these joints necessary to optimize one or more objective functions that represent human-performance measures, which govern how and why a human moves in a particular way, given a particular scenario.

In order to provide gross motion, a kinematic model for the upper body is developed, constituting the torso, spine, shoulders, and arms. represents the rotation of each joint, as shown for the general series of links in Figure 2.1.

Figure 2.1: General Kinematic Model

Each joint angle is associated with a local coordinate system. is the vector of n joint angles in an n-DOF model and represents a specific posture. is the position vector in Cartesian space that describes the location of the end-effector as a function of the joint angles, with respect to the global coordinate system. In this case, the end-effector is the tip of the index finger, and we are concerned with finding the values of the joint angles when the position of the end-effector is constrained with respect to the global coordinate system. Typically, the end-effector is required to contact a pre-determined target point, as shown in Figure 2.1. The set of all target points that the end-effector can contact, given a fixed global coordinate system, is called the reach envelope.

In this study, we consider one- and two-arm posture prediction in the context of the complete Santos^TM model. A 21-DOF model for the torso and right arm is used, is shown in Figure 2.2.

Figure 2.2: 21-DOF Kinematic Model

through represent the spine. through represent the shoulder and clavicle. through represent the right arm.

2.2.1 Denavit-Hartenberg Method

The position of the end-effector in Cartesian space , for a given set of joint angles , is determined using the Denavit-Hartenberg (DH) method (Denavit and Hartenberg, 1955). This method involves a matrix notation and approach for relating the position of a point in one coordinate system to another coordinate system, by using a unique transformation matrix. A local coordinate system and a local transformation matrix are associated with each joint, describing its configuration with respect to the previous joint and previous coordinate system. Multiple transformation matrices can be combined to determine the position of any point on the kinematic system with respect to any local coordinate system or with respect to a global coordinate system, based on all of the joint angles.

The first step with the DH-method is to embed reference frames (coordinate systems) at each DOF such that the local z-axis of each frame represents the axis of motion (rotation in our case) for the corresponding DOF. The z-axis for each reference frame, corresponding to a single degree of freedom, is shown in Figure 2.2. The x-axis of each frame must be perpendicular to both the z-axis of the current frame and the z-axis of the previous frame. Then, a transformation matrix relates position and orientation in the i^th frame to the ^th frame. This matrix is expressed in terms of the angle measured from the to about the , the distance from the to along the , the angle measured from the to about the , and the distance from the to along the . These four parameters are shown in Figure 2.3, where is the displacement (rotation) within frame-i.

Figure 2.3: DH-Parameters

Note that the values for these parameters can depend on the joint angles.

The independent transformation matrices can be multiplied to form a cumulative transformation matrix, which describes the position of a point in any reference frame with respect to any other reference frame. For instance, the cumulative transformation matrix translates coordinates of a point that are given in terms of reference frame-5, into coordinates in terms of reference frame–2. Using this idea, the global position-vector for the end-effector of the human model is given as follows:

where is the position of the end-effector with respect to the n^th frame. is actually an augmented vector written as . Note that the 0^th frame is the global reference frame (global coordinate system). In addition, although there is an n^th frame, shown as , , and in Figure 2.2, there is no joint angle associate with the n^th frame. That is, there is no (recall that is the joint angle from the -axis to the -axis).

The transformation matrix can be decomposed as follows:

As suggested earlier, this matrix transforms the coordinates of a point in reference frame-i to coordinates in terms of frame-. The matrix R is responsible for the rotation of frame-i with respect to frame-, and the vector p represents the translation of frame-i with respect to frame-, as shown in Figure 2.4.

Figure 2.4: Reference Frame Transformation

One can relate the three vectors in Figure 2.4 as follows:

Based on , R is written as follows:

The rows of R act to resolve each component of v into a specific component in frame-. For instance, the component resolves into the X-direction (determines the X-component of ), resolves v_y into the X-direction, and resolves v_z into the X-direction. The columns of R represent the direction of the axes for frame-i, in terms of frame-. For instance, the x-axis of frame-i points in the direction with respect to frame-.

2.3 OVERVIEW OF MULTI-OBJECTIVE OPTIMIZATION

The general MOO problem is posed as follows:

Find:

to minimize:

subject to:

where k is the number of objective functions, m is the number of inequality constraints, and e is the number of equality constraints. is a vector