An intelligent avatar that exists in an interactive, real-time environment requires a visual persona on par with the sophistication of the processes that predict its posture, motion, and balance.  This not only means photo realistic human appearance but also realistic movement of human skin, muscles, and skeletal system.

Fortunately the entertainment industry has been working diligently on this problem for over 15 years.  This industry has expended a great deal of time, effort, and money to help develop the technology and craft necessary to create digital content that is indistinguishable from the real world - at least as seen on 75mm film.  Today, the hardware and software tools required to build, animate, and render photo realistic digital humans are very mature and can be obtained for a few thousand dollars.  As well, state of the art methodologies in digital human modeling are under constant development.  Results are freely and regularly published within internet communities interested in digital content creation and often include detailed tutorials for application-specific tool sets.  Most importantly, many of the concepts, techniques, and technologies used to create photo realistic digital humans can be leveraged for real time.

The development of any 3-dimensional model can be broken down into three parts:

(1)    Creating the Polygonal Mesh: Current limitations in real time technology require all 3-dimensional objects to be defined by a polygonal mesh.  (see beck-figure-1)

beck-figure-1

Although the polygonal mesh defines the overall shape of the subject, it lacks the visual queues required to differentiate material properties of, for example, skin from cloth. See beck-figure-2)

beck-figure-2

            (2)    Define Shaders: A shader contains data that determines what happens when simulated light strikes a surface.  For example, if an object is expected to appear as if it is made of glass, one of the attributes in that object’s shader would have to allow light to pass through its surfaces. If that object were made of solid glass, another attribute in that shader would be required to address its refractive index. Or if the glass object is not completely transparent then the object’s shader must provide some level of translucence and perhaps even color, etc. The visual queues required to convey convincing skin textures are a different type of challenge but the use of shaders to define skin characteristics is the same. (See beck-figure-3 and beck-figure-4)

beck-figure-3

beck-figure-4

            (2)    Create Joints: By themselves, three-dimensional models, shaders, and textures can be used to create photo-realistic renderings. But realistic movement of these models requires a hierarchical joint structure. (see beck-figure-5)

A hierarchical joint structure – commonly referred to as an inverse kinematics (IK) skeleton – is a series of interdependent local coordinate axes strategically positioned at locations within the three-dimensional model to provide rotational pivot points at the shoulders, elbows, knees, etc.  (See beck-figure-6)

beck-figure-6

To visually simulate the elasticity of real human skin as the joints are exercised, the amount of movement any region of the avatar’s skin has during a given joint’s rotation must be defined. This is a well-known animation technique called “skin weighting” and addresses the aesthetic issue that would otherwise cause a three-dimensional model to tear or break at the joints when rotated (see beck-figure-7 and beck-figure-8)

beck-figure-7                                                      beck-figure-8

Typically, this technique is accomplished subjectively through interactive tools – much like using a can of spray paint - that allow 8-bit gray level values to be applied to joint-specific regions of the skin. The higher the gray level value, the greater the effect a given joint has on that region.

beck-figure-9

Beck-figure-9 shows the skin weight associated with an immovable joint below the first spine joint. Note the high gray level values around the groin area, which cause the geometry in that region to be completely immoveable. Note also the decreasing gray level values above the groin area, allowing the skin geometry increasing elasticity the further it is from the groin region.

Calculating every movement made by an avatar in a real-time environment is not realistic.  Differentiation between critical movements (CM’s) and non-critical movement (NCM’s) was addressed.  For example, asking the avatar to move a 20lb object from point A to point B would represent a CM.  Waiting for the avatar to walk from its current location to where a 20lb object is located prior to picking it up *might* represent a NCM.  If the avatar’s interaction with the 20lb object is the subject of interest to the user, the avatar could simply disappear from its current location and then reappear in front of the 20lb object.  One of the goals of this project, however, is to provide the user with a certain amount of immersion into the avatar’s environment and towards this end it makes sense to have the avatar walk to the 20lb object before the CM begins.  The ability to use NCM’s – a walk-cycle or something as simple as making the avatar look as if it is breathing, for example - requires a series of avatar state awareness-related routines to be implemented in the real time environment in order for this functionality to exist.

Current issues regarding the use of NCM’s involve the practicality of creating human-like movement using traditional animation techniques versus the requirements of the mathematical models used to predict posture, motion, and path. Specifically, the mathematical model requires joints with multiple degrees of rotational freedom to be represented by component joints - each with a single degree of rotational freedom.Since skeletal joints are represented in three-dimensional space by a hierarchical structure of strategically located local coordinate axes, there is no such thing as a joint with only one degree of freedom.The mathematical model simply treats them that way by applying rotations to one, and only one axis for any given component joint. And as long as the mathematical models for posture, motion, and path prediction are always used to move the avatar, this is not a problem. However, computing all movements, all the time for any given avatar is not only impractical but also unnecessary.

Non-critical movements are created using traditional animation techniques outside the real-time environment.These techniques include motion capture, key frame animation, and inverse kinematics that make it difficult to maintain the “single degree of freedom per joint” restrictions required by the mathematical models in the real-time environment. In other words, once a NCM is implemented to move an avatar, the rotation of a component joint about the two axes not represented by its assigned single degree of freedom may no longer be equal to zero.The result of this is that the posture, motion, or path predicted will be inaccurate by an amount involving the cascading effect of all the rotations on all the axes not represented in the mathematical model of the avatar’s skeleton.This can be significant and we are currently pursuing three approaches to the problem:

            1. Compute the rotations required by the component joints for any given multiple degree of freedom joint such that an instantaneous dispersion of accumulated rotations to the single degree of freedom component joints allows the avatar to remain in its current position. This prepares the component joints for accurate posture, motion, and path prediction. For example, let’s assume that after a NCM is applied (making the avatar appear to scratch its head and look around the room) the component joints representing the right shoulder’s three degrees of freedom have rotations in x, y, and z as follows:

Shoulder-right-1 (-55, 2, -15)

Shoulder-right-2 (-9, 18, 2)

Shoulder-right-3 (-77, 5, -25)

Any computations for CM’s would now be inaccurate because the right shoulder has effectively been given nine degrees of freedom.  In order for the computations for CM’s to be accurate, the component joints for the shoulder must appear as follows:

Shoulder-right-1 (X, 0, 0)

Shoulder-right-2 (0, Y, 0)

Shoulder-right-3 (0, 0, Z)

. . . . where X, Y, and Z allow the shoulder to maintain a posture identical to the nine degree of freedom posture resulting from the use of the NCM that caused the avatar to scratch its head.

This will be the best solution but current attempts to implement this in the real time environment have not been successful.

            2.      Apply rotations to the component joints to force them into the single degree of freedom requirements of the mathematical models used to predict posture, motion, and path prior to computing CM’s.

This works but is visually confusing to the user as there is no reason for the avatar to make this (often strange looking) adjustment to posture prior to performing a given task.

            3.      Apply rotations to the component joints to force them into the single degree of freedom requirements of the mathematical models used to predict posture, motion, and path during implementation of a CM.

This works nicely visually and allows the avatar to accurately reach the destination posture but changes the predicted motion and path and is therefore not an acceptable solution for the long term.  However, this is currently the short-term solution that has been implemented, as the inaccuracies of motion and path are not as critical in the short-term if the end posture is correct.  We continue to work on solving approach (1) and expect to come up with the solution in the near future.

There are two issues that require further study of the mechanics and anatomy in relation to human movement (kinesiology).  These issues involve 1) motion in the shoulder and 2) topology of the polygonal mesh that represents our current avatar’s skin.

1.) Motion in the Shoulder: While the current mathematical model can predict postures that accurately place an end-effector (a finger for example) at a specified target location, the results can cause the shoulder to appear dislocated at the extremes of the joint limits.  (See beck-image-10 and beck-image-11)

beck-image-10                                                         beck-image11

Research indicates that the system of joints in the shoulder region of our current skeletal model does not contain enough detail to address this issue.  Details follow:

The arrangement of joints in the current skeletal model is based on a feasibility study for a 15-degree of freedom joint structure.  In the current skeletal model (see beck-figure-14 and beck-figure-15 below), the shoulder system is approximated as follows:

-         The humerus (upper arm) is anchored to, and pivots at, a point at the top of the shoulder.

-         The shoulder joint is attached to, and pivots at, a point on the clavicle (collar bone).

-         And the clavicle is simply attached to the spine.

beck-figure-14                                                         beck-figure-15

Close examination of the shoulder region in an actual human skeleton shows that:

-         The humerus (upper arm) is anchored to, and pivots at, a point on the scapula (shoulder blade).

-         The scapula is anchored to,  and pivots at, a point on the clavicle (collar bone).

-         The clavicle  is anchored to,  and pivots at, a point on the sternum (breast bone).

-         The sternum is attached to the spine via the rib cage.