Virtual Reality for Santos

During the 1950’s the US government began to explore the use of computers to visualize data while designing a new radar system. This visualization was slowly developed into flight simulators and telerobotics. During the 1970’s Hollywood began to get involved with computer graphics while creating special effects for movies. Later, towards the end of the 80s, computers were becoming small and cheap enough that people were beginning to buy them for personal use, which eventually lead to the booming of the gaming industry. The developments in affordable high performance computers, Hollywood special effects, computer gaming, and simulators came together in the early 1990’s to create a lot of excitement for virtual reality. Throughout the 90’s the technologies associates with virtual reality were rapidly evolved. The use of virtual reality had become feasible for many private companies. Today, nearly every large company uses virtual reality for some form of product development. Because of this, Santos™ is being specifically designed with the flexibility to not only work on a standard desktop computer but to also work within virtual reality. But what is virtual reality? It really depends on whom you ask. There appears to be no standard definition of what virtual reality is. This chapter will discuss what the Virtual Soldier Research (VSR) defines as virtual reality, current available solutions, and methods being explored for interacting with Santos™ in virtual reality. Then this chapter will cover the current virtual reality systems available at the Virtual Soldier Research lab and their hardware design. Finally, an argument for the use of virtual reality will be presented along with some example of implementation.

6.2 What is Virtual Reality

The “true” definition of virtual reality is quite fuzzy and hotly debated. Some will argue that a painting can be considered virtual reality because it represents a reality that is not really there. Others will say that virtual reality only exists in a completely immersive environment that stimulates many of our depth cues, such as surround sound, motion parallax, stereoscopic disparity, and complete field of view. Some will argue that simulators and telerobotics are forms of virtual reality while others will claim that these should be categorized separately. The purpose of this section is not to strictly define virtual reality but to define what is implied when Virtual Soldier Research refers to virtual reality. Virtual reality at VSR centers on an immersive environment that provides a wide field of view, stereoscopic display, 3D sound, the ability to track the user and objects in the real world, and allowing the user to interact with the virtual environment in real-time.

6.3 Current Available Technologies

Currently there are several available technologies that can be used to create a virtual environment. The previous section mentioned five specific elements that make up virtual reality at VSR. The technologies associated with each of these elements will be discussed in this section.

6.3.1 Field of View

Wide field of view is an important characteristic for immersive environments. A typical human has a total horizontal field of view between 160 and 208 degrees and a binocular field of view between 120 and 180 degrees. Since the FOV is so wide, it is impossible to completely fill the users vision with a single flat screen. Typically, a large flat screen is used to give the user the maximum field of view. Adding more screens around the user is often used to create a more immersive environment. It should be noted

Figure 1: Example of a 4-Wall Visualization Environment

that each screen requires it’s own projector and sometimes it’s own computer. It is typically most desirable to add a left and a right screen to fill the users peripheral vision. If the resources are available, a screen on the floor is added next. Adding a screen to the floor presents several unique problems. For instance, how the image is projected onto the floor. The easiest solution is the project the image down onto a smooth floor. The image needs to projected such that the users shadow is cast on the floor behind them. This setup will not work well if there will be more than one user in the visualization environment because of the shadows or if the visualization environment will have a ceiling. The other option is to have the floor image projected from underneath the floor. This requires that the entire visualization environment be raised so that a project or a mirror can be positioned under it to send an image up to the floor. Also, the floor must be made of a material that is strong enough for people to stand on, tough enough that it will not get scuffed up by people walking on it, and clear enough that a rear projected image will be seen by the user. After the floor has been added, the ceiling and the rear wall are usually added complete the cube. Having the user in a complete six wall cave is very visually immersive because it gives them the freedom to look in any direction. Another solutions that has been applied to fill the users field of view is to use a concave screen, as can be seen in many planetariums, simulators, and in some IMAX theaters.

Figure 2: a) IMAX Dome b) Example of a Concave Visualization Environment

Head mounted displays are sometimes used to show an images to the user. It should be noted that these displays usually only provide a FOV of about 30 degrees. Head mounted displays typically are coupled with a tracking device that monitors the position of the user’s head and updates the scene being displayed to the user based on where the user is looking and their movement. It is very important to have a fast and accurate method of tracking the head because the scene being displayed to the user must

Figure 3: Example of a Head Mounted Display

appear to be changing simultaneously with their head movement. Even a slight delay or inaccuracy will ruin the credibility of the environment and has the potential of making the user become sick. Another issue with head mounted displays is the quality of the optics and the brightness of the LCD for the display . Unfortunately, high quality optics and bright LCDs are physically heavy which make wearing the device uncomfortable after even a short amount of time. This has been avoided by mounting a head mounted display on a boom. Not only does this take the weight off the users head and neck but it also provides a very accurate and fast tracking system through the mechanical links of the boom.

Figure 4: Boom Mounted Display

6.3.2 Stereoscopic Display

The second element of virtual reality mentioned in the previous section is stereoscopic display. Stereoscopic display refers the ability to display one image to the left eye and a slightly different image to the right eye, thus simulating real life binocular disparity. Binocular disparity is the result of the eyes seeing the world from two slightly different perspectives. By comparing these two perspectives, the brain can better understand relative distance to various objects. For example, the closer an object is to the eyes the greater the difference between how the left eye sees the object and how the right eye sees the object.

Figure 5: View of a teapot as seen by the left eye, right eye, and both eyes in stereo

The crux of stereoscopic displays is how to display different images to each eye. This is fairly straightforward with a head mounted display, since the displays are so close to the eyes and each eye sees a different display device. When the display is on a large screen and to many users the solution usually requires the users to wear special glasses. These glasses either shutter or have special filters so that the eye only sees one of the

Figure 6: Examples of Stereo Glasses. From left to right: Blue/Red Color Anaglyph, Linear Polarized Glasses, and Shutter Glasses

images being displayed on the screen. A classic and cheap way of doing this is to have the user wear glasses with a red filter over one eye and a blue filter over the other. Then the two images are projected onto the screen at the same time with one of the projected images being predominately red and the other image being predominately blue. When the pair of images reach the glasses, one image is filtered out and the other image is allowed to pass through, thus allowing each eye to see its respective image. While this method may be cheap and easy, it does have some significant drawbacks. One is that objects can only be displayed in red and blue. Another is that the color filtering does not completely filter out the undesired image, resulting in ghosting.

Figure 7: A Blue/Red Anaglyph Picture

A similar method to this color anaglyph is to use polarized filters to create stereoscopic displays. This is achieved by using one projector to display the left eye image and a separate projector to display the right eye image. Each of the projected Text Box: Figure 8: Projectors for Polarized Display

images are sent through a filter that polarized the image to a specific angle before it hits the screen. The user is then required to wear special polarized glasses in which the lenses are made as to only allow light with a specific angle of polarization to pass through. Therefore, if a user were to look at the screen without the glasses, they would see two different images overlaid on each other. When they put on the polarized glasses, the image will be filtered

Figure 9: Polarized Stereoscopic Display

such that the right eye will see one of the images and the left eye will see the other. If the simulation is created correctly, the brain will interpret these two images as one image with depth. A polarized stereoscopic display is widely used today because it is very affordable and provides very nice images. However, there are some features that one must consider when designing this type of system. Depending on the quality of the polarization filters used, some ghosting may appear. Also, a special screen must be used which will maintain the angle of polarization of each image when it is presented to the user. Finally, most systems use linear polarization, which will fail if the user tilts their head too far to one side or the other. This can be avoided if circular polarizers are used but these tend to be more specialized and expensive. It should be noted that the type of polarization on the glasses must be the same as the type of polarization used on the filter.

The systems that have been discussed so far are all referred to as passive systems. There is another class of systems that are designated as active systems. These systems only require one projector and the user must wear a special set of glasses called shutter glasses. The glasses are designed so that the lenses alternately blink, blocking the view of one eye. In other words, the left eye lens blinks shut while the right eye blinks open, then the right eye blinks shut and the left eye opens. Typically this blinking is

Figure 10: Active Shutter Glasses Stereoscopic Display

accomplished with liquid crystals lenses, which can be switched between transparent and opaque quickly with electronic stimulation. The blinking is usually done at such a high frequency that the user does not even notice it is happening. This is then synchronized with the refresh rate of the display device via either a wire or infrared signals so that every time the screen refreshes the lenses flip. Each time the display device refreshes it alternates between showing the left image and the right image. Therefore, when the left image is displayed the right eye is blinked shut. When the display refreshes the right image is displayed, the left eye blinks shut, and the right eye blinks open. The key to a quality active system is the refresh rate of the display. Studies have shown that the human eye typically becomes insensitive to flickering above 50Hz. Since the glasses blink in synch with the refresh rate of the display, each eye will be subjected to a flicker that is half the refresh rate of the display. Therefore, to have a flicker-free display the refresh rate must be at least 100Hz. 120Hz is usually considered desirable for an active sysem. This high frequency makes the display devices quite a bit more expensive. Typically, a standard monitor or projector only display with a maximum refresh of 85Hz. Among all the stereoscopic display systems that are widely available today, the active stereo provides the best images; there is no ghosting or loss of color. Historically, the brightness of the display has been a problem but with the new DLP projectors that support 120Hz refresh rate this has become less of an issue, provided the projectors can be afforded.

In important element in presenting a stereoscopic scene is the ability to create a left eye image and a right eye image. The “gaming” way of doing this is to simply shift the pixels on the screen left or right a specified distance. While this provides an effect of sorts, it does not provide good depth cues because all the objects have equal separation. In reality, the separation should change depending on the objects relative depth. Most VR visualization packages provide a more accurate method of separating the eyes by allowing the user to input two parameters, the intraocular distance and the zero parallax frame. The intraocular distance is simply the distance between the eyes. The zero parallax frame is usually a plane that is set a fixed distance from the eyes where there is zero separation. The further an object is from that plane, the more separation is should have. The graphics card then calculates the actual left and right eye and alternates

Figure 11: Stereo Separation. The left image shows stereo separation by equally shifting all the pixels of the display to one side. The right image shows true separation with the zero parallax distance set on the cone.

sending these images out to the display device. The current method of choice for rendering is using openGL with Quadbuffer. If the video output is being sent to an active system then all the images will be sent to one projector. If a passive system is being used, the left image and the right image need to be separated into two separate video signals, with each signal going to a different projector. This separation is either completed with a special box external to the computer or by the graphics card itself.

6.3.3 3D Sound

3D sound is also another important characteristic in an immersive virtual reality. This is accomplished by embedding sound on an object within the scene. As the user moves around object, the sound moves appropriately. This is supported by most VR development software and is not very difficult. The only hardware that is needed is a good audio card that supports Dolby Digital 5.1 and a set of speakers placed around the visualization environment.

6.3.4 Tracking

Tracking the user in the visualization environment is needed not only to allow the user to interact with the virtual environment but to also allow the virtual environment the “see” the user. There are many trackers available on the market today. The most common trackers use electromagnetics, lasers, or optics to track an object.

The electromagnetic tracker is composed of a transmitter and a number of receivers. The transmitter contains a magnetic coil that emits a magnetic field. The receivers each contain electromagnetic coils that detect the magnetic field created by the transmitter. Both the transmitter and the receivers are connected to a box, which can decipher the position and orientation of each receiver based on how the receiver is reading the magnetic field. The main benefit of this system is that it does not require line of sight between the transmitter and the receiver so occlusion is not a problem. It does have problems working in areas with ferrous metals and electromagnetic fields, as these tend to warp the magnetic field created by the transmitter.

Figure 12: Electromagnetic Tracking System

Laser trackers typically use a pair of lasers that are mounted separated from each other in a scanning unit. These lasers both continually sweep through the same volume in front of the scanning unit. A sensor, which is usually plugged into the scanning unit via a long cord, is then placed in the swept volume. On this sensor are three diodes that are sensitive to the lasers. When one of the lasers shines on a diode, it sends a signal back to the scanning unit. By knowing where each laser is scanning at each moment and getting feedback from the sensor about when a laser shines on it each of the diodes, it can resolve the position and orientation of the sensor. The benefit of a laser tracker is that it is extremely accurate but it requires that all the diodes be visible to the scanning unit. These units also tend to be very expensive, can track only one object and have a small envelope of trackable area.

Figure 13: Laser Tracking System

Optical trackers use a series of cameras to look at the object from various angles. The object is equipped with markers that are either small diodes that emit light (usually red or infrared) or small reflective balls that reflect light that is shined onto the object. The cameras, which are attached to a computer, are designed to pick up a certain wavelength of light. The computer can then compare the location that each camera records the light coming from and, by knowing the location of each camera, determine the location of the marker. By adding a second marker to the object, the orientation of the object can be determined. By adding more cameras the tracking volume can be increased, the accuracy is increased, and problems associated with occlusion are reduced. Another nice feature of optical tracking is the ability to track many different objects. By clustering markers in unique arrangements on different objects, the computer can identify the unique arrangements and know what object or body it is tracking. Also, if reflective markers are used there are no wires attached to the user, allowing for greater freedom of movement.

Figure 14: Optical Tracking System

6.3.5 Real Time Interaction

The final ingredient mentioned that is required for virtual reality is real time interaction between the virtual environment and the user. Real time means that Santos™ immediately reacts to the users input. The traditional methods of interacting with a computer are obviously with a keyboard and a mouse. These tools work great while working at a desk with a 2D interface on the computer but they become quite awkward when trying to interact with a 3D environment. There are many new gadgets that have been introduced which allow the user to interact in 3D such as a gyroscopic mouse, SpaceBall, Wanda, P5 Glove, and Pinch Gloves. Most of these devices work in

Figure 15: Interactive Devices. From left to right: Pinch Gloves, P5 Glove, SpaceBall, and Wanda

conjunction with a tracking device and are highly programmable to perform user specified tasks. Still, many find these devices onerous for tasks that require anything but simple interaction. VSR is exploring the use of existing technologies to make the flow of information between Santos™ and the user more natural. The next section will discuss interacting with Santos™ in an immersive virtual environment.

6.4 Interacting with Santos™ in an Immersive Environment

In general, interaction can be thought of as information being traded between two entities. If this information is allowed to flow freely both ways, the level of interaction is very high. Over the past 20 years humans have become adept at interacting with a computer through a keyboard and mouse. For many people, the flow of information in and out of the computer requires very little encoding or decoding, thus becoming somewhat natural to the user. This interaction is preferred when the user is sitting at a desk in front of a monitor. However, Santos™ is being designed to not only work on a desktop computer but to also be able to live in a virtual environment. So, while the user will still have the option of interacting with Santos™ at the desktop, VSR believes that the future of Santos™ will be interaction in real time virtual reality because the tasks that Santos™ will perform are more easily observed and evaluated in a 3D environment. Still, for this paradigm to work, an interface needs to be designed that allows as close to natural communication between Santos™ and the user. Currently there is no such interface. If the user is standing in a visualization environment observing Santos™ in virtual reality, a keyboard and mouse would be quite awkward to work with. This section will briefly discuss two types of interactions that VSR is hoping to incorporate into an immersive environment that will facilitate a natural interaction with Santos™. These interactions are WiFi connection and vocal communication. It should be noted that gesture recognition is also being explored but this will not be discussed in this section because it was covered in a previous chapter.

6.4.1 Wireless Interaction

The 2D interaction that is used to interact with a desktop computer generally does not work well in virtual reality. Displaying and using menus, buttons, and text is quite difficult because the resolution of the image being displayed on a large screen requires the text to be large, which makes buttons huge and the menus takes up a lot of valuable space. Also the stereoscopic separation of the scene makes it hard to read text and locate buttons with a 2D pointer, like a mouse. However, reading text and pushing buttons are vital to working with a computer program and a user with a 2D display most easily accomplishes these tasks. It was decided that Santos™ needed to have the ability to interact with a 2D display while in virtual reality. Therefore, he was equipped with the ability to send and receive TCP/IP data over a WiFi connection. WiFi has become the wireless standard that is used for most wireless connections. Laptops, Pocket PC’s and Smartphones all have some models equipped with WiFi technology. Many universities, libraries, businesses and even coffee shops have installed a wireless network that supports WiFi devices. Santos™ has been designed to listen to a specific port on his host computer. Any device that can call this port and send TCP messages has the ability to

Figure 16: PDA Interaction with Santos™

communicate with Santos™. Essentially, a person sitting in a Seattle coffee shop that is equipped with a WiFi network can use their PDA to call up Santos™ in Iowa City, ask him to do something, and get information back. While this specific scenario may not be

Figure 17: Wireless Communication with Santos™

particularly useful, it is a good example of the flexibility provided by this technology. What is useful is the ability to program many different devices to act as a 2D interface to Santos™. For example, a user standing in a 6-walled visualization environment can utilize a PDA to enter an exact target location for Santos™ to try to touch. After Santos™ touches the object, he can send data back the users PDA regarding his comfort level and joint angles. Perhaps the user is not satisfied with the angle of the elbow, so on the PDA they change the limits of rotation and ask Santos™ do touch the target again. While the data being transferred between Santos™ and the user is easy to do with a 2D interaction, imagine how awkward it would be if the user tried this same task but with buttons and menus that were in the 3D environment. Entering the target position and joint limits would be extremely difficult unless the user was expected to carry around a keyboard.

6.4.2 Vocal Interaction

Vocal communication is another form of interaction being investigated. With this type of interaction not only would the user be able to talk directly to Santos™, but also Santos™ would be able to respond verbally. This would require a device that could record messages that the user speaks, encode it into something that Santos™ can understand, and send the message to him. This will be accomplished by leveraging existing voice recognition programs to record the commands and change it into a text string. This string can then be analyzed for commands and sent to Santos™ via the TCP connection mentioned above. Incorporating current technology used by the gaming industry to make characters talk can then be used to create Santo’s™ reply. Imagine being able to simply say “Santos™, touch the chair”. When the voice recognition software hears the word “Santos™” it knows a commands is being issued, which is encoded and analyzed to drop out the word “the”, resulting in the string of commands “touch chair” which are sent to Santos™ who understands the command “touch” and knows the object “chair”. If he didn’t know what “chair” was, he may ask for the user to identify what a chair is. The user can then point to the chair and, with the use of a tracking system, Santos™ would learn what the object “chair” is.

6.5 Current Virtual Reality Systems at VSR

Currently VSR has 3 VR systems and another being constructed. This section will discuss these four systems and the hardware associated with them.

6.5.1 Three Wall Active System

The oldest system at VSR is a three wall active system that will soon be retired/replaced with the new system. This system was created about three years ago when the Digital Humans Lab first became interested in virtual reality. This system is

Figure 18: 3 Wall Active System

powered by a cluster of three Windows based PCs equipped with Wildcat graphic cards. Each PC is attached to a powerful CRT projector and is responsible for the display on one of the walls. Each projector is capable of refreshing at 120Hz so that the users cannot see flicker of the shutter glasses. The three walls, which measure 8 feet by 10 feet each, are made of ¾ inch thick acrylic and are specially treated to amplify the brightness of the projected image. This visualization environment is equipped with pinch gloves, a gyroscopic mouse, and an electromagnetic tracking device capable of tracking up to four receivers. Since this is an active system all the users are required to wear shutter glasses, which are synchronized to the projectors with infrared emitters.

6.5.2 Single Wall Passive System

Another system at VSR is a rear projected single wall passive system. A single Windows based PC equipped with an nVIDIA graphics card controls this system. The

Figure 19: Single Wall Passive System

graphics card has two heads and is responsible for splitting the left image from the right image. These video feeds are sent to two SXGA LCD projectors that are mounted on top of each other and lined up so that when they both display the same image on the screen simultaneously the images overlay each other perfectly. Each projector is fitted with a polarizing filter that polarizes each of projected images in a unique way. Both images are project onto a special 8 foot by 10-foot screen that is designed for rear projected polarized light. Since this is a passive system, the users are required to wear polarized glasses. This is a rear-projected system so the user is allowed to stand right next to the screen. The area in front of the screen is also tracked with an optical tracking unit using infrared light The user is able to interact through the tracking system and/or by using a gyroscopic mouse.

6.5.3 Portable Passive System

A portable virtual reality system has also been created at VSR. This system allows Santos™ to be taken on the road and easily displayed. This system is also a passive system that can be controlled by either a standard PC or a laptop. If the video card of the computer allows two video outputs and has the ability to split the left image and right image then it is in charge of separating the video feeds. Otherwise a separate box is used to split a single quadbuffer video feed into a left and right eye video feed. These video feeds are sent to two small XGA DLP projectors that are securely mounted in a portable box. These projectors are fitted with polarizing lenses. Both projectors shine onto a portable 8x10 screen specially designed for front projected polarized display. Again, since the system is polarized, the users will be required to view it with polarized glasses. Since this system is portable and designed to be set up quickly, it does not have any trackers with it. Interaction is achieved with a mouse and keyboard.

Figure 20: Portable Passive System

6.5.4 Six Wall Active System

The final system is still being constructed. This system is a 6-wall visualization environment with active stereo. This system will provide a complete immersive environment for the user. It will be controlled by six PC based computers all connected on a LAN with each computer responsible for one wall. The images will be projected with state of the art DLP projectors capable of refreshing at 120Hz. The inside of the cube will be tracked with either an electromagnetic or optical tracking system. It will also be equipped with pinch gloves. This system is scheduled to be operational by the beginning of 2005.

Figure 21:Plans for the 6 Wall Active System

6.6 Why Use Virtual Reality

The driving force for the creation of Santos™ is the reduction of time and cost that a company must invest while prototyping a new concept. Currently there are many software packages available that allow designers to predict such things as durability, fluid flow, and dynamic responses of parts and systems before they are ever created, which significantly reduces the resources needed for prototyping. VSR is striving to add another level of prediction by introducing a digital human that can inspect systems and concepts in the same way a real human could if they were presented with the same thing in real life. In order for this to work, Santos™ will need to be modeled as close to a person as possible. He will need to move like a human, interact like with his environment like a person interacts with the real environment, and interact with the user as another person would. This type of interaction requires information to flow between the user and Santos™ in a manner that is similar to the way information is traded between people, i.e. real time interaction through both visual and vocal means. This interaction can only be achieved through virtual reality. The user must be able to easily observe Santos™ in an immersive environment and move around as he works to see him from many different angles. The user must be tracked so that Santos™ can see the user and understand gestures and indications that the user may use, such as pointing at an object. There must be vocal communication between Santos™ and the user.

6.7 Examples of Implementation

In this final section two scenarios will be presented to illustrate VSR’s vision for the future of Santos™ and his role in industry. The first scenario will describe an immediate future implementation of Santos™. This is what we expect Santos™ to be able to complete in less than a year. The second scenario will describe an example of how Santos™ can be used within a few years.

6.7.1 Immediate Future Implementation

A leading manufacture of construction equipment is in the process of designing a new dump truck designed specifically for mining. They are concerned about some the comfort of the cab and some maintenance issues. Due to deadlines, the cab is analyzed first. The cab was modeled in Pro/E so the model with all the important parts was imported into Real-Time Simulation code, where Santos™ is currently living. Santos™ is manually seated in the drivers seat and asked to touch various buttons in the cab Other than sitting him in the seat, no direct manipulations of Santo’s™ joints are needed. The user is able to quickly indicate to Santos™ where he should touch, how his hand should be orientated, and what type of grasp he should be using through the use of a target hand. Once Santos™ understands these three parameters, he calculates in real time the motion needed to move and reports back several parameters such as comfort level, metabolic information, individual muscle forces, and joint torques needed for a wide selection of body types. Next, Santos™ is asked to manipulate a lever that appears to be in a slightly awkward position. Dynamic analysis had been completed on the lever and it was found that for the lever to be engaged, it needed to have 5 N of force applied along a specific direction. The target hand is placed on the lever so Santos™ understands where his hand needs to go and the user indicates the force needed to manipulate the lever. Santos™ performs the task and reports back various parameters, which suggest that the position of the lever is fine for touching but the angle of the force doesn’t allow the operator to use an efficient set of muscles to engage the lever. The user then simply changes the direction of the force, Santos™ tries it again and reports back that it is much more comfortable to push the lever in that direction.

Next the user wants to test some maintenance issues regarding the fuel pump, which is buried deep in the engine compartment. The designer is concerned with not only if the mechanic can get their hand in there but weather they will be able to turn the wrench with enough force. They position Santos™ in the general location that the mechanic would stand and indicate how Santo’s™ would grab the wrench. They also indicate the amount of force needed to turn the wrench. Santos™ then calculates the joint angles needed to get his hand on the wrench without intersecting any of the CAD geometry and then optimizes it to apply the most torque to the wrench. Since all this is taking place in a virtual reality environment, the user is allowed to change the opacity of the CAD objects inside the engine compartment and easily inspect Santo’s™ arm position. Santos™ reports back the muscle forces that are needed to make the required amount of torque. After looking at these values, the user decided that this requires more muscle force than many mechanics will be able to generate. After inspecting the posture Santos™ chose and looking at the surrounding geometry, the user decides to try removing a part that looks to be significantly restricting his movements. By simply deactivating that part and asking Santos™ to reexamine the movement, a new report is generated which shows more of the torque can be generated with the bicep, which the user feels that most mechanics will be comfortable with.

6.7.2 Near Future Implementation

A company is designing a new assembly line machine that will weld several large pieces of steel together. This process involves placing four various shapes of steel into a jig, tightening four bolts on the jig to secure the pieces together, then going over to a console to push an activate button. When this button is pressed, a safety gate closes over the welding area and the operator is asked to confirm the jig pattern being used. The four steel parts will arrive on various conveyor belts. The designers have enlisted Santos™ to determine several factors about the system. Some of the things they want Santos™ to tell them are if one person is able to handle the steel parts without lifting mechanisms, how long a single person will be able to perform this task, how long the user should be expected to perform the task the they are to work a standard shift, and, by observing Santo’s™ movements while performing the tasks, if there are any safety issues that should be addressed. First, the basic machine, steel parts and operators station is modeled in Pro/E. Next these items are brought into the Santos™ environment, given physical properties (if applicable) and Santos™ is explained the task. This is accomplished by placing the four steel objects at their final position in the jig and telling Santos™ “move to here”. Now, these objects can be moved to random places in the simulation and when Santos™ is asked to start working, he will find them and place them in the proper place. Loading the jig is identified as the first task. The next task is tightening the bolts on the jig. Santos™ has already been taught how to tighten bolts so to describe this task the user simply identifies each of the bolts, how many turns are needed, and the wrench that he needs use. The third task is for Santos™ to walk over and activate the machine by pressing a button to close the safety gate and another button to confirm the jig number. Since Santos™ already understands how to press buttons, this task is explained to him by simply identifying the buttons. When the welding process is over, the machine automatically unloads the finished part and the process is ready to start again. Now that Santos™ understands what he needs to do, the user can start the analysis. The four steel pieces are places in various locations and the process starts. Since Santos™ is aware of his environment, he can locate the parts, walk over to each one (avoiding any obstacles that may be in the way), determine the best way to pick it up, carry it back to the jig, and load it in. After loading all four items he locates the wrench and uses it to tighten the four bolts. Finally, he walks over to the buttons and presses them in sequence, completing all the tasks. Santos™ is now able to report back how much energy was used, how many times he could complete the task before fatigue became an issue, and if he could comfortably complete the tasks. By observing Santos™ working, the user also noted that one of the bolts on the jig was awkward to reach and that the operator would be able to push the button to activate the safety gate while their arm was still in the welding area. The designer was also able to try placing the four steel pieces and the wrench in various locations to see how this affected the effort of the operator.