For the second time, Boeing and Noldus have partnered to work on the integration, testing, and validation of a simulator system in order to test effects of new and innovative Boeing developments on workload and impact on pilot behavior. During this project, the Noldus solution was integrated with the high-fidelity Apache AH D/E simulator system at Boeing.
Evaluating pilot behavior
Seated in separate crew stations, the communication between pilots is mainly verbal. Both pilots can concentrate on the systems (screens) and watch the surroundings (through the windows), but are the pilots paying attention to the right cues? In what way is their behavior affected when new technology is introduced in the cockpit? The challenge for Noldus and Boeing was to develop a system that captured the complete picture.
What signals to focus on when evaluating pilot behavior? Pilot behaviors such as pilot reaction, head up, head down, blinking, movements, tasks, and actions, and Boeing AH-64 D/E Apache activity data made available by Boeing (speed, heading, subsystems used, etc.) needed to be combined. Moreover, by integrating video and audio with Smart Eye 3D eye tracking systems, capable of tracking single-eye situations, each to be used in combination with HDU/fighter helmet in a (fixed-base) Boeing AH-64 D/E Apache simulator, Noldus empowered Boeing to incorporate a large number of signals. As a result, the organization is able to evaluate pilot behavior in great detail.
Synchronization is key
What sets Noldus apart as a partner, is the experience with creating powerful and accurate integration and synchronization of a simulator, eye tracking, video, and audio. Since separate recording devices are difficult to keep in sync, Noldus creates a connection between the systems to ensure complete accuracy.
Data output providing Boeing with valuable insights consists of:
- 2x video with gaze overlay (and audio)
- 2x videos
- Gaze events (raw and fixations)
- Blink events
- Simulator parameters
- Pupil diameter for both pilots, corrected for light conditions
- % eyes opened (fatigue indication)
- Subjective feedback from pilots (questionnaires)
- Time-stamped markers and comments
All real-time integrated in The Observer® XT. In addition to accurate synchronization, this software package allows researchers to make connections between different data modalities and detailed analyses of the frequency, duration, and order of events.
With 3D eye tracking from Smart Eye, it is easy to combine the flexibility of a head mounted system with the advantages of remote eye tracking. Why accept a head mounted system that is cumbersome to set up and might affect the behavior of the test person? Boeing and Noldus set out to create the perfect set up. With a sample rate of 60 or 120 Hz, real time data sharing and a view angle of 90 degrees (approximately 42” screen at 27” with 2 cameras) - 360 degrees (with 8 cameras), Smart Eye offers revolutionary technology. Since Noldus core software packages support the integration and combined data analysis of third party systems, Noldus is the best partner to deliver integrated systems capable of multimodal analysis. This offers insights into complex human factors processes that cannot be obtained by analyzing single data modalities.
The Noldus approach
All A-class components are integrated and tested completely matching user demands and wishes. To ensure the best results for Boeing, the project is subdivided into four phases: requirements analysis, design, development, and implementation.
- After the first project (Embedded Measures Lab), experts from Noldus matched Boeing’s requirements to another solution from Noldus. The experts translated the questions into technical system requirements. This was done through on-site visits and interviews.
- In the design phase, the technical requirements were transformed into actual system designs: data flows, technical components, and wiring schemes.
- In the development phase, Noldus’ technicians (almost the same team as the first project) configured the systems and built additional functionality. The set up was tested in-house at Noldus HQ, where technical expertise is concentrated.
- Noldus’ consultants made sure that the lab was installed correctly, on-site of course, and, a second visit is upcoming.
Working in this structured way has many advantages: It ensured that the system met Boeing’s requirements and that it was properly tested and implemented. After the implementation phase, Noldus’ role is not over. We’re in close contact with Boeing to keep aiming for the best possible solutions.
Embedded Measures Lab
In another project, Boeing and Noldus Information Technology together designed a mobile human factors lab. This Embedded Measures lab is used for measurements of situational awareness and fatigue, along with prototype testing. Working in a structured way ensured that the system met Boeing’s requirements and that it was properly tested and implemented. After the implementation phase, Noldus’ role was not over. There is technical support, delivered by the local office in Tacoma, WA.