TNO & Royal Netherlands Navy

Testing human-machine interaction

In 1997, the Royal Netherlands Navy ordered four new Air Defence and Command Frigates (ADCFs), which were commissioned from 2001 to 2004. At the end of 2005, the Dutch navy and TNO Defence, Security, and Safety started an evaluation of the Ship Control Centre (SCC). This evaluation has two main focal points:

Usability testing of the total human-machine interaction system of the SCC.
Operator load assessment of the operators in the SCC.

For this evaluation, several critical scenarios were designed and tested with navy personnel on board of three different ADCFs. During one-week trips, a large amount of data was recorded to evaluate the system on several variables. The illustration below shows the set-up of the operator (left) and the manager (right). For the operator, three screens of the SCC are given at the top left.

Experimental set-up of the operator (left) and manager (right).

The following instruments were used to record the data:

Between the three SCC screens, a small camera was installed to record a front view of the operator.
Under the three screens, an eye tracker was installed to record pupil dilation and eye blinks.
The operator wore a heart rate monitoring device, to record heart and respiratory rate (the belt across the shoulder).
A wireless headset was used to record high quality audio. These audio data were recorded together with the front view video in The Observer.
On the ceiling, a digital camera was installed to record a rear view of the operator.
After each scenario, the participants had to do a video evaluation of the tasks just performed. They were asked to rate task complexity and effort every minute on a five point scale using a handheld device, called the Workload watch.

The right part of the illustration above shows the set-up of the manager. The only differences with the part of the operator are the absence of the eye tracker and the usage of The Observer on a laptop. The recording of the data from the heart rate monitoring device, eye tracker, and Workload watch was done externally (i.e. not in The Observer). Data was imported afterwards using the External Data Module of The Observer. All other data was recorded directly on both The Observer systems. Although some data was recorded externally, all systems were linked to one of The Observers, and The Observers were connected to each other using a LAN network. Special configuration of the system made it possible to start and stop all devices at the same time, by operation of the laptop Observer.

Between the three screens of the control centre, a small camera was installed to record a frontal view. In front of the operator (close to the keyboard), the eye tracker collects information about eye blinks and pupil dilation. On the right image, you see an overview of the operator and the manager performing a scenario. Both are wearing a headset to record high quality audio for voice-stress analysis and a heart rate monitoring device to record heart rate and respiratory measures (the black belt over the shoulder).

After recording all data on board of the ships, an extensive analysis of the data was performed. Heart, respiratory and blink rate, pupil dilation, voice-stress, and facial expression were all analyzed externally and imported into The Observer again. Scoring of the scenario (i.e. which tasks the operator and manager performed) was also done off-line. With the integration of all this information in one system, we could isolate interesting intervals during the scenarios and identify critical points in the evaluation of the human-machine interaction system.

On the left image, you see two digital cameras record overview videos of the operator and the manager. On the right image, you see The manager evaluates a scenario he just performed. On the laptop video and sound are played. The handheld device (Workload watch) is used for input of subjective complexity and effort scores.