TNO & Royal Netherlands Navy

Testing human-machine interaction


In 1997, the Royal Netherlands Navy ordered four new Air Defence and Command Frigates (ADCFs), which were commissioned from 2001 to 2004. At the end of 2005, the Dutch navy and TNO Defence, Security, and Safety started an evaluation of the Ship Control Centre (SCC). This evaluation has two main focal points:

  1. Usability testing of the total human-machine interaction system of the SCC.
  2. Operator load assessment of the operators in the SCC.

For this evaluation, several critical scenarios were designed and tested with navy personnel on board of three different ADCFs. During one-week trips, a large amount of data was recorded to evaluate the system on several variables. The illustration below shows the set-up of the operator (left) and the manager (right). For the operator, three screens of the SCC are given at the top left.

TNO Navy Ship Bomb Sea

 

 

TNO Navy setup operator manager

Experimental set-up of the operator (left) and manager (right).

The following instruments were used to record the data:

  1. Between the three SCC screens, a small camera was installed to record a front view of the operator.
  2. Under the three screens, an eye tracker was installed to record pupil dilation and eye blinks. 
  3. The operator wore a heart rate monitoring device, to record heart and respiratory rate (the belt across the shoulder). 
  4. A wireless headset was used to record high quality audio. These audio data were recorded together with the front view video in The Observer
  5. On the ceiling, a digital camera was installed to record a rear view of the operator. 
  6. After each scenario, the participants had to do a video evaluation of the tasks just performed. They were asked to rate task complexity and effort every minute on a five point scale using a handheld device, called the Workload watch.

The right part of the illustration above shows the set-up of the manager. The only differences with the part of the operator are the absence of the eye tracker and the usage of The Observer on a laptop. The recording of the data from the heart rate monitoring device, eye tracker, and Workload watch was done externally (i.e. not in The Observer). Data was imported afterwards using the External Data Module of The Observer. All other data was recorded directly on both The Observer systems. Although some data was recorded externally, all systems were linked to one of The Observers, and The Observers were connected to each other using a LAN network. Special configuration of the system made it possible to start and stop all devices at the same time, by operation of the laptop Observer.

 

Between the three screens of the control centre, a small camera was installed to record a frontal view. In front of the operator (close to the keyboard), the eye tracker collects information about eye blinks and pupil dilation. On the right image, you see an overview of the operator and the manager performing a scenario. Both are wearing a headset to record high quality audio for voice-stress analysis and a heart rate monitoring device to record heart rate and respiratory measures (the black belt over the shoulder).

TNO Navy operating room one male
TNO Navy operating room two males
 


After recording all data on board of the ships, an extensive analysis of the data was performed. Heart, respiratory and blink rate, pupil dilation, voice-stress, and facial expression were all analyzed externally and imported into The Observer again. Scoring of the scenario (i.e. which tasks the operator and manager performed) was also done off-line. With the integration of all this information in one system, we could isolate interesting intervals during the scenarios and identify critical points in the evaluation of the human-machine interaction system. 

On the left image, you see two digital cameras record overview videos of the operator and the manager. On the right image, you see The manager evaluates a scenario he just performed. On the laptop video and sound are played. The handheld device (Workload watch) is used for input of subjective complexity and effort scores.

TNO Navy operator and cameras
TNO Navy man behind desk laptop headphones
 


Royal Netherlands Navy 

Defence Materiel Organization 

Directorate Materiel Royal Netherlands Navy 

Department of Naval Architecture and Marine Engineering 

P.O. Box 20702 

2500 ES The Hague 

The Netherlands 

www.marine.nl/index.html


TNO 

TNO Defence, Security, and Safety 

Kampweg 5 P.O. Box 23 

3769 ZG Soesterberg 

The Netherlands

www.tno.nl/