In 1997, the Royal Netherlands Navy ordered four new Air Defence and Command Frigates (ADCFs), which were commissioned from 2001 to 2004. At the end of 2005, the Dutch navy and TNO Defence, Security, and Safety started an evaluation of the Ship Control Centre (SCC). This evaluation has two main focal points:

1. Usability testing of the total human-machine interaction system of the SCC.

2. Operator load assessment of the operators in the SCC.   

The first Air Defence and Command Frigate, Hms. De Zeven Provinciën

For this evaluation, several critical scenarios were designed and tested with navy personnel on board of three different ADCFs. During one-week trips, a large amount of data was recorded to evaluate the system on several variables. The illustration below shows the set-up of the operator (left) and the manager (right). For the operator, three screens of the SCC are given at the top left.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Experimental set-up of the operator (left) and manager (right).

The following instruments were used to record the data:

1. Between the three SCC screens, a small camera was installed to record a front view of the operator.
2. Under the three screens, an eye tracker was installed to record pupil dilation and eye blinks.
3. The operator wore a heart rate monitoring device, to record heart and respiratory rate (the belt across the shoulder).
4. A wireless headset was used to record high quality audio. These audio data were recorded together with the front view video in The Observer.
5. On the ceiling, a digital camera was installed to record a rear view of the operator.
6. After each scenario, the participants had to do a video evaluation of the tasks just performed. They were asked to rate task complexity and effort every minute on a five point scale using a handheld device, called the Workload watch.

The right part of the illustration above shows the set-up of the manager. The only differences with the part of the operator are the absence of the eye tracker and the usage of The Observer on a laptop. The recording of the data from the heart rate monitoring device, eye tracker, and Workload watch was done externally (i.e. not in The Observer). Data was imported afterwards using the External Data Module of The Observer. All other data was recorded directly on both The Observer systems. Although some data was recorded externally, all systems were linked to one of The Observers, and The Observers were connected to each other using a LAN network. Special configuration of the system made it possible to start and stop all devices at the same time, by operation of the laptop Observer.

After recording all data on board of the ships, an extensive analysis of the data was performed. Heart, respiratory and blink rate, pupil dilation, voice-stress, and facial expression were all analyzed externally and imported into The Observer again. Scoring of the scenario (i.e. which tasks the operator and manager performed) was also done off-line. With the integration of all this information in one system, we could isolate interesting intervals during the scenarios and identify critical points in the evaluation of the human-machine interaction system.  

 
Between the three screens of the control centre, a small camera was installed to record a frontal view. In front of the operator (close to the keyboard), the eye tracker collects information about eye blinks and pupil dilation.

 
Two digital cameras record overview videos of the operator and the manager.

 
The manager evaluates a scenario he just performed. On the laptop video and sound are played. The handheld device (Workload watch) is used for input of subjective complexity and effort scores.

Royal Netherlands Navy
Defence Materiel Organization
Directorate Materiel Royal Netherlands Navy
Department of Naval Architecture and Marine Engineering
P.O. Box 20702
2500 ES The Hague
The Netherlands 
URL: http://www.marine.nl/schepen/eskader/fregatten/lcf [in Dutch]

TNO
TNO Defence, Security, and Safety
Kampweg 5 P.O. Box 23
3769 ZG Soesterberg
The Netherlands
URL: http://www.tno.nl/defensie_en_veiligheid/index.xml