Development and Validation of the Controller Acceptance Rating Scale (CARS):
Results of Empirical Research

Katharine Lee, Karol Kerns, Randall Bone, and Monicarol Nickelson

Abstract

The measurement of operational acceptability is important for the 
development, implementation, and evolution of air traffic management 
decision support tools (DSTs). The Controller Acceptance Rating Scale 
(CARS) was created at NASA Ames Research Center for the development 
and evaluation of the Passive Final Approach Spacing Tool. CARS was 
modeled after a well-known pilot evaluation rating instrument, the 
Cooper-Harper Scale, and has since been used in the evaluation of the 
User Request Evaluation Tool, developed by MITRE's Center for 
Advanced Aviation System Development. This paper provides a 
discussion of the development of CARS and an analysis of the 
empirical data collected with CARS to examine construct validity. 
Evaluations of both DSTs showed that interrater reliability of CARS 
scores was good to excellent in terms of controller consistency and 
agreement. Subjective workload data collected in conjunction with the 
CARS show that the expected set of workload attributes was correlated 
with the CARS. The analysis also demonstrates that CARS ratings were 
sensitive to the impact of DSTs on controller operations. 
Recommendations for future CARS development and its improvement are 
also provided.