NASA’s Pilot Safety Survey Yields Unreliable Data
Source: National Research Council

Data from NASA’s National Aviation Operations Monitoring Service (NAOMS) project, a survey administered to pilots in order to track safety-related incidents during air travel, should not be used to measure rates or trends in safety in the National Airspace System, says a new report from the National Research Council. Deficiencies in several aspects of the survey design make the data gathered unreliable.

The NAOMS survey — jointly managed by NASA and the Battelle Memorial Institute and given to more than 29,000 pilots from April 2001 through December 2004 — was developed as a means to statistically track rates of safety-related incidents and detect trends in those rates over time. Pilots answered questions regarding numbers of hours and flights flown, and numbers of incidents observed. Although the survey employed a number of generally accepted practices — such as computer-assisted telephone interviews and professionally trained interviewers — several flaws in the design and implementation of the study affected the usefulness of the data that were gathered.

According to the committee that wrote the report, many of the survey questions had deficiencies in structure and wording — including complex structure, multiple-part questions, and vague phrases — making them potentially difficult for respondents to digest during a telephone interview. These problems may have led to varying interpretations and judgments on the part of the pilots, thereby impacting the effectiveness of the survey at accurately measuring safety-related incidents.

Substantial fractions of the reported non-zero counts of events and reported flight legs and hours flown had implausibly large values, the committee said. Furthermore, many respondents appear to have rounded off event counts or reported flight hours to convenient numbers (e.g., numbers ending in 0 or 5), raising serious concerns regarding the accuracy and reliability of the data. Some of these problems could have been prevented by spending more time ensuring accuracy during the interview and data-entry stages, the report says.

Read full report for free online. (National Academies Press)

This entry was posted on Sunday, November 15th, 2009 at 5:59 pm and is filed under Air travel, Government and politics, Safety. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.