Skip Ribbon Commands
Skip to main content
Bookmark and Share this page


Remarks to the Federal Aviation Administration Shared Vision of Aviation Safety Conference, San Diego, CA
Robert L. Sumwalt
San Diego, CA

“The Value of Data Collection and Analysis: NTSB Perspective”  

Good morning. Thank you for that kind introduction, Tom. Thank you for inviting the NTSB to help kick off this conference. I am excited to be here. Tom, you and your staff have done a wonderful job of putting this together. But, I congratulate those of you who have taken the time to attend. Over 600 people here. That is terrific!

I’ll begin by using an analogy. I envision these safety programs as vital components of an imaginary wall that we are trying to build to protect companies from bad things. When I say “bad things,” in this context, I am thinking of accidents, incidents and events with undesired outcomes.

Think about the various safety programs that are out there – programs like Internal Evaluation Programs (IEP), Line Operations Safety Audits (LOSA), Aviation Safety Action Programs (ASAP), Flight Operations Quality Assurance (FOQA), Audits, Quality Assurance, Employee Feedback, Procedures, Standards and Training, etc.

Each of these programs are like the boulders that are used to build the wall. Each is stacked on top of another and we end up with a fairly good barrier of protection. Keep in mind this mental picture, because I will refer back to it a few times during our discussion this morning.

The Safety Board has long recognized importance of aviation safety programs. I will illustrate our interest in these programs by discussing an accident involving a major air carrier, a regional air carrier, and a Part 135 operator.

Major air carrier accident

On December 1, 1974, TWA flight 514, a Boeing 727, crashed while on a non-precision approach to Runway 12 at Washington’s Dulles. The investigation reveled that there was confusion in the cockpit regarding approach procedures. The crew descended prematurely and impacted Mount Weather, about 25 miles northwest of Dulles. There were 92 fatalities

Like many other accidents, there was at least one precursor event. Oddly, six weeks earlier, another crew flying for another air carrier, made the same mistake, on the same approach to the same airport. But, in that case, the crew trapped the error and an accident did not occur.

As it turned out, some months earlier, that air carrier had implemented a Flight Safety Awareness Program, whereby crews could report, without fear of reprisal, any flight safety-related situation that they felt warranted attention. After those company pilots realized the potential confusion surrounding the “cleared for the approach” clearance, they reported the event to their FSAP and all pilots at that carrier were alerted of the potential for confusion.

And, although pilots at that airline were properly notified of the potential for confusion, there was no formalized system in place to ensure that this safety-related information was shared across all airlines and operators.

According to the Safety Board’s report of TWA 514, “… the FAA has no formal system for pilots or controllers to report unsafe conditions involving … flight procedures…” The report highlighted that witnesses at the Safety Board’s public hearing of this accident said that “… the FAA [should] establish a system to enable pilots and controllers to report operational hazards with immunity provided for the person making the report.”

Much to the FAA’s credit, before the Safety Board’s investigation of TWA flight 514 was completed, in May 1975, the FAA issued Advisory Circular AC-00-46 to form Aviation Safety Reporting Program (ASRP). Of course, the ASRP is the program under which NASA’s Aviation Safety Reporting System (ASRS) resides.

Regional air carrier accident

The Safety Board recently deliberated an accident involving a Pinnacle Airlines flight 3701. On this nighttime repositioning flight, the regional jet crew evidently decided that since it was just the two of them, they would, as they told ATC before things went sour, “… have a little fun…” Post accident analysis reveals that the crew performed a number of unauthorized actions, including intentionally causing the stall warning system to activate on three occasions, imposing dangerous sideloads on the aircraft’s tail structure by intentionally mishandling the rudder, allowing the first officer to occupy the captain’s seat while the captain sat in the first officer’s seat, along with a series of other serious errors.

The crew elected to climb to the airplane’s maximum authorized altitude of 41,000 (FL 410), despite the fact that they were flight planned for 33,000 (FL 330). The aircraft was only climbing a few hundred feet per minute, so to increase the climb rate, the crew selected a 500 foot per minute climb. As they did this, airspeed began to bleed off from the recommended climb speed. When the aircraft leveled at FL410, the aircraft was about 40 knots below the recommended climb speed. Because the aircraft was behind the power curve, despite climb power being applied, it began decelerating, leading to a high altitude stall and loss of control. The high altitude upset disrupted airflow through the engines and both flamed out. Unfortunately, the crew was unable to restart either engine and they paid for this behavior with their lives. 

A question that emerged early in the investigation was “why was the crew at 41,000?” After all, they were only flight planned to be at FL 330.

What NTSB investigators learned was that there was an unofficial “410 Club” at this airline. On repositioning flights such as this one, pilots would climb to 41,000 just so they could say they had done so. The Safety Board was curious if the airline knew of this “410 Club,” and how did the airline monitor and keep track of potential safety issues, such as crews flying at FL 410. At the time of the accident, the accident airline had no effective programs to collect and analyze safety data. They did not have a FOQA or ASAP program and they had never conducted a LOSA.

Remarkably, when asked how they ensured that crews were operating according to standard operating procedures (SOP) during repositioning flights, the company’s chief pilot stated: “Same way I do any flight being conducted to SOP. We look at the reports. We look at the numbers, you know, did they leave on time, did they not leave on time, and if anyone is on the jump seat doing a check. That’s the only way I know if any flight I have is being conducted per SOP.”
Investigators learned that the company did have a safety hotline where crewmembers could call to report safety concerns. However, investigators uncovered that no one used the hotline.

In totality, the airline had no effective means of monitoring and keeping in touch with what was going on. In essence, the airline had placed holes in their safety wall. 

The Safety Board’s investigation concluded that LOSA observations can provide operators with increased knowledge about the behavior demonstrated by pilots during line operations. The Board recommended that FAA require that all Part 121 operators incorporate into their oversight programs periodic LOSA observations and methods to address and correct findings from these observations.

Another Safety Board conclusion stemming from this accident was ASAP and FOQA programs would provide air carriers with a means to evaluate the quality of their operations. The Board recommended that the FAA strongly encourage and assist all Part 121 regional air carriers to implement an approved ASAP and FOQA program.

Those recommendations were issued in January of 2007. So, in preparation for this conference, I was curious to see how many carriers have implemented such programs. I took an informal look at the number of large carriers versus regional carriers that had ASAP and FOQA programs. And for these purposes, I considered a large carrier to be any carrier that was operating aircraft at least the size of Boeing 737, Airbus A320 or MD 717.

I found that 93 percent of 14 large carriers in my survey had ASAP programs, whereas 91 percent of the 21 regional carriers had these programs. There is amazingly very little difference here, and I applaud the industry for efforts to see that so many large and regional carriers have implemented ASAP programs. I further encourage the rest of the carriers to follow suit.

However, when I looked at FOQA programs I found quite a difference between large and regional carriers. Whereas 86 percent of the large carriers had FOQA programs, only ten percent of the regional carriers had them.

Remarkably, the regional air carrier that was involved in the above-mentioned accident is now one of only two regional carriers that have both ASAP and FOQA programs. I challenge all air carriers to incorporate FOQA programs.

Part 135 Accident

This next accident illustrates that aviation safety programs are not just for large companies or airlines. They are needed at smaller operators, as well.

On September 20, 2003, a Sundance Helicopters AS350BA crashed into the wall of the Grand Canyon while conducting air tour operations of the canyon. All seven occupants died in the crash.

The Safety Board’s investigation found that the accident pilot repeatedly flew recklessly and well outside of operator’s operations policy and FARs, flying regularly with extreme pitch and roll attitudes with passengers on board.

Here are photos taken from the accident flight by sight seeing passengers. Unfortunately, the person who took these pictures never saw them. (Note: the complete accident report, along with photos, can be found at  Safety Board investigators were able to determine that from this photo, the helicopter was being maneuvered at a 69 degree bank angle. And here is a photo where it was determined that pitch was 55 degrees nose down. And in this photo, notice the airspeed is somewhere around 145 knots, indicating a very steep descent into the canyon. Now keep in mind that this flight was being conducted with passengers on board who only wanted a good view of the Grand Canyon – not people wanting a thrill seeking ride of their lives as if they were at an amusement park.

This was not the only time the pilot operated this way, though. Here is a still shot from a video that investigators obtained from a flight with this pilot two years before this fatal crash. Notice that the helicopter was at a 99 degree bank to the left.

The accident investigation report stated: “Although Sundance had procedures in place to prevent unprofessional practices and the type of reckless behavior exhibited by the accident pilot, there was no emphasis on these procedures to ensure that the pilot adhered to them.” The Safety Board determined that the probable cause of the accident was, in part, “the pilot’s disregard of safe flying procedures and misjudgment of the helicopter’s proximity to terrain…” Contributing to the accident was “the failure of Sundance Helicopters to provide adequate surveillance of their air tour operations…”

Now, I ask: How would you like for an accident report to read this way: Contributing to the accident was the failure of YOUR COMPANY to provide adequate surveillance of YOUR operations…
How do you collect data and analyze it?  How do you keep your finger on the pulse of your operations? Are you taking proactive measures to protect your employees and passengers? Do you have multiple data sources? Data collection program such as ones discussed in this conference will help ensure that you are doing all you can to provide adequate surveillance of your operations.

Safety Management Systems

I’ll conclude by briefly mentioning Safety Management Systems (SMS). A large focus of this conference will, no doubt, be SMS, and the Safety Board has a recommendation for air carriers to establish SMS programs.

Recall from the beginning of the presentation that I envision aviation safety programs to be vital components of an imaginary wall that we are trying to build to protect companies from bad things. I said these programs are akin to the boulders used in building a wall. Each are stacked on top of another in an attempt to create a safety strong barrier. But, as we envision the imaginary wall, notice that there are gaps where the boulders don’t fit together nicely. These holes weaken our wall, which increases the chance that the wall can be compromised.

So, what could you do to strengthen the wall? You could fill-in those gaps by adding mortar. The mortar would solidify the wall and make it stronger. It would help ensure that nothing falls through the gaps, because the gaps are minimized.

Well the mortar in this example is analogous to SMS programs. We take individual safety programs (analogous to boulders) and we fill-in the gaps with mortar (analogous to SMS). When combined, we have a much more effective wall to protect our company from bad things.

When a company has SMS, they are systematically attending to those things it believes are important. SMS ensures that companies manage and value safety, just as they manage other vital business functions. For example, all companies value and manage finances. To ensure they are properly managing finance, they appoint a Chief Financial Officer (CFO), they conduct financial audits in accordance with General Accepted Accounting Practices (GAAP) and they use financial procedures, controls and have accountability. The CEO and CFO are required to file periodic Sarbanes-Oxley statements to certify that the company finance are being properly managed and reported.

By having SMS, a company employs similar methodologies to ensure that they are providing the correct attention to safety.   


At the NTSB Training Center, there is a plaque at the building’s entrance. The plaque states: “From tragedy we draw knowledge to improve the safety of us all.”

That’s what we do at the Safety Board. We take tragedy and try to learn from it so that it doesn’t happen again. But, that approach is reactive. You don’t have to wait for an accident. Your approach should be proactive – prevent the accident before it leads to tragedy. Employing aviation safety programs is an excellent way to be proactive.

I applaud your efforts. I challenge you to interact this week and participate in the discussions during this conference. And then take this valuable information back to your organizations and improve safety within your operations!

Thank you for your attention.