In a preliminary report into a recent incident where the UK’s air traffic control systems caused major flight disruptions, NATS identified the cause as a software failure on its automatic flight plan processor.
While the issue was fixed several hours later, NATS detailed in the report that the preliminary number of canceled flights was more than 1,500 on August 28, 2023.
NATS confirmed that the preliminary number of canceled flights was more than 1,500 on August 28, 2023, with some 575 flights also being delayed on the same day.
Cancellations continued to roll over into the next day.
In total, NATS saw 5,500 flights operating in the UK airspace on August 28, 2023.
Receiving the flight plan
According to the report, the NATS ATC System was operating normally, with no upgrades and no critical systems malfunctioning at the start of the day. All backup systems were also operating normally, with the company’s technical services personnel monitoring all systems according to procedure.
The sequence of events that led to the outage began prior to 4:00 am local time (UTC +1), when an unnamed airline filed a flight plan which included a portion of the flight transitioning through UK airspace.
NATS clarified that firstly, airlines enter flight plans and provide them to EUROCONTROL’s Integrated Initial Flight Plan Processing System or IFPS, which is the central Flight Planning tool for the International Civil Aviation Organization (ICAO) European Region. Subsequently, the flight plan is then transferred to all relevant Air Navigation Service Providers (ANSP), including those who will oversee the flight transiting through their respective airspaces.
In the UK, NATS receives this data and then proceeds to distribute it to relevant UK ATC units, using the Aeronautical Message Switch – United Kingdom (AMS-UK) system. For en-route operations, NATS processes these flight plans at the Swanwick Centre, where the IFPS data is converted into a format that is compatible with the UK’s National Airspace System or NAS using the Flight Plan Reception Suite Automated (FPRSA-R) Sub-system.
After it has been converted, the UK’s NAS then forwards the information to the relevant ATC officer (ATCO).
“FPRSA-R has a primary and backup system monitored both by dedicated Control and Monitoring (C&M) systems and also an aggregated central C&M system,” NATS noted in the report, adding that further resilience is provided by its systems storing four hours’ worth of previously filed flight plan data in the event that the automatic flight data plan processing fails.
As a last resort, NATS can input the flight plan data manually, which it eventually had to do on August 28, 2023, to minimize travel disruption within the UK.
Two identical waypoints
The flight plan that triggered the system malfunction was to transition through UK airspace, with the FPRSA-R processing the entry and exit points for the flight.
At 08:32 am local time (UTC +1), EUROCONTROL’s IFPS sent over the filed flight plan to NATS’ FPRSA-R sub-system, which extracts the relevant entry and exit points for UK-based ATCOs to then be able to see the flight plan data.
However, the filed flight plan included two exit waypoints “along its route that were geographically distinct but which have the same designator”, with both being outside the UK. One waypoint was at the beginning of the route, while the other was at the end, separated by 4,000 nautical miles (7,408 kilometers).
While the ICAO and other regulatory bodies have worked to remove identical waypoint names, duplicates remain.
“In order to avoid confusion latest standards state that such identical designators should be geographically widely spaced,” NATS stated in its report.
Still, the FPRSA-R software, after successfully identifying the flight plan’s UK airspace entry point, began looking for the exit waypoint. NATS noted that even if “there is no requirement for a flight plan to contain an exit waypoint from a Flight Information Region (FIR) or a country’s airspace” the software was designed to deal with such a situation and will proceed to search for the “next nearest point beyond the UK exit point”.
Since that was not present, the FPRSA-R moved to search for the next waypoint, coming across the duplicate waypoint.
“Having found an entry and exit point, with the latter being the duplicate and therefore geographically incorrect, the software could not extract a valid UK portion of flight plan between these two points,” NATS continued.
“This is the root cause of the incident,” the organization concluded, adding that this ruled out cybersecurity as a contributing cause of the meltdown.
According to NATS, it began limiting air traffic within the UK airspace starting at 11:00 am local time (UTC +1), resuming normal operations at 6:03 pm local time (UTC +1). According to the preliminary report, slowdown of traffic resulted in a total delay of 65,250 minutes that was attributed to the shutdown, with 2,000 fewer flights operating within the UK airspace, per the NATS Analytics short-term forecast.
Furthermore, due to the incident taking place during the early morning, east-bound flights from the Americas were already airborne, thus reducing their potential exposure to the shutdown of the automatic flight plan processing. Nevertheless, 4% of Transatlantic arrivals and departures were canceled on August 28, 2023.
Preventing another shutdown
The incident was the first of its kind since NATS began using the FPRSA-R software in 2018.
Following the shutdown, NATS has taken several actions to prevent such an event from happening again, with more being in progress. The actions that were taken are as follows:
- An operating instruction to allow a prompt recovery of the system with enhanced monitoring of FPRSA-R
- The addition of specific message filters to filter out any flight plans that fit the conditions that caused the incident
- Permanent change in the FPRSA-R software, which prevents critical exceptions from recurring under the conditions of the incident.
The software change was delivered to NATS on September 4, 2023, with the company set to “fully assure” the update before deploying it to its live systems.
NATS’ preliminary report only highlights key issues following the incident, with the company set to examine additional factors and design features that led to the automatic flight plan data processing system shutting down on August 28, 2023.
Furthermore, the company said that the UK Civil Aviation Authority (CAA) “will decide what further areas it would like to examine”.
“NATS would welcome any further independent oversight by the CAA,” the company added.
On September 6, 2023, the UK CAA said that it will begin an independent investigation into the incident.
“The initial report by NATS raises several important questions and as the regulator we want to make sure these are answered for passengers and industry,” Rob Bishton, the joint-interim chief executive at the CAA, said.
“If there is evidence to suggest NATS may have breached its statutory and licensing obligations we will consider whether any further action is necessary,” Bishton added.