A computer failure that hobbled border-screening systems at airports across the country last August occurred after Homeland Security officials deliberately held back a security patch that would have protected the sensitive computers from a virus then sweeping the internet, according to documents obtained by Wired News.
The documents raise new questions about the $400 million US-VISIT program, a 2-year-old system aimed at securing the border from terrorists by gathering biometric information from visiting foreign nationals and comparing it against government watch lists.
The Aug. 18 computer failure led to long lines at international airports in Los Angeles, San Francisco, Miami and elsewhere, while U.S. Customs and Border Protection, or CBP, officials processed foreign visitors by hand, or in some cases used backup computers, according to contemporaneous press reports.
This article has been reproduced in a new format and may be missing content or contain faulty links. Contact wiredlabs@wired.com to report an issue.
Publicly, officials initially attributed the failure to a virus, but later reversed themselves and claimed the incident was a routine system failure.
Click here for full-sized diagram
Click here to download PDFBut two CBP reports obtained under the Freedom of Information Act show that the virulent Zotob internet worm infiltrated agency computers the day of the outage, prompting a hurried effort to patch hundreds of Windows-based US-VISIT workstations installed at nearly 300 airports, seaports and land border crossings around the country.
"When the virus problems appeared on (CBP) workstations Thursday evening, the decision was made to push the patch, immediately, to the … US-VISIT workstations. Most workstations had received the patch by midnight and US-VISIT was back in operation at all locations," reads a CBP summary of the incident.
The Department of Homeland Security's US-VISIT program office declined to comment on the documents.
Former White House cybersecurity adviser Howard Schmidt says the incident is typical of a large agency struggling with complex networks and evolving threats. "We've got catching-up to do in all areas, particularly areas having to do with national security and public safety," says Schmidt. "I hope you and I, 10 years from now, look back and say, 'Wow, I'm glad we survived that.'"
Launched in January 2004, and expanded since then, US-VISIT is a hodgepodge of older databases maintained by various government agencies, tied to a national CBP-run network of Windows 2000 Professional workstations installed at U.S. points of entry. The system has processed more than 52 million visitors, and allowed border officials to intercept more than 1,000 wanted criminals and immigration violators, according to DHS. Some US-VISIT locations are now testing gear to read new RFID-equipped passports.
While the idea of US-VISIT is universally lauded within government, the program's implementation has faced a steady barrage of criticism from congressional auditors concerned over management issues and cybersecurity problems. Last December, the DHS inspector general reported that the program might be vulnerable to hackers.
The nearly 6-year-old Windows 2000 operating system was a particularly burdensome choice on Aug. 9, when Microsoft announced a vulnerability in the software's plug-and-play feature that allowed attackers to take complete control of a computer over a network. In an unusually quick mating of vulnerability with attack, it took only four days for a virus writer to launch an internet worm, called Zotob, that spread through the security hole.
Operating somewhat more slowly, it took CBP officials until Aug. 16 — a full week after Microsoft released a patch for the hole — to start pushing the fix to CBP's Windows 2000 computers. But because of the array of peripherals hanging off of the US-VISIT workstations — fingerprint readers, digital cameras and passport scanners — they held off longer on fixing those machines, for fear that the patch itself might cause a disruption.
"The push was not made to the US-VISIT workstations during the initial install due to concerns with the possible impact of the patch on the unique workstation configurations," reads one of the CBP reports.
Officials — not unreasonably, say security experts — wanted to test the patch before installing it. But as a consequence, hundreds of computers networked to sensitive law enforcement and intelligence databases were left with a known vulnerability — a security hole rated "critical" by Microsoft because it allows attackers to take control of a machine remotely.
It wasn't until Zotob made itself at home on the CBP network Aug. 18 that the agency launched a fevered effort to secure the US-VISIT terminals, which sit on local area networks that are in turn connected to CBP's wide area network.
Even as officials raced to install the patches, the US-VISIT computers were failing at major U.S. entry points around the country, including airports in Dallas, Houston, Los Angeles, Miami, New York, San Francisco and Laredo, Texas, according to press reports at the time.
A DHS spokesman told the Associated Press the next day that a virus caused the outages. But in December, a different DHS spokesman told CNET News.com that there was no evidence that a virus was responsible, and that it was merely one of the routine "computer glitches" one expects in any complex system.
The newly released documents call that claim into question.
The government did not part with the pages lightly. After an initial FOIA request was rebuffed, Wired News filed a federal lawsuit, represented by Megan Adams, a law student at the Stanford Law School Cyberlaw Clinic. Only then did CBP release six pages of heavily redacted documents, including one page that is completely blacked out. (The lawsuit is ongoing.)
The redactions leave it unclear whether the virus itself shuttered the system, or whether the patch, or the process of installing it, contributed to the outage. For example, one sentence reads, "Initial reports confirmed that the US-VISIT workstations were (redacted) impacted" by the virus. The blacked-out portion might as easily read "severely" as "not."
Other redactions appear less tactical: A public Microsoft security bulletin is included, but with the bulletin number (MS05-039) blacked out.
Perhaps most significantly, the pages do not reveal how the Zotob virus made its way onto the private CBP network — an ominous migration that demonstrates that computers used in protecting U.S. borders are accessible, via some path, from the public internet, and could be subject to tampering.
"That machine was reachable from some network, that was connected to some other network, that was connected to the internet," says Tim Mullen, a Windows security expert and CIO of security firm AnchorIS. "There was some series of connections that manifested itself in those machines getting compromised."
A September report by the DHS inspector general found computer security at CBP wanting. In a scan of 368 devices on CBP networks, investigators identified 906 security vulnerabilities rated as medium or high risk. They criticized CBP for failing to implement a comprehensive security testing program, among other issues.
"Our vulnerability assessments identified security concerns resulting from inadequate password controls, missing critical patches, vulnerable network devices and weaknesses in configuration management," the report concludes. "These security concerns provide increased potential for unauthorized access to CBP resources and data."
In a second report in December focused on US-VISIT, the inspector general concluded that the mainframe databases at the backend of the system were generally secure. But investigators found vulnerabilities elsewhere in the system's architecture that "could compromise the confidentiality, integrity and availability of sensitive US-VISIT data."
In particular, the report found system vulnerabilities at the U.S. points of entry where the US-VISIT workstations are operating. It blames the weaknesses on poor communications between administrators in the field and those at US-VISIT's Virginia data center. In February, the Government Accountability Office — Congress' investigative arm — followed up with its own investigation of the program, faulting US-VISIT for not having an overall security plan.
Besides management issues, the system has been criticized as a slapdash effort at stringing older technology together into a modern security screening system. "Biometrics have been introduced into an antiquated computer environment," the 9/11 Commission noted of the program. "Replacement of these systems and improved biometric systems will be required."
Schmidt agrees, though he says the problem is hardly limited to US-VISIT. "We have to start moving at industry speed, not government speed, when it comes to the deployment of new technologies," says Schmidt. Instead of running Windows 2000, "I'd be racing to run the beta of the next generation of operating system … and not worry about legacy stuff that we know isn't going to be supported too much longer and has had issues."
Prior to infecting CBP, the Zotob virus reportedly caused disruptions at The New York Times, ABC and CNN's headquarters in Atlanta, as well as some offices on Capitol Hill. In late August, the FBI announced the arrest of two men in connection with the worm: 18-year-old Farid "Diabl0" Essebar in Morroco, and a 21-year-old Turkish man named Atilla Ekici, known online as "Coder."