The first sign of trouble on the New York Stock Exchange was a color — a sickly yellow.
On the hand-held computers on the cavernous trading floor, that color meant one thing: the Big Board was down.
What began Wednesday morning with a seemingly workaday software glitch soon escalated into one of the most startling computer outages in Wall Street history — and, for the Big Board, a race against the clock.
At the 9:30 a.m. opening bell, traders’ orders for some stocks weren’t reaching the proper destinations for processing. Techies were frantic to fix the problem. At about 9:32 a.m., they succeeded.
Two hours later, boom.
One floor trader started shouting, “My handheld’s down! My handheld’s not working!” He and other traders hurried over to a ramp on the trading floor where NYSE executives usually meet with them to explain any problems. Not today. Three hours later, still nothing. Everyone was just standing around.
“The order flow wasn’t being entered into the display books on the trading floor,” said Pete Costa, president of Empire Executions Inc. who’s worked at the exchange in downtown Manhattan for 34 years. “As soon as that happened, the exchange shut down to understand what was going on.”
Every computer screen “went this pukey, canary yellow color,” Costa said. “That means the stock has stopped trading.”
Seven thousand orders were sent but never executed, he said. The system had to be rebooted. That took about 45 minutes.
The big concern was getting the exchange up and running as soon as possible. The 4 p.m. closing bell loomed. That’s when the NYSE sets stock prices that indexes and mutual funds use to calculate their values.
“My initial reaction was, ‘That’s OK, I hope they can reopen for the close,’” said Jamie Selway, head of the electronic brokerage at Investment Technology Group Inc. in New York. At the open and before the close, orders can be routed to other exchanges, he said. But at the close, the world needs the NYSE.
The traders at Cambridge Global Payments, a global foreign- exchange and payments provider in Toronto, “kind of paused” when the NYSE went down, said Karl Schamotta, the firm’s director of foreign-exchange research and strategy.
“The trading floor took a moment to collect its thoughts before we moved forward,” he said. Because of the meltdown in China’s stock markets and concern over a possible Greek exit from the euro zone, “we’re sitting in a situation where traders have their fingers on the trigger globally, so there’s certainly a fear complex.”
There was little time for exchange officials to pause. Art Cashin Jr., a managing director for UBS Financial Services Inc. and one of six executive floor governors at the NYSE, answered his phone a few minutes before 2 p.m. and said, breathlessly, “We’re dealing with the glitch. We’re rushing to try to get it open.” And hung up.
The exchange was back in business by 3:10 p.m.
In Greenwich, Connecticut, Marc Pfeffer had two worries about the shutdown.
The first was that it was due to hacking. That concern was laid to rest by a U.S. Department of Homeland Security official soon after the shutdown.
The other was the release of the minutes from the Federal Reserve’s June meeting at 2 p.m. The opinions of the central bankers drive sentiment for stock-pickers worldwide, and Pfeffer, a senior portfolio manager at CLS Investments, was anxious that a rush of orders might prove unwieldy with the largest U.S. exchange out of commission.
It turned out that he needn’t have worried.
“From where I sit, it doesn’t seem to have had any impact at all,” Pfeffer said. “And for our business, it’s been business as usual.”
That’s because the NYSE is one of 11 exchanges and more than 50 private venues where American stocks change hands. While today’s issue affected transactions on the company’s main market, investors could still buy and sell stock elsewhere.
“One of the benefits of a decentralized, competitive market is that it makes the markets more resilient to an outage in a single market,” said Adam Nunes of Hudson River Trading in New York. “ The market was less resilient to this kind of outage 10 years ago.”
Written by Sam Mamudi, Leslie Pocker, and Ryan Hoerger of Bloomberg