> The cause? A coder had mistakenly programmed a router to send placeholder bids as live orders. If not for the good graces of the options exchanges, the bank would have lost $500 million, according to the U.S. Securities and Exchange Commission. Cancellations and price adjustments reduced that to $38 million
It basically outlines a cascade of failure in controls, bad configuration defaults, and poor SDLC. In particular:
>In addition, the firm’s operation and management of its electronic “circuit
breakers” did not effectively block the erroneous orders sent on August 20. These circuit
breakers existed to prevent erroneous orders by halting all message traffic to the exchanges once
that traffic had exceeded a certain rate. However, on August 20, the firm’s control personnel
repeatedly lifted the circuit breakers blocks between 8:44 a.m. and 9:32 a.m., thereby permitting
additional erroneous orders to be sent to the exchanges. Before lifting the circuit breaker blocks,
the control personnel did not obtain authorization from the responsible technology employees, as
required under written firm policies.
>The firm’s policies relating to the manual “lifting” of those circuit breakers were
not disseminated to or fully understood by the employees responsible for deciding when the circuit
breakers should be lifted, and, prior to August 20, 2013, GSCO personnel had lifted circuit breaker
blocks shortly after learning of the block and while still investigating the cause of the circuit
breaker trip.
From what I remember, this circuit breaker was notorious for raising so many false positives that control personnel just got used to lifting it without thinking.
That’s really a terrible place to put people in. From what this says, the SEC put part of the blame on the employees who lifted the circuit breakers without approval, and also state that employees didn’t know that they needed approval?
Before lifting the circuit breaker blocks, the control personnel did not obtain authorization from the responsible technology employees, as required under written firm policies.
The firm’s policies relating to the manual “lifting” of those circuit breakers were not disseminated...
It's interesting to read the full SEC report because it's a little more complicated than that: https://www.sec.gov/litigation/admin/2015/34-75331.pdf
It basically outlines a cascade of failure in controls, bad configuration defaults, and poor SDLC. In particular:
>In addition, the firm’s operation and management of its electronic “circuit breakers” did not effectively block the erroneous orders sent on August 20. These circuit breakers existed to prevent erroneous orders by halting all message traffic to the exchanges once that traffic had exceeded a certain rate. However, on August 20, the firm’s control personnel repeatedly lifted the circuit breakers blocks between 8:44 a.m. and 9:32 a.m., thereby permitting additional erroneous orders to be sent to the exchanges. Before lifting the circuit breaker blocks, the control personnel did not obtain authorization from the responsible technology employees, as required under written firm policies.
>The firm’s policies relating to the manual “lifting” of those circuit breakers were not disseminated to or fully understood by the employees responsible for deciding when the circuit breakers should be lifted, and, prior to August 20, 2013, GSCO personnel had lifted circuit breaker blocks shortly after learning of the block and while still investigating the cause of the circuit breaker trip.
From what I remember, this circuit breaker was notorious for raising so many false positives that control personnel just got used to lifting it without thinking.