It seems like every week now we’re seeing security incidents happening that would have made mainstream news just a few years ago, the big ones lately have been the North Korean hackers who Trojan horse ta the JavaScript ecosystem, the open AI model that committed a felony to get answers to a benchmark and that time anthropic accidentally open source cloud code at 4 am via an NPpm source map, but sometimes it’s good to remember that even before AI the software has always been held together with duct tape and the poverty of open source Mainers, and there’s no better example of this than,Damage one company managed to do to itself 14 years ago this month with organic human retardedness in about 45 minutes on a random Wednesday, Night Capital, who at one time was responsible for 10% of all stock trading happening in the United States a lost 10 million in a minute with this one weird trick It is August 27, 2026 and you’re watching the code report in 2012.
如今几乎每周都会出现一些安全事件——这些事若放在几年前绝对能上主流新闻头条。最近的典型案例包括:朝鲜黑客通过木马程序入侵JavaScript生态圈;某开源AI模型为获取基准测试答案不惜”犯罪”;还有那次Anthropic公司凌晨4点通过NPPM源码映射意外泄露云端代码。但有时我们需要记得,早在AI时代之前,软件世界就始终靠着”胶带式修补”和开源社区的穷困维持着,最典型的例子莫过于14年前某个周三的45分钟里,某公司因人为失误造成的自我毁灭——骑士资本(Night Capital)曾处理全美10%的股票交易,却在那个早晨亏损4.4亿美元,四个月后便遭拆分出售。本期视频我们将剖析史上最昂贵软件故障的成因,并学习这个”一分钟亏千万”的奇葩操作。现在是2026年8月27日,您正在收看2012年的代码事故报告。
While the rest of us were planking on gas station roofs and awaiting the Mayan apocalypse, the night capital was quietly operating as the biggest market maker on Wall Street as it processed $20 billion in trades every day. If you remember back to the gametop Robin Ho saga from a few years ago. night Capital was basically the Ctadel of its time. whenever an ape like you or me would buy a stock from a brokerage like Etra that the brokerage didn’t actually execute the trade themselves. Instead, they’d often pass it off to night who would fill the order and pocket the spread at its core was a system called.
当普通人还在加油站顶棚玩平板支撑、等待玛雅末日预言时,骑士资本已悄然成为华尔街最大的做市商,日均处理200亿美元交易。若您还记得几年前游戏驿站轧空事件中的Citadel,当年的骑士资本就是这种级别的存在。当散户通过E*Trade等券商下单时,券商实际会将订单转交给骑士资本执行——其核心是名为MRS的订单路由系统。
MRS, which was an order router that took large orders and chopped them up into smaller orders to fill them across the market at the best prices, it was fast, reliable and more importantly it printed money for over a decade, then in the summer of 2012 the New York Stock Exchange decided it was sick of watching firms like Knight skim off all the retail orders before they ever reached the actual exchange so it created something called the Re liquidity program which was essentially its own version of Knight’s business model designed to get orders back with slightly better prices.
这个系统能将大额订单拆分为小额订单,以最优价格分散执行。它快速可靠,更重要的是持续印钞十余年。直到2012年夏天,纽交所厌倦了骑士资本等公司在零售订单到达交易所前截胡,于是推出”零售流动性计划”——本质是复刻骑士的商业模式,但提供略优价格。
The SEC approved it in June with a go live dateate of August 1t, and because Knight was ironically also one of the exchange’s biggest customers, they were forced to implement it.So that’s what they started to do.And like any large financial institution responsible for the livelihood of hardworking blue-colored Americans, they did it with dignity and grace. Just kidd they did it in the most lazy way possible at the time, deep in the Smars codebase was an old feature flag that hadn’t been used since 2003, but it was never deleted flipping the flag would trigger a test function called powerpeg which would execute a bunch of buy orders so Knight could watch how a stock’s price would respond. And since the whole point was to push prices around, it was designed to buy aggressively at the current market price.
SEC于6月批准该计划,定于8月1日上线。讽刺的是,作为纽交所最大客户之一的骑士被迫配合改造。他们开始了”庄严体面”的代码重构——才怪!实际采用了最偷懒的方案:代码库深处有个2003年后就废弃的功能开关,触发后会激活名为Powerpeg的测试函数。这个原本用于模拟市场冲击的功能会以市价疯狂买入股票(根本不考虑成交价差)。
Z concern for getting a good deal but when the engineers in 2012 needed to switch on the new retail liquidity program rather than create a feature flag, they just reused the old Powerpeg1 and swapped out the logic behind it and they would have gotten away with it if this code was only running on a single server or if they had any Devops on the team or if it hadn’t been for those meddling kids in their dock but I assume none of those were true back then their deployment strategy took inspiration from the Pony Express and that they’d have a guy manually copy code changes to their eight servers across several,
2012年的工程师们为启用新功能,竟直接复用了这个废弃开关,仅替换底层逻辑。若系统仅运行在单服务器上,或团队有Devops流程,或部署流程不这么原始(像驿马快信般靠人工拷贝代码到8台服务器),本可避免灾难。可惜历史没有如果——部署时他们遭遇计算机领域两大难题之一(缓存失效与命名问题),导致8台服务器中7台更新成功。
Unfortunately, while deploying the retail liquidity change, they hit one of the two hardest problems in computer science and only seven out of the eight servers got the update then on August 1, 2012, when Night flipped the feature flag, the seven of their servers processed orders correctly while the eighth1 woke up powerpeg from its nine- year hibernation and it got to work on its DJ strategy of buying high and selling low, Night immediately noticed something was up, but as you can imagine, someone who gets themselves into the situation is unlikely to have a good way of getting themselves out of it.
2012年8月1日,当骑士启用新功能时,7台服务器正常运作,第8台却唤醒了沉睡九年的Powerpeg,开始执行”追涨杀跌”策略。骑士虽立即察觉异常,但正如您所料——能犯这种错的人通常也不懂如何止损。
So they panicked, assumed the problem was in their new code and rolled back the seven healthy servers, which meant now all eight were running Powerpeg and to be honest, that feels like a suitable name for what the market was doing to them. in the 45 minutes, it took them to figure out the issue and flip the feature flagback they had executed 4 million trades across 154 stocks and were now the proud owner of a new 3 to 440 million and its stock had dropped 75% in two days
慌乱中他们误判问题在新代码,回滚了7台正常服务器——导致全部8台开启Powerpeg模式(这名字倒很贴切,市场确实在”强力peg”他们)。在排查问题的45分钟里,系统疯狂执行400万笔交易,涉及154支股票,最终持仓高达70亿美元。某名为Wizard Software的仙股更从3美元莫名飙至14美元。当硝烟散尽,骑士已亏损4.4亿美元,股价两日内暴跌75%。