The Edge Case is What Matters
Software people are now in charge. But they're not trained to consider the mind-boggling destructive power of rare unforeseen events, and that's already creating big problems.
Long before dawn in June of 2022, I woke my eldest daughter to go with me on a story. NBC News had asked me to illustrate the crushing power of inflation, and I’d decided that the best way to do it was a profile of a Bay Area bakery called Semifreddi’s. I don’t remember exactly why it was necessary for my 11 year old to come with me for a few hours on that summer day — you can read this post for a look into the near-disastrous clusterfuck of trying to be a parent and husband and news correspondent at the same time — but having her with me, sleepy and grouchy as she understandably was, gave me tremendous pleasure. Being away from my family at all hours for that job meant that the rare chance I had to bring a kid with me to work was a lovely thing. We met the rest of my crew, shook hands with the owner, Tim Frainier, and, donning hair nets, we were shown through the vast bread-making factory that provides everything from granola to focaccia for grocery stores and restaurants across a 40-mile radius.
As we finished our tour of the place, Frainier pointed out with great reverence the last stop his products make before being loaded into a delivery truck. It was a metal detector.
“Every bag of granola, every crouton, every baguette goes through it,” he said proudly.
Why was this necessary?
“You saw all the machines we work with here,” Frainier explained. “Eventually one of those bolts could come loose and fall out, and if someone bites into a piece of metal in their morning bun, that’s it for my business.”
I was astounded. In my work as a technology journalist, and especially having just written a book about what goes wrong when companies unleash prototype technology like AI on the world, I was used to CEOs who talk about “iterative development” and “always looking to improve” but almost never about actually preventing a one-in-a-million occurrence like this. I rarely if ever encountered a leader who took responsibility for anticipating what the industry calls “edge cases.” The “use case” for a product is the stereotypical, middle-of-the-road customer whose need for it is obvious, and defines the market opportunity. Software folks — especially those who are used to working against the pressures of the limited startup money they’ve been handed — are taught to race for a “minimum viable product” that can be released as quickly as possible to early adopters who will give feedback and help iron out the kinks. The “edge case,” meanwhile, is that rare occurrence, that secondary or even tertiary result, that has clearly negative effects but is so rare relative to the overall size of the user base that the company usually doesn’t spend much time worrying about it, certainly not enough to slow down or halt a product launch.
This is a software-development attitude, because in software, scale tends to solve this sort of problem: the more users you have, the more bugs they report, the more updates you release, and the smaller the edge cases become as a percentage of the total. But the owner of Semifreddi’s considered the edge case his fundamental responsibility, and he considered anticipating and obliterating it the minimum standard necessary to remain in business. For him it was unimaginable to even risk the possibility that a customer might be harmed by his product.
This is not how it works in tech, as I’ve experienced again and again. Just weeks before my trip to Semifreddi’s I'd interviewed David Holz, the CEO of AI-imagery company Midjourney. I had bought a portrait of Denzel Washington not realizing it was Midjourney-generated, and wound up interviewing Holz as part of a story about the experience. As we sat down together, someone had already used Midjourney to produce lifelike images of then-former President Trump in various arrest scenes, including wearing an orange jumpsuit. In fairness, I think Holz made a good-faith effort to think about some of the potential downsides of his product with me. (And I was unhappy to see John Oliver use my story to make fun of Holz’s appearance.) Still, when I asked Holz whether his company should be held responsible for misleading imagery (as in China, which holds the CEOs of such companies personally liable), his answer was what I’d expect from a software maker.
“If I’m in the muffin business, and I bake 10,000 muffins, and someone gets food poisoning, am I supposed to stop making muffins?” he asked.
A long pause grew between us. “Well, yes,” I told him. “The FDA would say exactly that.”
(Quick aside: Spanish Prime Minister Pedro Sanchez told a crowd at Davos recently that social media companies should be held responsible for “poisoning society.” He likened it to the responsibility that restaurants bear when their food makes customers sick. Funny how these analogies stick.)
But back at the bakery I didn’t get to mention all this to Frainier, or build a story around it, because as we stood together in front of his prize metal detector, the one installed specifically to protect us all from the edge case, my phone rang. The Dobbs decision had just been handed down, Roe vs. Wade was now overturned, and the news desk in Los Angeles was telling me to relocate to San Francisco and immediately begin asking people on the street what they thought of it.
So I apologized to Frainier — ah, the many times I have spent days negotiating access and time for an interview only to suddenly split an hour into the shoot — hustled my daughter into the car, and raced to foist her on unsuspecting friends nearby so I could get into the city. I spent the rest of the day speaking to mostly distraught San Francisco women and shocked foreign tourists.
I’m sharing all this because for me it was a hugely formative experience. I remember looking over at my daughter as we drove away from Semifreddi’s with dawn finally peeking over the hills, having a full-on parental panic attack. I imagined all the steel washers in her breakfast, all the times AI might mislabel her in some crucial administrative system, all the untenable pregnancies she might face, in a nation that increasingly acts as if her well-being isn’t its responsibility if she happens to fall into the edge case.
Now, of course, a long, sudden list of casually destructive presidential actions and executive orders are coming out of the White House that not only fail to take responsibility for the edge case, they if anything make it more likely. Consider Trump’s appointment of noted conspiracy theorist Robert F. Kennedy, Jr., as Secretary of Health and Human services. Now a blanket prohibition on international cooperation and public statements by any federal health official means the end of crucial global data-sharing, and no report or warning can be released without prior approval by a presidential appointee, just as we enter a new age of global pandemics like Avian flu, which is already gathering strength in the United States and threatening to mutate. Or consider the executive order decreeing that Americans are now to be officially considered whatever gender they were “at conception.” Geneticists have since pointed out that fetuses do not have a sex at conception, and don’t generally develop male characteristics for at least six weeks, making Trump’s order a de facto end of sex and gender distinctions of any kind. This is the sort of misguided throwing-around of political weight that can get us into enormous real-world trouble down the road.
Of course, the Trump administration isn’t the first to overlook broader harms. Consider the legacy of military bases, which for years have been regularly decommissioned and handed off to local governments, under several presidents from both parties. Most municipalities in a nation desperate for added housing would be eager for the land and buildings. Sadly, legal doctrine for decades allowed the armed forces to walk away from environmental responsibility without addressing the often dangerous levels of pollution remaining on a base, leaving cities on the hook for an often unaffordable cleanup. (Brooks Air Force Base, redeveloped into a mixed-use development by the city of San Antonio, is a rare exception.) The Trump administration’s lack of focus and anticipation isn’t unique — it’s common to much of American political and commercial activity.
What is unique is putting tech people in charge. This is the first time we’ve seen federal functions handed to people from the “move-fast-and-break-things” era who are by and large trained to experiment on their users, not to try to foresee the edge case. This is what we’re seeing with the Department of Government Efficiency, led by Elon Musk and staffed by young engineers mostly linked to his companies, as Wired has reported. Musk has famously brought the speed and attitude of software to the automotive industry, where studies have found that Tesla leads the industry in accidents and fatalities (Tesla executives have disputed these conclusions), and whose regulator, the NHTSA, raised safety concerns that led to the recall of more than 2 million vehicles and concluded that Tesla’s autopilot system has “led to foreseeable misuse and avoidable crashes” in hundreds of collisions. Now the young engineers working at DOGE, several of whom have reportedly worked at Tesla in the past, are gaining access to everything from the Federal Payment System (which administers benefits to millions of Americans) to the National Oceanic and Atmospheric Administration (the nation’s top agency when it comes to monitoring and reporting climate data).
Think about the potential secondary effects of DOGE’s actions against the US Agency for International Development, which is seeing its staff cut from more than 10,000 worldwide personnel to fewer than 300 and is in the midst of a 90-day freeze on the billions it spends on foreign aid. (Musk said in a live event on X that President Trump had agreed with him that “we should shut it down,” and an NBC News film crew captured a worker removing signage from the agency’s Washington headquarters on Friday.)
The billions in discontinued funding includes medical supplies and other assistance to Palestinians in Gaza. The direct effects on a population already seeing climbing rates of polio and cholera are clear. The secondary effects could include those on a generation of Palestinians whose direct experience of the end of lifesaving support from the United States (not to mention the threat of US occupation, as President Trump keeps insisting he’s committed to) predisposes them to hostility, if not violence, toward American citizens wherever they encounter them. In addition, it could upend any chance at negotiating a ceasefire between Israel and the Palestinians. As a representative of Refugees International told ABC News…
I think one of the facets of this that people are not necessarily connecting is that the humanitarian access to Gaza -- which is underpinned by USAID, State Department funds -- was a central feature of the bargain that underpins ceasefire. So, if you remove aid to Gaza, whether directly or as a second-order effect of the aid freeze, this raises the risk of a broader ceasefire collapse.
That’s an edge case that’s hard to quantify the way one might in the typical review process of a software company, but is obvious, and obviously calamitous, to those with experience in the region.
There is a place for efficiency in government, and as I’ve written, history has seen many examples of improving civil infrastructure through technology and innovation. But as President Obama pointed out during a conversation at Carnegie Mellon in 2016, government is for the problems that the market doesn’t want to solve, or cannot. As he put it…
Government will never run the way Silicon Valley runs because, by definition, democracy is messy. This is a big, diverse country with a lot of interests and a lot of disparate points of view. And part of government’s job, by the way, is dealing with problems that nobody else wants to deal with.
Sometimes I talk to CEOs, they come in and they start telling me about leadership, and here’s how we do things. And I say, well, if all I was doing was making a widget or producing an app, and I didn’t have to worry about whether poor people could afford the widget, or I didn’t have to worry about whether the app had some unintended consequences … then I think those suggestions are terrific.
Sometimes we get, I think, in the scientific community, the tech community, the entrepreneurial community, the sense of we just have to blow up the system, or create this parallel society and culture because government is inherently wrecked.
We are seeing that destructive instinct being deployed right now, in a grand, unregulated, live experiment on us all. The state is now a minimum viable product, and increasingly, the edge case isn’t part of the plan.
WOW! I see daily the tragic effects of an “edge case” life-shattering event playing out before me. A buddy received an injection which is commonly known to cause “temporary blindness for usually no more than 48 hours” which has drastically reduced his formerly perfect vision 2.5 years AFTER that fateful shot. Nearly blind at 53 is unforgivable!
Terrific piece on precisely what’s going wrong.