Microsoft Culpability - Steve Gibson

I've been listening to Steve Gibson's "Security Now" podcast for close to 15 years. He knows the internet inside out and has done many a deep dive into how crypto works. He coined the term "spyware" way back when. He's also a programmer and has created and developed SpinRight, a utility for verifying, maintaining and repairing disk drives. In his podcast he spends a goodly amount of time talking about the Windows operating system, which he uses. 

Though he has often been critical of Microsoft, this is a bit strong even for him. I have taken the transcript of a recent podcast and am pasting it here. The other person in the conversation is the host of the "Security Now" podcast, Leo Laporte.  There will be follow-up comment by Steve in a week or so and I will post it as a reply. So buckle up! This is going to be a bumpy ride.
 

Steve: Oh. So as I mentioned at the top of the show, I didn't start off today's podcast with this title or topic in mind. Far from it. This section for today was originally up where it usually is, with the generic security news, under the title of Patch Tuesday Redux. But sometimes it's necessary to step back and perform a bit of a reality check. One piece of news from last week hit me as being so unconscionable that, as I started to explore it and what it actually meant, it became clear that the only way to read the facts was that something has gone very wrong at Microsoft. I have no illusions that this podcast will change Microsoft's behavior. But perhaps it's time for us to think about changing ours.
 
So what follows is what I started off writing, so it starts off sounding like any other Patch Tuesday update. I said: "Last Tuesday, Microsoft released fixes" - it is a Patch Tuesday update - "Microsoft released fixes for 44 security vulnerabilities, with seven of the vulnerabilities being rated critical and three of those being zero-days. The other 37 were rated as being important. Even though the total of 44 is back to being fewer, 13 of the patches fixed remote code execution vulnerabilities, and eight were information disclosures. 


"The affected Microsoft products included .NET Core & Visual Studio, ASP.NET Core & Visual Studio, Azure, Windows Update, Microsoft Print Spooler Components, Windows Media, Windows Defender, Remote Desktop Client, Microsoft Dynamics, Microsoft Edge (the Chromium version), Microsoft Office, Microsoft Office Word, Microsoft Office SharePoint, and others.
 
"Perhaps the most prominent patch released last Tuesday dealt with the Windows Print Spooler Remote Code Execution vulnerability, which has been a major focus since its disclosure in June. And what makes Microsoft's recent performance all the more embarrassing is that the day following Tuesday's patch batch, last Wednesday, believe it or not, Microsoft acknowledged still another remote execution vulnerability in Windows. 


Leo: One super smart. And then one dumb.
 
Leo: So you do have VLANs.
 
Leo: Actually hardware LANs, really. 


Leo: Yeah, yeah. Very nice. Very clever. Okay, Steve. Let's hear it. What did Microsoft - what did they do this time?
 
Security Now! Transcript of Episode #832 Page 18 of 22
 
Print Spooler which it said it's working to remediate. This Print Spooler remote code execution vulnerability is being tracked as CVE-2021-36958 and carries a CVSS score of a mere 7.3." 


In their disclosure of this problem, Microsoft wrote: "A remote code execution vulnerability exists when the Windows Print Spooler service improperly performs privileged file operations. An attacker who successfully exploited this vulnerability could run arbitrary code with system privileges. An attacker could then install programs; view, change, or delete data; or create new accounts with full user rights." So no surprise there, right? That's the standard boilerplate for all the bad things that can happen, and also typically do happen, whenever we allow bad guys to remotely execute their code on our machines.
 
Then in the show notes I have a tweet from Victor Mata. He tweeted on August 11, which was last Wednesday: "Hey guys, I reported the vulnerability in December '20, but haven't disclosed details at MSRC's request. It looks like they acknowledged it today due to the recent events with print spooler." So now here we are again with another newly disclosed Windows Print Spooler RCE. That's not good. I mean, It's really not good. But what's difficult to understand is that we're also told that Microsoft was first made aware of this problem way back in December of 2020 by Accenture Security's Victor Mata of FusionX. So another remote code execution vulnerability in the Windows Print Spooler, which Microsoft has known about since December? And now it's mid-August. And now they're telling us about it and saying that they're scrambling to fix it. 


Will Dormann, CERT Coordination Center's Vulnerability Analyst, almost predictably tweeted in that thread that Victor started. He tweeted: "Sometimes I wonder why I bother writing things up and notifying vendors." Yeah, I'd wonder, too.
 
Microsoft is nothing if not a savvy software publisher with effectively unlimited financial resources. Microsoft's current cash on hand is $130 billion. $130 billion of cash just lying around right now. They could have afforded to hire a talented coder to fix this one problem without even noticing the expense. Not even a rounding error. So they must have decided and I'm really not kidding about this. They must have decided, in some gold-plated ivory tower somewhere, that bugs in their code don't really matter that much.
 
We all assume, "Oh my god, a remote code execution exploit. The sky is falling." But Microsoft clearly doesn't think so, or they'd prop up the sky if that was a problem. They certainly have the money to do so. But shhh, don't tell anyone. I don't think they really care anymore. 


Think about Microsoft's behavior all this year through the Exchange Server fiasco, which directly hurt and damaged so many of their own customers. Not other people's customers. Their customers. Their software. Does anyone think that they lost a single one of those Microsoft enterprise customers as a result? We know they didn't. Microsoft is the only game in town, and the prior investment in Microsoft's ecosystem is far too great. So what did not fixing those Exchange Server flaws quickly cost them? Nothing. 
And notice that the attackers are the ones who are increasingly being blamed. We're not blaming the victims of ransomware attacks. We're not blaming the faulty software which those attackers used to gain their foothold, exfiltrate victims' proprietary data, and encrypt their victims' machines. Now we're blaming the attackers. It's their fault for taking advantage of our flaws and weaknesses. It's their home government's fault for allowing them to do that. The U.S. Government is loudly screaming, "You'd better stop attacking us, or else," while Microsoft sits on another remote code execution flaw in Windows Print Spooler for eight months.
 
Security Now! Transcript of Episode #832 Page 19 of 22 


Microsoft's not dumb. They didn't get to be where they are by being dumb. Microsoft has always known what matters. And any unbiased appraisal of their demonstrated behavior this year would have to conclude that they are now only paying lip service to their software vulnerabilities, and only then because the politics of the situation requires them to at least appear to care. They are allowing a great many serious software vulnerabilities, of which they have been previously made aware, to remain unpatched for months, while sitting on $130 billion previously paid to them by those same customers who are being directly hurt by those easily patched vulnerabilities.
 
By delaying the repair of the Exchange Server vulnerabilities at the start of this year, which they were told of in 2020, but didn't bother to repair until they were being used to attack their own customers by the end of March 2021, they directly enabled those devastating attacks against their own paying customers. And so now we're learning of another case where they've known of a remote code execution vulnerability for eight months. How are we to understand any of this, except as the result of a brutal cost- benefit analysis? They have so much money that they could easily arrange to fix these things if they cared at all. 


It's not as if these are difficult problems. When it suddenly becomes an emergency, the problems are fixed and released immediately. And it's not as if these are unknown problems. Researchers are bringing these problems to them to fix, asking them to do that, literally handing them to them. But time and time again, Microsoft doesn't bother. Are they so busy working on Windows 11, getting those rounded corners just right, arguing about whether or not to force the new centered menu upon their Windows 11 users, that they can't spare even one employee to fix a serious problem that's been laid at their feet? 


At this point in 2021, I think we really need to stop and ask ourselves an important question. Is this the behavior of a company we should continue to support? Is this the behavior of a company that deserves our trust and loyalty? Really, do they have it?
 
What we have been seeing this year is culpable negligence on Microsoft's part. There is no possible excuse for their behavior. The only possible explanation is that they just don't care anymore. They have the money, they have the resources, and they're being handed the knowledge to prevent devastating attacks against their own customers who have enriched them. And they're doing nothing about it because they don't have to. They have a monopoly on desktop computing. There's no reasonable alternative to Windows. In the past, they used their monopoly to abuse their competitors. Now they're using it to abuse their own customers. 


Leo: Ooph. I can't disagree with you. I'm always, you know, trying to understand it, put myself in the other guy's shoes. And I just for the life of me can't come up with a good reason why they would, for instance, wait eight months to patch something, and wait till it's a zero-day, and then say, oh, I guess we'd better fix it. It just doesn't - I can't think of any reason why you might not want to fix it. 


Leo: Is it possible there are so many flaws, I mean, that there are literally tens of thousands of known serious critical flaws, that they just can't fix them all? They have to triage it and have to wait till it's a zero-day and then say, oh, well, okay, we can fix that one now? Is that possible? 


Security Now! Transcript of Episode #832 Page 20 of 22 


Steve: No, no, because they fixed 44 this month. It could have been 45. But it wasn't.
 
Steve: And they knew about it for eight months. And Leo, $130 billion. They could have a second whole Microsoft that just fixes bugs. 


Steve: They could. They could.
 
Steve: And it's not like they're being forced to discover them. They've got the whole security community finding them for them, showing them. Here's a problem. Here's a zero-day.
 
Steve: There is no cost to them, Leo. It does not cost them. 


Steve: They don't need to. They lost not a single customer. 


-Bill 

As promised, here are some further comments from Steve Gibson on the Microsoft Culpability take he made last week. Personally I think this is a bit of a stretch. What seems "reasonable" doesn't mean it's true. Sometimes there's just pure incompetence. Nevertheless, we know from Edward Snowden what kind of stuff the NSA is up to, so this is not beyond the realm. So, for whatever it's worth, here you go. This from the transcript of the Security Now podcast #833, on August 24, 2021. 

It may also be worth noting that Steve Gibson quoted a Microsoft source that said Windows is "Vulnerable by Design". There needs to be some context here, but generally Gibson has been critical of the Microsoft practice of turning everything on by default and leaving it up to users to turn off unneeded or unwanted functions. He was an early critic of the security vulnerabilities in the Windows Plug 'n Play feature. He labeled it "Plug 'n Pray".  

LEO: As we head into the final quarter of this episode, Steve is going to do a little thought experiment.

STEVE: Okay. Not long after we finished recording last week's podcast, which as we all know was titled "Microsoft's Culpable Negligence," I had another thought. Last week I said over and over, and drove the point home, that given Microsoft's effectively unlimited resources, and having clear knowledge of new highly critical vulnerabilities handed to them, and of the devastating impact the exploitation of those vulnerabilities would have, I was unable to see any rational explanation for Microsoft's behavior, other than that they had to be performing a brutal cost-benefit analysis and rationally deciding not to fix those vulnerabilities, you know, not to take the time to fix them. One way or another, for one reason or another, this had to be a deliberate decision because there was no way to parse the history that we have all been witness to that wouldn't cause any unbiased observer to conclude that what has been happening was exactly what Microsoft had decided should be happening, insane as that at first appears.

Last week I stated that there could be no other reason. Then I thought of one. There is an explanation that perfectly maps onto all of the evidence and exactly predicts the behavior we're all witnessing from Microsoft. And in this proposed model, the driving motivation is, indeed, a brutal cost-benefit analysis, but one that's even more brutal than we imagined. It's just not the obvious cost-benefit analysis I was focused upon and described last week. Last week I was assuming that it would only be hostile and malicious adversaries who would be attacking users of Microsoft's software. Thus the "cost" in the cost-benefit analysis would be the attacks themselves. But what if, instead, attacks were the benefits? And what if those benefits arising from attacks were so beneficial that they outweighed the cost to Microsoft, which we've already determined to be effectively negligible?

So then we have to ask, how could attacks on users of Microsoft's proprietary software be beneficial? Such attacks would be in the U.S. national interest if they were being conducted by the United States domestic intelligence services against U.S. foreign adversaries. I recall mentioning on this podcast many years ago that Microsoft routinely tipped off our U.S. intelligence agencies about recently discovered and not-yet-patched flaws in Windows, and in their various other products. On Security Now! Episode 426, which we recorded on October 16th, 2013, I quoted Bruce Schneier from a piece he wrote for The Atlantic titled "How the NSA Thinks About Secrecy and Risk." I'm going to read directly, verbatim, the first five paragraphs of that piece, which Bruce wrote nearly eight years ago.

He said: "As I report in The Guardian today, the NSA has secret servers on the Internet that hack into other computers, codename FOXACID. These servers provide an excellent demonstration of how the NSA approaches risk management, and exposes flaws in how the agency thinks about the secrecy of its own programs. Here are the FOXACID basics," Bruce wrote. "By the time the NSA tricks a target into visiting one of those servers, it already knows exactly who the target is, who wants him eavesdropped on, and the expected value of the data it hopes to receive. Based on that information, the server can automatically decide what exploit to serve the target, taking into account the risks associated with attacking the target, as well as the benefits of a successful attack.

"According to a top-secret operational procedures manual provided by Edward Snowden, an exploit named Validator might be the default, but the NSA has a variety of options. The documentation mentions United Rake, Peddle Cheap, Packet Wrench, and Beach Head, all delivered from a FOXACID subsystem called Ferret Cannon." He says: "Oh, how I love some of these code names." Then he says, in parens, "(On the other hand, EGOTISTICALGIRAFFE has to be the dumbest code name ever.)"

He says: "Snowden explained this to Guardian reporter Glenn Greenwald in Hong Kong. If the target is a high-value one, FOXACID might run a rare zero-day exploit that it developed or purchased. If the target is technically sophisticated, FOXACID might decide that there's too much chance for discovery, and keeping the zero-day exploit a secret is more important. If the target is a low-value one, FOXACID might run an exploit that's less valuable. If the target is low-value and technically sophisticated, FOXACID might run an already known vulnerability."

And here's the line: "We know that the NSA receives advance warning from Microsoft of vulnerabilities that will soon be patched." So he continues: "There's not much of a loss if an exploit based on that vulnerability is discovered. FOXACID has tiers of exploits it can run, and uses a complicated trade-off system to determine which one to run against any particular target. This cost-benefit analysis doesn't end at successful exploitation. According to Snowden, the TAO" - that's the Tailored Access Operations, of course we were all talking about that eight years ago - "operators running the FOXACID system have a detailed flowchart, with tons of rules about when to stop. If something doesn't work, stop. If they detect a PSP, a personal security product, stop. If anything goes weird, stop. This is how the NSA avoids detection, and also how it takes mid-level computer operators and turns them into what they call 'cyberwarriors.' It's not that they're skilled hackers, it's that the procedures do the work for them."

Okay. So in that fourth paragraph of that longer piece, famous security expert Bruce Schneier said: "We know that the NSA receives advance warning from Microsoft of vulnerabilities that will soon be patched." The revelations made by Edward Snowden and WikiLeaks stripped us of our innocence and matured our understanding of the true nature of the global cyber-intelligence world. Sometimes the need to gather intelligence requires, how shall I put it, an extreme lack of passivity.

So once again I'm being entirely serious about this. Think about it for a moment. Microsoft receives notification of a critical vulnerability from any of the world's many white hat hackers who are poking and prodding at their products. Say they get notice of a horrifically exploitable flaw in their email Exchange Server. The exploit is not publicly known, and its discoverer has agreed to keep it to themselves until sometime after it has been fixed.

So here's how this proposed timeline would play out. Microsoft thanks the security researcher hacker and promises to graciously throw them a bone by mentioning their discovery in their eventual disclosure. Perhaps they'll also receive a bug bounty, but of course only if they remain silent until the problem has been fixed and a sufficient number of systems have been patched.

Next, Microsoft then uses their well-established quiet backchannel to pass the researcher's findings on to the NSA and the CIA. Microsoft takes no active part in the development of an exploit because that would be crossing a line. And should it ever become common knowledge that this early information was provided to U.S. intelligence services, Microsoft is simply being a good citizen and helping our own domestic intelligence agencies to guard against attacks which might exploit this now-discovered flaw, yet not publicly disclosed.

And now Microsoft sits on it. Remember how Victor Mata, who reported a remote code execution vulnerability with full system privileges, tweeted Wednesday before last, on August 11th: "Hey guys, I reported the vulnerability in December 2020, but haven't disclosed details at MSRC's request. It looks like they acknowledged it today due to the recent events with print spooler." So when Microsoft finds themselves in receipt of a valuable vulnerability, do they immediately assign a CVS number? No. That would raise suspicion and speculation and might start people looking. Do they post a note about a new vulnerability that needs fixing and their intention to do so? Nope. Not only do they remain completely mum, they do not patch it. It is only of use to our own, shall we say, "proactive" intelligence agencies who have been informed of it, so long as it remains unknown and unpatched.

All evidence suggests that the reasoning here is that as long as it has not been discovered and publicly reported as being actively exploited in the wild, it's just like any of all those other undiscovered vulnerabilities that exist within Windows, and we all know there's certainly no shortage of those. In the juicy case of Microsoft's Exchange Server, it's been sitting there, previously undiscovered, for more than the past 10 years. So what's a few more months? And just think of all the benefit that our domestic intelligence agencies can reap from it until, and if, it eventually comes to light on its own. Or perhaps, as Bruce observes, as an unfortunate side effect of its active use by our own NSA or CIA. What was it that we heard about the Exchange Server exploit? That it could allow an adversary access to all of the server's local communication history? Think that might be of interest to some of our snoopier spooks?

And, finally, consider that it wasn't until the ProxyLogon vulnerabilities were suddenly found to be exploited by adversaries actively attacking users of Exchange Server that Microsoft finally jumped to attention and rushed out an emergency fix for the problem. Remember that they claimed to be getting ready to release the fix with the next Patch Tuesday? Uh-huh. And remember that something didn't smell quite right about that at the time? And then, since this sudden disclosure apparently caught them by surprise, and they were unable to deny how long ago they had been originally been informed about it by our old friend Orange Tsai, they then hinted that someone they told might have leaked the details. Right. More like someone they told might have been caught privately exploiting it.

I frequently hear that we have listeners who have been with us from the start, for all of these past 16 years. And those people will have heard every podcast we've produced, and they will know that they've never heard me once jump onto a conspiracy theory. That's not the way I roll, and I certainly have no intention of doing so now. I have no firsthand evidence of this, and I'm not particularly interested in digging up any. As we all know, I've got much better things to do. But we also know that absence of proof is not proof of absence. And any system such as this would be well designed to remain off the books and under the radar.

I've got a chart here in the show notes. It was made on March 24th, several weeks after Microsoft's emergency patch release for the ProxyLogon flaws. In it we see, okay, this is two weeks after the patch was made available, on March 24th. On the chart we see 2,496 still vulnerable Exchange servers in Russia, and 1,473 servers still vulnerable in China. Again, that's weeks downstream. I wonder how many Exchange servers in Russia and China might have been vulnerable before Microsoft's revelation?

If nothing else, this resolves the apparent mystery of Microsoft's culpable negligence by converting apparent negligence into reasoned neglect. I guess the question remains, which would we rather have, a negligent Microsoft or a diabolically neglectful Microsoft? Neither places the interests of their own customers first.

LEO: That makes sense.

STEVE: It does.

LEO: I mean, you never know, but...

STEVE: It's sad.

LEO: ...it makes sense, yeah. Well, I mean, it's only a possibility. It's not necessarily what's happening. But it makes sense.

STEVE: No, it's not. But if Bruce is right, and Microsoft is giving tips to the NSA, and the NSA has an interest in actively exploiting, I mean, and being proactive in their intelligence gathering, and I think we must all, given what we learned from Snowden and WikiLeaks, we must all understand that that's the case, I mean, we saw pictures of NSA nodes at central points of the Internet, you know, monitoring all the traffic that went through. We've seen pictures of the massive server farms next to sources of cold water so they're able to cool themselves. I mean, you know, this is big business. So how could they not, I mean, it would almost be malpractice for the NSA not to take advantage while the opportunity is there of the ability to get into remote Exchange servers and use them for gathering intelligence. How could they not? And then really, again, what's Microsoft's hurry? If it's really, really valuable, if it's useful for the U.S., if it's secret, and no one has discovered it in 10 years except one lonely researcher who's promised to keep it quiet with the carrot of a bug bounty once it's disclosed?

LEO: I don't see why Microsoft would do it. But okay.

STEVE: Again, how else do we, you know, last week we explored how could it possibly be that they're not fixing this thing in three months.

LEO: Right, right.

STEVE: And then when it's an emergency, they have a fix overnight. They immediately push out an emergency fix.

LEO: I think there are other explanations possible, as well, including just a sluggishness.

STEVE: And not caring.

LEO: I would hope that Microsoft wouldn't just bend over to the NSA or FBI and the interest of national security over the security of all of its users. And that seems - certainly we know Apple has said no to the FBI in the past. That just seems like a pretty craven thing to do. But maybe they did. Maybe they did. It's possible.

STEVE: Anyway...

LEO: We don't have any evidence of it, we should point out.

STEVE: Nope. No evidence. I had no explanation for it last week. It seemed like, you know, how could they just not care to that degree? Because lord knows they have the resources to fix anything that they're, I mean, it's not - they don't even have to find them. They're being given these. How can they not fix it? I just - you know? And the only thing I can think is, well, that sure would be handy. Again, if there were nearly 2,500 still vulnerable Exchange servers operating in Russia two weeks after the patch had been issued, imagine how many there were before.

LEO: Right. I also think the NSA is unlikely to speak to a company and say we'd like you not to fix that for a while. I'm sure they have enough other exploits they could take advantage of. But maybe. Maybe. I'm not saying it's not true. It's a fascinating theory.

STEVE: Interesting fodder, if nothing else.


-Bill