Microsoft Blames Monster Email Outage On Software Defect Introduced In July Fix

Microsoft says last week’s massive Microsoft 365 outage was caused by a software defect introduced in a July update that remained dormant until a Teams failover and a simultaneous 14% reduction in gateway capacity triggered a cascading failure. More than 80% of the affected gateway servers became unresponsive at the outage’s worst point, and because the months-old change could not simply be rolled back, engineers spent roughly 15 hours developing and deploying a patch that restored the previous code path. The Register reports: According to a preliminary post incident report published by Microsoft on November 15, the defect was introduced in July 2026. The update looked fine at the time, Microsoft says, having passed load testing and been deployed without any impact. However, the testing wasn’t good enough. Microsoft’s report states: “Our load testing tended to focus on weaker hardware with fewer cores.” The defect caused “lock contention inside an internal Windows process which caused the machine’s resources (CPU & memory) to be exhausted.”

“The contention issue is worsened on machines with more cores and newer hardware.” The fix containing the defect was prepared specifically for Microsoft 365 network gateway servers. Then two things happened at once. Teams failed over to the “primary” gateway that supports Exchange Online, creating a surge in load that was not, in itself, particularly unusual. At the same time, almost 14 percent of server capacity for the networking gateway was out of service, which wasn’t a problem by itself either: it was on its way to being restored and, according to Microsoft, “the service had sufficient capacity headroom.” Together, they pushed the gateway into the defect and chaos ensued. It was a classic cascading failure scenario. When a server loses connectivity, traffic gets routed to the next healthy endpoint. Which encountered the same problem. So traffic was rerouted again and… you can see where this was going. Within two hours, the defect was encountered across Microsoft’s fleet. “At its lowest,” the report states, “more than 80 percent of servers were unresponsive.”

So, what to do? Bringing servers back to service didn’t help — they were immediately overwhelmed and fell over again. “Our normally automated recovery processes were not designed to restore service from this broad unresponsive state.” A straightforward rollback was also not possible because the defect was months old. Instead, engineers developed patch for the patch or, in Microsoft’s words, “an update that returned the code path to the previously known good state in Windows.” The patch was applied to groups of servers and service was gradually restored.


Read more of this story at Slashdot.

MRDIMM 8800 MT/s vs. 12800 MT/s Memory Performance With AMD EPYC Venice

With last week’s AMD EPYC 9006 SP7 availability and the AMD EPYC 9686F and EPYC 9996 processor reviews, one of the most frequent questions/comments was around the impact that the new MRDIMM-12800 memory had on these Zen 6 server processors compared to prior gen and the Intel/NVIDIA server competition. Given that curiosity over the MRDIMM-12800 impact, here are the EPYC 9686F 2P benchmarks when re-testing at 8800 MT/s memory speeds.

Google Killed Its Dark Web Report, but You Can Still Monitor Data Breaches

Google’s Dark Web Report used to be the only consumer-focused platform that let users scan major data breaches, dark web marketplaces, and private forums to see if their personal information had ended up in the hands of hackers. But the company discontinued the service at the start of 2026, with new scans stopping on Jan. 15 and all accounts deleted on Feb. 16. 

Google says it sunsetted Dark Web Report because, although the information it provided was generally useful, the service failed to provide “helpful next steps” for users whose data was found on the dark web. Google promised a new service that would provide more actionable intel, but that service has yet to materialize, and there haven’t been any further announcements about it yet. 

Dark Web Report may not have been perfect, but it was the only early warning system of its kind that was available for personal use and let users run scans free of cost. Now that it’s gone, however, there are plenty of other services that offer the same service at varying costs and levels of effectiveness. Let’s talk through the nature of the insight that Google’s reports offered, how it collected that information in the first place, and what options exist in the market to replace it. 

How the Dark Web Report worked

Google launched the Dark Web Report in March 2023 as a premium feature available to Google One users, but later extended it for free to anyone with a Google account and moved it to the “Results about you” dashboard. 

You’d set up a monitoring profile with your name, date of birth, physical address, phone number, emails, and Social Security Number if you lived in the U.S. After that, you could head to the dashboard anytime you wanted to run a free scan across dark web forums and marketplaces, publicly disclosed corporate data breaches, and other hidden online spaces to see if any of your personal information had shown up there. It was easy to use—but not a lot of people knew it existed in the first place. 

What made Google pull the plug on Dark Web Report

It’s unclear what prompted the internal decision at Google, but in the absence of concrete next steps, it is reasonable to think the reports could create panic for the uninitiated. There’s no way to remove your information from the dark web once it’s already in circulation there. Still, having an early warning so you could reset 2FA or self-lock your SSN isn’t without merit. 

But perhaps more importantly, most people with Google accounts didn’t know and hadn’t heard of the Dark Web Report’s existence. Google has likely taken it as a hint that users want more active remediation tools than simple alerts, which is why it’s now directing users to the Password Manager, Passkeys, and “Results about you” dashboard instead—though, again, the company hasn’t followed through on a direct replacement for Dark Web Report, despite teasing such a feature.

A few alternatives to Google’s Dark Web Report

Tool Starting Cost Monitoring Features
Have I Been Pwned Free Email addresses, passwords, domain names
Mozilla Monitor Free Up to 20 email addresses
Google Password Checkup / Apple Passwords Free Saved passwords
Proton Pass Plus $1.99/month billed annually Proton addresses, aliases, up to 10 other emails
NordVPN Dark Web Monitor $1.69/month billed annually Up to 5 emails, phone, cards, and national ID
Surfshark Alert $2.99/month billed annually Emails, cards, personal ID numbers
Microsoft Defender $99.99/year 64 identity data types, plus credit score monitoring

Many of the top dark web scanners draw from the same well of information, scanning against major data breaches and dark web spaces. Several also offer breach notifications that send you email alerts when your details show up in a periodic scan. However, none of that is worth paying for because Have I Been Pwned or Mozilla Monitor cover those features quite well for free. Even the default password managers from Google and Apple come with some ability to monitor leaked credentials. 

As for paid options like Surfshark Alert or NordVPN Dark Web Monitor, these services earn their keep when you need to track phone numbers, credit cards, SSNs, and other national IDs—in addition to emails and passwords. NordVPN also has a separate service called Threat Protection Pro, which monitors dark web channels for stolen session cookies. If you’re already on a Microsoft 365 plan, Defender’s identity theft monitoring tools are nice to have because they don’t require a separate subscription, but they’re not currently available to users outside the United States. 

What to do when you get a dark web breach alert

There isn’t an easy way to scrub your information from the dark web once it ends up getting leaked. But you can take steps to protect sensitive information or financial data once you know there’s been a leak. If a password has been compromised, reset it immediately. Also, enable 2FA on the affected account for added security. Even better, switch to a passkey if that account supports it. If not, authenticator apps are a safer way to receive 2FA codes than standard SMS.

If your Social Security number has been leaked, freeze your credit with Equifax, Experian, and TransUnion. You can do this by contacting each agency separately. Freezing your credit will prevent unauthorized credit withdrawals and protect your credit score from abuse by attackers. If one of your credit or debit card numbers has ended up in a breach, call your bank to freeze transactions on that card immediately.

Finally, get in the habit of using email aliases instead of sharing your actual email address with every platform and service provider. Addy.io and Proton’s SimpleLogin are both popular services that let you generate temporary email aliases that point to your inbox.

[$] A new generation of futexes

Futexes
are a kernel subsystem that provides support for fast mutexes
in user space. While futexes can indeed be fast, they are not fast enough
for a number of users. Partly as a result of that, the implementation of
futexes has grown in complexity over the years, with two basic variants
on offer. The desire for faster futexes remains, though, and that inspires
new patches, including this series
from Suleiman Souhlal
adding a third futex class called “futex PING”
for “priority inheritance next generation”.

Mercedes’ clever new driving assist handles complex traffic like a champ

SAN FRANCISCO—When Ars first drove the new Mercedes-Benz CLA last year, the automaker also gave us a sneak preview of its next driver assist. Called Drive Assist Pro, it’s for collaborative partially assisted driving, and from the passenger seat at the time, the demo was impressive. Now the automaker is getting ready to start rolling out Drive Assist Pro to customers with cars running the new MB.OS operating system here in the US and in Germany, and it let us have some time behind the wheel.

While most automakers want a reputation associated with safety, Mercedes has long used passive and active technology to do so in its products. It was one of the first OEMs to introduce airbags, then a decade later started adding active safety systems like antilock brakes, electronic stability control, and adaptive cruise control. And so, by giving the car’s electronics the ability to control the brakes, and throttle, we arrive at the first level of assisted driving—called “level 1” in automotive engineerspeak.

“Level 2” assisted driving, where the car also controls the steering as well as the acceleration and braking, got famous with Tesla Autopilot. But the first lane-keeping assists predate it by almost a decade in passenger cars, when Toyota introduced a system in 2004. The human driver behind the wheel is still in charge, responsible for situational awareness (and anything that arises from a lack thereof), with the idea that long drives become less tiring and a little more effortless when the car does the boring job of keeping a constant speed or a constant distance to the car ahead and between the white lines.

Read full article

Comments

Manjaro Project e.V. non-profit organization created

Frank Vandermeiren has announced
the creation of the Manjaro Project e.V., a nonprofit organization in Germany
that is independent of the Manjaro GmbH & Co KG company. This is following a governance struggle that took place
earlier this year between a number of Manjaro
Linux
contributors and Philip Müller, project lead and CEO of the Manjaro
company.

At the moment, we are still working on optimizing the new development
infrastructure, which was partly necessitated by the ongoing (A.I.) bot attacks
against our old infrastructure.

However, as you may have noticed, the last Testing
Updates
and Stable
Updates
were initiated by the Manjaro Team without supervision from Philip
Müller — with special thanks to @Yochanan for his hard work!

Dennis ten Hoove has indicated
that “a transfer of company assets to the e.V. has not yet taken place“,
and the group’s next step is to ready the nonprofit for the transition and
“reach out and engage in dialogue with the company“.