How the Government Hacks You, Chapter 10: Rowhammer

What is Rowhammer?

Rowhammer is a hack that’s so effective and flexible that the entire way computer memory is currently handled needs to be examined. It’s an attack on the way bits are stored in typical DRAM sticks, and the countermeasures available are seldom up to the task of thwarting a Rowhammer assault.

The issue with DRAM is the high level of density that the industry has achieved. Each ‘row’ of bits is jammed tightly up against the next. Every molecule of wasted space is discouraged by manufacturers.

Rowhammer takes advantage of that incredible density. It floods adjacent rows of memory with several rapid-fire write requests. By disturbing the same memory space over and over again, these write functions cause bits to flip in the next row over. The attack hammers a memory block row by row to flip ‘sympathetic’ magnetic signatures in the adjacent memory spaces. Thus the names: ‘Row’ and ‘Hammer’.

The Google research team assigned to Rowhammer presents the attack succinctly:

“Rowhammer is a DRAM vulnerability whereby repeated accesses to one address can tamper with the data stored at other addresses. Much like speculative execution vulnerabilities in CPUs, Rowhammer is a breach of the security guarantees made by the underlying hardware. As an electrical coupling phenomenon within the silicon itself, Rowhammer allows the potential bypass of hardware and software memory protection policies.”

A successful Rowhammer attack allows the perpetrator to change the very nature of what’s stored in memory. For example, they can add admin flags to normal user accounts. They can add firewall rules that will allow them to install malware. And while they’re at it they can turn off logging, redirect the next backup to a friendly Cloud server, and set up their next great botnet.

Needless to say, Rowhammer is quite flexible. It works on every operating system and on dozens of hardware types including many firewalls, switches, and routers. So why hasn’t Rowhammer gotten more attention? Why aren’t people screaming bloody murder for the industry to fix this dangerous loophole in the way memory is structured?

There are a few reasons. Firstly, every time an exploit was demonstrated, there was a supposed fix ‘coming up’ in the near future. Some of the first attacks were against DDR3 memory, but the industry insisted it could never happen with DDR4. Of course, it could, even working against the current generation of error correction… but let’s not get ahead of ourselves. Suffice it to say, that every time there was an exploit, industry professionals said they had a fix, even if the fix didn’t actually work.

The second reason there hasn’t been a huge outcry is that it’s a hardware issue at heart. And people have already bought billions of dollars of this hardware. There’s not going to be a recall on every stick of memory sold in the past five years. It would cripple the industry. And what exactly would they replace the recalled products with, when nothing seems to work against the more sophisticated versions of the attack?

So outside of government intelligence and a few security gurus here and there, it’s a matter of hoping that the ‘wrong people’ don’t get a hold of a more complex framework that can automatically launch effective versions of the attack. Why cause a panic, right?

A Brief Rowhammer History

Around 2014, the first Rowhammer attacks were discovered and tested. Carnegie Mellon University and Intel Labs published a paper (the first link in this chapter) about it, and not much else happened for a little while.

Then in 2015, Mark Seaborn and Thomas Dullien wrote an attack that could take over a Linux system from an unprivileged account via the Google Native Client (NaCl) sandbox. This caught a lot of security consultants off guard, as it seemingly came out of the blue. Attacks directly against the way that hardware stored data at the most primitive level weren’t even on their radar at the time.

Later that same year, a Rowhammer exploit was created that used Javascript. It was a high-level exploit of a low-level flaw, pushing past all of the abstraction layers in between. Effective, impressive… and frightening. Daniel Gruss, Clementine Maurice, and Stefan Mangard had created a proof of concept that could quite literally change the digital world as we knew it.

But all of these attacks were taking place in labs. They certainly worked, but the sophistication hadn’t hit the ‘real world’ yet. Nothing to worry about, right? Unless of course a government or large private organization had already thrown money at the development of an easy-to-use Rowhammer framework. More on the likelihood of that later.

In 2018, a plethora of new Rowhammer-related attacks emerged. Remember when we mentioned that the memory error correction scheme known as ECC was designed to stop errant bit flipping? Turns out, not if three or more bits are flipped at a time. That meant the defense against Rowhammer attacks would need to be multi-layered to be effective.

And direct access was no longer required to execute such attacks. Welcome Throwhammer onto the scene. It was security researchers at Vrije Universiteit in Amsterdam and the University of Cyprus who cracked this aspect of the attack vector. As an RDMA attack, it’s mostly applicable to Cloud and virtual machine architecture. Still, it took them exactly 64 milliseconds to make a successful attack over the network. This was a network card attack, so no logins were required.

Have we mentioned the second ECC-breaking attack that same year, which also included an OS-level denial of service using Rowhammer, all completely hidden by Intel's Software Guard Extensions (SGX)? Yeah, that was also happening.

But still, these were all theoretical, lab-tested attacks. A couple years passed, and all this worry about Rowhammer started dying off. That was because the new 10 nm-class DDR4 memory was becoming more common, with the vulnerable DDR3 and early DDR4 models quickly getting phased out. Newer DDR4 included Target Row Refresh (TRR), which should have neutered Rowhammer once and for all. Advances in the market win again, right?

Oops. Seems like the new TRR-enhanced DDR4 wasn’t at all safe either. With six years to tweak their designs and add safeguards, the industry failed against SMASH. The early 2021 innovation was a Javascript attack at the browser level, taking around 15 minutes to completely compromise Firefox.

Still, the hardware people were narrowing down the number of attack vectors, right? Not according to Google, who published brand new Rowhammer-related exploits in May 2021. Half-Double is an attack that can actually flip bits two rows over, instead of just targeting adjacent bits.

This was an evolutionary advance in the understanding of how Rowhammer was really working at the substrate level. In Google’s own words:

“Unlike TRRespass, which exploits the blind spots of manufacturer-dependent defenses, Half-Double is an intrinsic property of the underlying silicon substrate. This is likely an indication that the electrical coupling responsible for Rowhammer is a property of distance, effectively becoming stronger and longer-ranged as cell geometries shrink down. Distances greater than two are conceivable.

Google has been working with JEDEC, an independent semiconductor engineering trade organization, along with other industry partners, in search of possible solutions for the Rowhammer phenomenon. JEDEC has published two documents about DRAM and system-level mitigation techniques (JEP 300-1 and JEP301-1).

We are disclosing this work because we believe that it significantly advances the understanding of the Rowhammer phenomenon and that it will help both researchers and industry partners to work together, to develop lasting solutions. The challenge is substantial and the ramifications are industry-wide. We encourage all stakeholders (server, client, mobile, automotive, IoT) to join the effort to develop a practical and effective solution that benefits all of our users.”

So as frightening as this new attack might seem, perhaps the research that it spawns will eventually spell the end of Rowhammer. If the hardware industry listens. And if the resulting cost, however high it ends up being, is paid. Those are some big ‘ifs’.

Some tests were being developed for more modern implementations of Rowhammer. But before they got out of the lab, a new Rowhammer attack was discovered in 2023. The Centauri attack is, admittedly, innovative as heck. It is a device fingerprinting attack, meant to detect a user’s unique set of hardware devices and settings. It can get a 95% accurate fingerprint in under 10 seconds, and a 99.91% accurate fingerprint in three minutes.

At the moment, assume all systems are vulnerable (though most home users simply won’t be of interest to the average attacker of this level of sophistication, unless a framework is developed for broad distributed attacks).

What Are Governments Doing About Rowhammer?

Nothing.

Herein lies the problem. For many years DDR manufacturers, motherboard manufacturers, less-than-ethical security firms and consultancies, and individual security practitioners have all been spreading misinformation about their ability to stop Rowhammer-based attacks. Time and time again, they claim they have a fix. Time and time again, they’ve been caught lying to the public.

People bought more expensive ECC memory as a defense against Rowhammer. Businesses switched to specific Cloud services because they claimed to be immune to Rowhammer. There was a real financial impact. It’s a clear-cut case of false advertising.

The U.S. government has a mandate about truth in advertising, and methods they can use to go after companies and industries that blatantly lie about the safety of their customers. To date, they’ve taken no action against these people.

The U.K. government, similarly, has truth in marketing standards that are supposed to span all mediums. But they too have failed to go after the firms that pushed their snake oil.

If those countries are doing nothing, there’s little hope for any other countries driving tech standards. The Advertising Standards Council of India is largely toothless. EU directives have proven to be less than effective.

Governments already know that this particular industry can’t be trusted, as evidenced by the DRAM price-fixing scandals. Still, they had zero reaction when the biggest memory companies in the world made false claims of stability and security. It’s fallen upon the public to launch class action lawsuits against the memory industry for their most recent price-fixing incident. Nobody is even talking about their deceptions regarding Rowhammer, since there’s more money and less burden of proof in price-fixing cases.

So don’t hold your breath if you are hoping that governments around the world will hold companies accountable for their false claims about Rowhammer defenses. It’s unlikely to happen at this point. Nor should you expect government regulation, standards, or research to be of any help in the matter.

Are Governments Using Rowhammer Right Now?

Simply put: It’s too early to tell.

It normally takes two to three years to get these attacks out of the labs and into the wild. That means agencies who were eyeing the three highly fruitful lab attacks staged in 2018, the two that broke ECC and the network attack potential of Throwhammer, may already have them in the field. They would just now be getting a sleek framework put together for Half-Double. Given the budgets and level of organization involved, the U.S. and China are likely the first two entities that will implement Rowhammer in the field.

But it’s possible that these governments haven’t made significant advancements past what Google was already able to achieve. Again, it depends on the budget they’re willing to throw at it.

And even if they���re already successful, or are about to be successful, in executing Rowhammer attacks in the wild, the public won’t hear about it for years. Unless they’re caught in the act and identified, or unless an internal leak happens, we won’t know anything about it.

So instead of guessing, we’ll move right onto the methods that you can mitigate the ability for public or private entities to use Rowhammer against you in a useful way.

Rowhammer Mitigation

In the face of uncertainty, reasonable precautions need to be taken. In that spirit, let’s discuss ways to mitigate the effectiveness of Rowhammer attacks.

On the hardware side, there’s some hope in the form of a technology called Error Check and Scrub (ECS). Because the spec for DDR5 includes on-die ECC3, the addition of ECS does provide some hope for the automatic detection and correction of bit flips, even three or more at a time.

It looks like at least some of Intel’s Alder Lake and Sapphire Rapids chips can support those DDR5 standards. Similarly, AMD’s Zen 4 chips plan to embrace DDR5. But don’t get your hopes up just yet. Aside from the fact that you can’t exactly trust every industry at its word, there’s a second layer that needs to embrace this change: The motherboard manufacturers. If they don’t also move quickly to support all of the features of DDR5 with ECC3 and advanced ECS, we’ll all be left waiting and hoping.

And of course, there’s legacy hardware out there with memory that can’t easily (or cheaply) be swapped: Routers, switches, firewalls. Tens of billions of dollars worth of Rowhammer-vulnerable hardware just sitting there, waiting to be hacked. So once again, we hope that this hardware will be phased out before the attacks become more mature. And once again, we wait.

In the meantime, monitoring is one of the main methods of understanding when a system is being attacked by Rowhammer. Turning on monitoring for uncached memory accesses in critical systems is step number one. These ‘cache misses’ can commonly be viewed in hardware performance monitors. A sudden spike in those might mean a Rowhammer attack.

A second measure can make Rowhammer attacks harder, or at least slower. Increasing the refresh rate of a system’s memory. Some forms of Rowhammer require relatively ‘long’ windows between refreshes to do their damage. Again, there’s no 100% mitigation against all types of Rowhammer attacks, but this can help in some cases.

Other solutions involve system virtualization and extra abstraction layers. Those are highly situational, so realistically using them as mitigation depends on what you’re using the computing power for. It’s certainly a discussion that can be had with your Cloud provider. However, this solution is unrealistic for most home users.

Any mitigation and increased logging is going to have some impact on system performance. How much detection and protection each individual needs against Rowhammer depends on whether or not they’re likely to be targeted. At the end of the day, every user needs to make that assessment for themself.

The Future Legacy of Rowhammer

Our role in reporting attacks like Rowhammer has nothing to do with ‘prediction’. It’s not done so that we can gloat sometime in the future when one or more governments get caught using such attacks. ‘We told you so’ is cold comfort when the critical systems that the world relies on start to crumble.

Our role in presenting these facts is to allow the reader to examine the past behavior of sophisticated government intelligence agencies, examine the power of Rowhammer, and draw a logical conclusion as to what comes next.

Understand that there are no ‘rules of fair play’ when it comes to government hacking, espionage, or surveillance. How many countless times in previous chapters have we called out governments for breaking the laws of their own lands to push forward their departmental agendas? Dozens? Hundreds by now, depending on whether or not you read the chapters in order?

If there is a vulnerability, government agencies will examine it in detail, and use it to their advantage whenever possible. We’ve never seen a case of internal communications, not leaked by Snowden or anyone else, that have said ‘This tactic is too mean’ or ‘This vulnerability goes too far’. To them, these are simply tools in a toolbox. Of course, they’re going to be using Rowhammer as soon as they have a viable framework to make it easy, affordable, and reliable.

So this isn’t a prediction, this is simply an opportunity for mitigation before either public or private sector entities fully take this attack into the wild. Rowhammer is just an example of hardware-level vulnerabilities that become possible as we scale smaller and smaller. As stated in Rowhammer: A Retrospective:

“RowHammer is an example, likely the first one, of a hardware failure mechanism that causes a practical and widespread system security vulnerability. As such, its implications on system security research are tremendous and exciting.” - Onur Mutlu and Jeremie S. Kim - Carnegie Mellon

You do not want security researchers to be excited. It’s never a good sign.

Google’s Half-Double and the Centauri attack aren’t the last in a series of discoveries about the vulnerability of DDR memory. They are stepping stones.

Bit flipping becomes more and more possible by the day, in every single device type that stores data magnetically or electromagnetically. If we continue down this route, someone will find a way to apply Rowhammer’s principles to home computer SSDs. Perhaps even to processor and video card caches.

ECC was never meant to handle attacks like this. It was designed to stop something called ‘soft errors’, a phenomenon that we’ve known about since the 1990s. They include rare events like alpha particle decay, cosmic ray impact, incidental crosstalk, and the like. DDR3 was fighting against a dragon with nothing but a paper sword. DDR4 upgraded that defense to a small buckler shield, but it could in no way compete with an attack as sophisticated and natively powerful as Rowhammer.

In short: We need the right tools for the job. And quickly.

Hopefully, the future of computing will be optical and holographic. Though optical mediums may come with their own ‘spooky’ aspects, they differ in the way the usable fields can be manipulated. So perhaps this is a long-term solution to attacks like Rowhammer, while also reducing a system’s heat production, and power consumption, and increasing storage density.

But we’re a long, long way off from cheap and affordable optical and holographic computing. If DDR5 motherboards don’t support the features that yield significant improvements against bit-flipping attacks, expect an era of Rowhammer and related exploits that include heavy usage by the more sophisticated government intelligence organizations throughout the world.

Stayed tuned for the next chapter. If you missed the last one, you can catch up right here.