Invented by JO; Myung Hyun
Memory is the heart of computers. How fast and smart a computer feels depends a lot on how it uses its memory. Today, let’s explore a new way to make computers use their memory better and cheaper, based on a recent patent application. We will look at why this is important, how it builds on older ideas, and what makes this invention stand out.
Background and Market Context
Computers are everywhere—phones, data centers, cloud servers, and even your fridge. The biggest computers, called warehouse-scale computers, help run the internet. These mega-machines power search engines, video sites, online stores, and more. They are made up of thousands of smaller computers (nodes) that work together.
But there’s a problem. Not all data is used the same way. Some data is “hot”—needed right now. Some is “cold”—not needed for a while. Most of the time, about a quarter of the data in these big computers is cold. But storing all data, even the cold stuff, in the fastest memory (like DRAM) is very expensive.
To save money, companies use “memory tiering.” This means they put hot data in fast, costly memory, and cold data in slower, cheaper memory. For years, this worked with two layers: fast DRAM for hot data and slow flash memory (like NAND) for cold data. But now, this is not enough. Data is growing fast, and the ways we use data are changing. Some data is not quite hot or cold—it’s “warm.” Also, many new types of memory have appeared, each with their own pros and cons.
Cloud providers and big companies want computers that are both quick and cheap. They want to make sure expensive memory is used only where it matters most. This is not just about saving money; it is about keeping up with growing demand, staying green, and making sure services are always fast. The need for smarter memory use is greater than ever.
Scientific Rationale and Prior Art
The idea of tiered memory is not new. For years, engineers have used two main types of memory in big computers:
– DRAM: Very fast, but expensive and uses more power.
– NAND Flash: Slower, but much cheaper and stores more data.
Older systems would look at how often data was used (frequency) and decide if it was hot (used a lot) or cold (used a little). The hot data stayed in DRAM, and the cold data was moved to flash. This made sense when there were only two kinds of memory and the rules were simple.
But as computers and applications have changed, these old ways fall short. Here’s why:
1. More Types of Memory: Now, there are more options beyond just DRAM and flash. There’s SRAM (even faster than DRAM), and new types being developed. Each has its own speed, cost, and how long it lasts.
2. More Data Temperatures: Data isn’t just hot or cold. Some data is “warm”—not used every second, but more than cold data. The two-level approach can’t handle this middle ground well.
3. Changing Workloads: With different apps running on the same computer, what counts as hot or cold can change. For some apps, “hot” means used in the last minute; for others, it means used many times in the past hour.
4. Endurance Issues: Flash memory wears out over time, especially if written to often. If the system keeps moving data back and forth without care, it can wear out flash drives early.
5. Performance Problems: If the system doesn’t move the right data at the right time, users may see delays. If cold data stays in fast memory, it wastes space. But if hot data is stuck in slow memory, apps get slow.
Some past inventions tried to improve memory tiering. They allowed for more than two layers, or more ways to decide if data is hot or cold. But many were hard to set up, slow, or could not be easily tuned for different apps. Others did not address endurance or made it hard to scale the system as needs changed.
The biggest challenge has been to make memory tiering smart, flexible, and automatic—without making computers slower or more complex to manage. This is where the new patent application shines.
Invention Description and Key Innovations
This new patent application offers a smart, easy, and flexible way to handle memory in modern computers, big and small. Let’s break down how it works and why it’s different.
1. Three Tiers of Memory
The system uses three layers of memory:
– Tier 1 Memory: This is the fastest (like DRAM).
– Tier 2 Memory: Fast, but not as fast as Tier 1 (can also be DRAM or other fast memory).
– Tier 3 Memory: This is slower but much cheaper (like NAND flash).
The system can also have a “Tier 1 candidate memory,” which acts as a waiting room for the hottest data before it moves to Tier 1. This is often made of very fast SRAM.
2. Smart Memory Movement (Tiering)
The core of this invention is the “memory tiering module.” This part of the system decides where each piece of data should live. It does this in two ways:
– Inter-memory tiering: Moves hot data between Tier 1 and the memory module (Tiers 2 and 3).
– Intra-memory tiering: Moves warm and cold data between Tier 2 and Tier 3 inside the memory module itself.
This two-way approach lets the system act quickly and keep the right data in the right place, without bothering the main processor more than needed.
3. Tracking Data “Temperature”
The system does not just look at how often something is used. It can watch many things, like:
– How often the data is used (frequency)
– How recently it was used (recency)
– Even other traits, like file type
The rules for what counts as hot, warm, or cold can be changed easily by the main computer (the host CPU) or by a smaller processor inside the memory module (the embedded CPU). This makes the system flexible for different apps and needs.
4. Fast, Independent Operation
The memory module can run its tiering work on its own, without waiting for the main CPU. For example, the embedded CPU can move cold data from Tier 2 to Tier 3 when space runs low, or promote warm data up when it starts to get used more. At the same time, the host CPU can ask for hot data to be moved to Tier 1.
This means the system can handle many apps and workloads at once, each with its own rules, without slowing down.
5. Customizable Algorithms
The system can use smart rules for moving data, like:
– Least Recently Used (LRU): Move out the data not used for the longest time.
– Least Frequently Used (LFU): Move out data used the least.
– Adaptive Replacement Cache (ARC): A mix of both.
It can even prefetch data it thinks will be needed soon, based on patterns.
6. Protecting Flash Memory (Endurance)
To make sure flash memory lasts, the system controls how often it writes to it. It can remove pages no longer needed and slow down writes if the flash is being used too much. It also keeps track of how many times flash has been written to, to make sure it stays within safe limits.
7. Easy to Control and Grow
The system comes with many easy-to-use commands (APIs) for setting up and checking how memory is being used. Each “tiering instance” can be controlled on its own, so different parts of the computer can use memory differently. This makes the system easy to scale up as data grows.
8. Works with Modern Hardware and Standards
The system is designed to work with CXL (Compute Express Link), a new standard that helps CPUs connect to different types of memory quickly. While the patent uses CXL as an example, it can work with other types of memory hardware too.
9. Real-World Example
Imagine a data center running search, email, and video all at once. The search engine needs fast access to the most recent searches (hot data), while old emails and videos are not needed right now (cold data). This system can make sure the hottest data is always in DRAM, warm data in the next layer, and cold data in flash. If the search engine starts needing old emails more, the system notices and moves them up to faster memory, all by itself.
10. Key Benefits
– Saves money by using less expensive memory where possible.
– Keeps computers fast and responsive by making sure hot data is always close by.
– Makes it easy to adjust to new apps or changing needs.
– Protects hardware by controlling how much it is used.
– Grows with your data, without needing a full redesign.
Conclusion
As computers get bigger and data keeps growing, smart memory management becomes more important than ever. This new patent application offers a way to use every type of memory to its fullest, keeping costs down and speed up. Its flexible, smart, and automatic system makes it perfect for the fast-changing world of cloud computing, data centers, and beyond.
By letting the computer move data to the right place at the right time, it helps companies serve billions of users without waste. This is not just a small step forward—it is a big leap for how we think about memory in computers. For engineers, cloud providers, and hardware makers, this invention opens the door to faster, smarter, and more affordable computing.
Click here https://ppubs.uspto.gov/pubwebapp/ and search 20250363046.