8 AI Technology Topics That Connect Back to Storage and Memory

GetUSB isn’t becoming an artificial intelligence website. There are already plenty of those, and most days the world probably doesn’t need another article explaining that AI is going to change everything.

What has caught our attention is something a little more practical.

The deeper you get into how artificial intelligence actually works, the more the conversation starts sounding familiar. Memory bandwidth matters. Storage matters. NAND flash matters. DRAM matters. PCIe matters. Moving enormous amounts of data from one place to another matters. These are subjects we’ve been following for years, long before every product announcement needed the letters “AI” somewhere in the first paragraph.

Over the past several months we’ve written a number of articles looking at AI from that hardware perspective. Rather than chasing every new model or software announcement, we’ve been interested in what has to happen underneath all of it.

Looking back, eight articles in particular fit together surprisingly well.

Portable AI Accelerators vs Cloud AI: What Small Businesses Should Actually Pick?

One of the more interesting changes in AI is that running an AI workload doesn’t automatically mean sending everything to a massive data center. Smaller AI accelerators are making local inference practical for applications that don’t require the horsepower of a giant cloud-based model.

In our article about portable AI accelerators versus cloud AI, we looked at the tradeoffs between the two approaches. Cloud AI provides enormous computing resources, but local AI brings some interesting advantages of its own, including privacy, predictable operating costs, lower latency and the ability to continue working without an internet connection.

The important point isn’t that one approach wins. It is that AI processing is starting to spread out. Some jobs belong in the cloud. Some make considerably more sense sitting on a local computer or edge device.

Inside an AI Computer: Why Modern AI Systems Consume So Much Memory

A GPU gets most of the glory in an AI computer, but a powerful processor isn’t particularly useful if it spends its time waiting for data.

That led us to look inside an AI computer and explain why modern AI systems consume so much memory. AI models contain huge numbers of parameters, and those parameters have to be stored, loaded and moved through the system while calculations are taking place.

This is one of the recurring themes behind AI hardware. Computing power has increased dramatically, but feeding that computing power has become a problem of its own. The processor might be capable of doing the work, but only if the rest of the system can keep up.

NAND Isn’t Going Away, But AI Servers Now Depend on More Than Flash

Given our background, NAND flash was an obvious place for us to start digging.

AI isn’t making NAND obsolete. Quite the opposite. Models, training data, databases and generated information all need persistent storage, and flash remains extremely important. But AI servers now depend on much more than NAND flash because storage capacity is only one part of the problem.

An SSD can hold a tremendous amount of information, but the processor can’t perform calculations directly against everything sitting on an SSD at full speed. Data has to move through different levels of the memory hierarchy, each with its own balance of capacity, speed and cost.

AI has made that hierarchy much more visible.

What Is High Bandwidth Memory and Why Does AI Depend on It?

High Bandwidth Memory, usually shortened to HBM, went from being a fairly specialized memory technology to something regularly mentioned in discussions about AI hardware.

There’s a good reason.

In our explanation of what High Bandwidth Memory is and why AI depends on it, we looked at the problem HBM is designed to solve. Modern GPUs can perform an extraordinary number of calculations, but conventional memory architectures can become a bottleneck when those processors need enormous amounts of data delivered continuously.

HBM puts extremely fast memory very close to the processor and provides much greater memory bandwidth than conventional memory arrangements. It is expensive and complicated, but when you’re trying to keep an AI accelerator fed with data, bandwidth can be just as important as raw processing speed.

Think of it as owning a very fast factory. Making the machines inside the factory faster doesn’t accomplish much if the loading dock can’t deliver raw material quickly enough.

Why DRAM Alone Can’t Keep Up With AI Anymore

DRAM isn’t disappearing either. There is simply too much data.

We explored this in why DRAM alone can’t keep up with AI anymore, because the obvious solution to a memory shortage would seem to be installing more memory. Unfortunately, at AI scale, that gets expensive very quickly and still doesn’t solve every bandwidth and capacity problem.

This is where the different layers of AI storage start making more sense. HBM can provide extraordinary bandwidth close to the processor. DRAM provides larger pools of fast working memory. NAND flash provides much greater persistent capacity. Each technology does something the others can’t do economically.

AI isn’t replacing one type of memory with another. If anything, it is forcing computers to use more types of memory at the same time.

Why Hard Drives Are Still Critical for AI Infrastructure

This one might be my favorite contradiction in the whole AI hardware story.

We spend all this time talking about incredibly sophisticated GPUs, HBM, NVMe storage and advanced semiconductor manufacturing, and then sitting farther down the chain is technology based on spinning magnetic platters.

The hard drive is very much alive.

In why hard drives are still critical for AI infrastructure, we looked at the enormous amount of information AI companies need to retain. Training datasets, model checkpoints, backups, archives and other bulk data don’t all need to sit on expensive high-performance flash storage.

At some point economics takes over.

When you’re storing information by the petabyte, cost per terabyte becomes awfully important. Hard drives remain one of the most economical ways to provide enormous amounts of online storage. AI may represent some of the most advanced computing technology we’ve ever built, but apparently it still needs a really big closet to put everything in.

Why AI Is Moving Compute Closer to Storage

For decades, the normal computer architecture has been fairly straightforward: store the data somewhere, move it into memory, send it to the processor and do something with it.

That works beautifully until the amount of data being moved becomes enormous.

Our article about why AI is moving compute closer to storage looks at an interesting response to that problem. Instead of constantly moving huge datasets back and forth across a system, some architectures are beginning to move portions of the computation closer to where the data already lives.

There is a simple logic behind this. Moving data consumes time, bandwidth and power. If you can reduce the distance that data needs to travel, you can sometimes improve the efficiency of the entire system without simply throwing a faster processor at the problem.

This is another place where AI is changing the traditional definition of computer performance. The fastest component isn’t necessarily the answer. Sometimes the bigger improvement comes from moving less data.

KV Cache: The AI Memory Reservoir Keeping GPUs From Running Dry

KV cache sounds like one of those subjects you can safely ignore until you realize it helps explain why AI inference can consume such surprising amounts of memory.

When a language model generates a response, it doesn’t want to recalculate everything it already processed every time it produces the next token. The KV cache acts as a memory reservoir, retaining information from previous calculations so the model can reuse it.

That’s great for performance. It isn’t free.

As conversations become longer and more users interact with a model, KV cache requirements can grow dramatically. Once again, we arrive at the same AI hardware problem from a different direction: where do you put all this information, and how quickly can you get it back to the processor?

KV cache is a good example of why looking only at GPU specifications doesn’t tell you very much about the performance of an entire AI system.

Apparently We Keep Coming Back to Memory and Storage

When we wrote these articles, there wasn’t a plan to build an eight-part AI series. The subjects simply kept crossing paths with the technology we already follow.

That might actually be the interesting part.

Strip away the chatbots, generated images and futuristic predictions and artificial intelligence becomes a massive data-processing problem. Data needs to be stored. It needs to be moved. It needs to be placed into memory. It needs to reach processors fast enough to keep extremely expensive hardware busy.

Suddenly NAND, DRAM, HBM, hard drives, PCIe and storage architecture don’t sound like side topics. They’re part of the machinery making modern AI possible.

And lately we’ve started looking past even the hardware.

Our more recent article, The Biggest AI Bottleneck Isn’t Software – It’s Electricians, takes the story outside the server. Building AI infrastructure requires enormous amounts of electrical power, cooling and physical construction. At some point you can have the GPUs, memory and storage sitting on a loading dock and still have a problem: somebody has to build the infrastructure required to turn all of it on.

That is probably where our AI coverage will continue to go. Not chasing every new AI announcement, but following the places where AI bumps into storage, hardware and the physical realities of computing.

Because those are usually the parts of the story that interest us anyway.