The Other IBM Big Iron That Is On The Horizon
The Hot Chips conference is underway this week, historically at Stanford University but this year as was the case last year, is being done virtually thanks to the coronavirus pandemic. There are a lot of new chips that are being discussed in detail, and one of them is not the forthcoming Power10 chip from IBM, which is expected to make its debut sometime in September and which was one of the hot items at last year’s Hot Chips event.
Previewing IBM Telum Processor
The one processor that IBM is talking about, however, is the “Telum” z16 processor for System z mainframes, and unlike in times past, IBM is revealing the latest of its epically long line of mainframe central processing units (1964 through 2022, and counting) before they are launched in systems rather than after. We happen to think IBM had hoped to be able to ship the Telum processors and their System z16 machines before the end of 2021 and the transition from 10 nanometer to 7 nanometer processes at former foundry partner GlobalFoundries to 7 nanometer processes at current foundry partner Samsung has delayed the z16 introduction from its usual cadence. As it stands, the z16 chip will come out in early 2022, after the Power10 chips with fat cores (meaning eight threads per core and only 15 cores per chip) come to market. The skinny Power10 cores (four threads per core but 30 cores on a die) used in so-called “scale out” systems are not expected until the second quarter of 2022. It is rough to change foundries and processes and microarchitectures all at the same time, so a delay from the original plan for both z16 and Power10 are to be expected.
It will be up to a judge to accept IBM’s lawsuit against GlobalFoundries, which we talked about back in June, or not accept it, and it will be up to a jury to decide economic damages should Big Blue prevail and win its case in the courts. Or, Intel could buy GlobalFoundries and settle the case and have IBM as its foundry partner. There are a lot of possible scenarios here. The good news is that IBM and Samsung have been able to get the z16 and Power10 processors designed and are ramping production on the Samsung 7 nanometer process, trying to drive up yields. If IBM could not deliver these chips in the near term, it would not be saying anything at this point. Like when the process shrink with the Power6+ or the Power6+ were not panning out, for instance.
The Telum z16 processor is interesting from a technology standpoint because it shows what IBM can do and what it might do with future Power processors, and it is important from an economic standpoint because the System z mainframe still accounts for a large percentage of IBM’s revenues and an even larger share of its profits. (It is hard to say with any precision.) As the saying goes around here, anything that lets IBM Systems stronger helps IBM i last longer.
Besides, it is just plain fun to look at enterprise server chips. So, without further ado, take a gander at the Telum z16 processor:
According to Ross Mauri, the general manager of the System z product line, “Telum” refers to one of the weapons sported by Artemis, the Greek goddess of the hunt, known for bow hunting but also for her skill with the javelin. This particular javelin has to hit its enterprise target and help Big Blue maintain its mainframe customer base and make them enthusiastic about investing in new servers. The secret sauce in the Telum chip, as it turns out, will be an integrated AI accelerator chip that was developed by IBM Research and that has been modified and incorporated into the design, thus allowing for machine learning inference algorithms to be run natively and in memory alongside production data and woven into mainframe applications.
This is important, and bodes well for the Power10 chip, which is also getting its own native machine learning inference acceleration, albeit of a different variety. The z16 chip has an on-die mixed-precision accelerator for floating point and integer data, while the Power10 chip has a matrix math overlay for its vector math units. The net result is the same, however: Machine learning inference can stay within the compute and memory footprint of the server, and that means it will not be offloaded to external systems or external GPUs or other kinds of ASICs and will therefore be inside the bulwark of legendary mainframe security. There will be no compliance or regulatory issues because customer data that is feeding the machine learning inference and the response or recommendation from that inference will all be in the same memory space. For this reason, we have expected for a lot of machine learning inference to stay on the CPUs on enterprise servers, while machine learning training will continue to be offloaded to GPUs and sometimes other kinds of ASICs or accelerators. (FPGAs are a good alternative for inference.)
Partner Preview of Telium 7nm Processor
The Telum chip measures 530 square millimeters in area and weighs in at about 22.5 billion transistors. By Power standards, the z16 cores are big fat ones, with lots of registers, branch target table entries, and such, which is why IBM can only get eight fat cores on that die. The Power10 chip, which we have nick-named “Cirrus” because IBM had a lame name for it, using the same 7 nanometer transistors can get sixteen fat cores (and 32 skinny cores) on a die that weighs in at 602 square millimeters but has only 18 billion transistors. The Telum chip will have a base clock speed of more than 5 GHz, which is normal for recent vintages of mainframe CPUs.
A whole bunch of things have changed with the Telum design compared to the z14 and z15 designs. IBM has used special versions of the chip called Service Processors, or SPs, to act as external I/O processors, offloading from Central Processors, or CPs, which actually do the compute. With this design, IBM is doing away with this and tightly coupling the chips together with on-die interconnects, much as it has done with Power processors for many generations. Mainframe processors in recent years also had lots of dedicated L3 cache and an external L4 cache that also housed the system interconnect bus (called the X-Bus). The z15 chip implemented in GlobalFoundries 14 nanometer processes had a dozen cores and 256 MB of L3 cache, plus 4 MB of L2 data cache and 4 MB of L2 instruction cache allocated for each core. Each core had 128 KB of L1 instruction cache and 128 KB of instruction cache. It ran at 5.2 GHz, and supported up to 40 TB of RAID-protected DDR4 main memory across a complex of 190 active compute processors.
With the z16 design, the cache is being brought way down. Each core has only 32 MB of L2 cache, which is made possible in part because the branch predictors on the front end of the chip have been redesigned. The core has four pipelines and supports SMT2 multithreading, but it doesn’t have physical L3 cache or physical L4 cache any more. Rather, according to Christian Jacobi, distinguished engineer and chief architect of the z16 processor, it implements a virtual 256 MB L3 cache across those physical L2 caches and a virtual 2 GB cache across eight chips in a system drawer. How this cache is all carved up is interesting, and it speaks to the idea that caches often are inclusive anyway (meaning everything in L1 is in L2, everything in L3 is in L3, and everything in L3 is in L4), which is a kind of dark silicon. Why not determine the hierarchy on the fly based on actual workload needs?
To make this virtual L3 cache, there are a pair of 320 GB/sec rings. Two chips are linked together in a single package using a synchronous transfer interface, shown at the bottom two thirds of the Telum chip and four sockets of these dual-chip modules (DCMs) are interlinked in a flat, all-to-all topology through on-drawer interfaces and fabric controllers, which run across the top of the Telum chip. At the bottom left is the AI Accelerator, which has more than 6 teraflops of mixed precision integer and floating point processing power that is accessible through the z16 instruction set and is not using a weird offload model as is the case with CPUs that offload machine learning inference to GPUs, FPGAs, or custom ASICs. This accelerator, says Jacobi, takes up a little less real estate on the chip than a core does. And clearly, if IBM wanted to raise the ratio, it can add more accelerators. This ratio is interesting in that it shows how much AI inference that IBM expects – and that its customers expect – to be woven into their applications.
That is the key insight here.
This on-chip AI Accelerator has 128 compute tiles that can do 8-way half precision (FP16) floating point SIMD operations, which is optimized for matrix multiplication and convolutions used in neural network training. The AI Accelerator also has 32 compute tiles that implement 8-way FP16/FP32 units that are optimized for activation functions and more complex operations. The accelerator also has what IBM calls an intelligent prefetcher and write-back block, which can move data to an internal scratchpad at more than 120 GB/sec and that can store data out to the processor caches at more than 80 GB/sec. The two collections of AI math units have what IBM calls an intelligent data mover and formatter that prepares incoming data for compute and then write-back after it has been chewed on by the math units, and this has an aggregate of 600 GB/sec of bandwidth.
That’s an impressive set of numbers for a small block of chips, and a 32-chip complex (four sets of four-sockets of DCMs) can deliver over 200 teraflops of machine learning inference performance. (There doesn’t seem to be INT8 or INT4 integer support on this device, but don’t be surprised if IBM turns it on eventually, thereby doubling and quadrupling the inference performance for some use cases that have relatively coarse data.)
Jacobi says that a z16 socket with an aggregate of 16 cores will deliver 40 percent more performance than a z15 socket, which had 12 cores. If you do the math, 33 percent of that increase came from the core count increase; the rest comes from microarchitecture tweaks and process shrinks. We don’t expect the clock speed to be much more than a few hundred megahertz more than the frequencies used in the z15 chip, in fact. There may be some refinements in the Samsung 7 nanometer process further down the road that allow IBM to crank it up and boost performance with some kickers. The same thing could happen with Power10 chips, by the way.
One final though, and it is where the rubber hits the road with this AI Accelerator. A customer in the financial services industry worked with IBM to adapt its recurrent neural network (RNN) to the AI Accelerator, allowing it to do inference on the machine for a credit card fraud model. This workload was simulated on a z16 system simulator, so take it with a grain of salt. It illustrates the principle:
With only one chip, the simulated System z16 machine could handle 116,000 inferences per second with an average latency of 1.1 milliseconds, which is acceptable throughput and latency for a financial transaction not to be stalled by the fraud detection and for it to be done in real time rather than after the fact. With 32 chips in a full System z16 machine, the AI Accelerator could scale linearly, yielding 3.5 million inferences per second with an average latency of 1.2 milliseconds. That’s a scalability factor of 94.3 percent of perfect linear scaling, and we think this has as much to do with the flat, fast topology in the new z16 interconnect and with the flatter cache hierarchy as it has to do with the robustness of the AI Accelerator.
IBM updates its mainframe processor to help AI
IBM's Telum processor will have on-chip acceleration for artificial intelligence inferencing.
IBM has introduced a new CPU for its Z Series mainframe that’s designed for transactions like banking, training, insurance, customer interactions, and fraud detection.
The Telum processor was unveiled at the annual Hot Chips conference and has been in development for three years to provide high-volume, real-time inferencing needed for artificial intelligence.
The Telum design is very different from its System z15 predecessor. It features 8 CPU cores, on-chip workload accelerators, and 32MB of what IBM calls Level 2 semi-private cache. The L2 cache is called semi-private because it is used to build a shared virtual 256MB L3 connection between the cores on the chip. This is a 1.5x growth in cache size over the z15.
The CPU comes in a module design that includes two closely coupled Telum processors, so you get 16 cores per socket running at 5Ghz. IBM Z systems pack their processors in what are known as drawers, with four sockets per drawer. The Telum processor will be manufactured by Samsung using a 7nm process, as compared to the 14nm process used for the z15 processor.
Stopping Fraud
IBM mainframes are still heavily used in online transaction processing (OLTP) and one of the problems that bedevils OLTP is that fraud usually isn’t caught until after it is committed.
Doing real-time analysis on millions of transactions is just not doable, particularly when fraud analysis and detection is conducted far away from mission-critical transactions and data, IBM says. AI could help, but AI workloads have much larger computational requirements than operating workloads.
“Due to latency requirements, complex fraud detection often cannot be completed in real-time—meaning a bad actor could have already successfully purchased goods with a stolen credit card before the retailer is aware fraud has taken place,” the company said in a blog post announcing Telum.
So the new chip is designed for real-time, AI-specific financial workloads. Just how it will work is not exactly known. Telum-based z16 mainframes are not expected until the second half of 2022.
A brief overview of IBM’s new 7 nm Telum mainframe CPU
A typical Telum-powered mainframe offers 256 cores at a base clock of 5+GHz.
From the perspective of a traditional x86 computing enthusiast—or professional—mainframes are strange, archaic beasts. They're physically enormous, power-hungry, and expensive by comparison to more traditional data-center gear, generally offering less compute per rack at a higher cost.
IBM MainFrame Life Cycle History
This raises the question, "Why keep using mainframes, then?" Once you hand-wave the cynical answers that boil down to "because that's how we've always done it," the practical answers largely come down to reliability and consistency. As AnandTech's Ian Cutress points out in a speculative piece focused on the Telum's redesigned cache, "downtime of these [IBM Z] systems is measured in milliseconds per year." (If true, that's at least seven nines.)
IBM's own announcement of the Telum hints at just how different mainframe and commodity computing's priorities are. It casually describes Telum's memory interface as "capable of tolerating complete channel or DIMM failures, and designed to transparently recover data without impact to response time."
When you pull a DIMM from a live, running x86 server, that server does not "transparently recover data"—it simply crashes.
IBM Z-series architecture
Telum is designed to be something of a one-chip-to-rule-them-all for mainframes, replacing a much more heterogeneous setup in earlier IBM mainframes.
The 14 nm IBM z15 CPU that Telum is replacing features five total processors—two pairs of 12-core Compute Processors and one System Controller. Each Compute Processor hosts 256MiB of L3 cache shared between its 12 cores, while the System Controller hosts a whopping 960MiB of L4 cache shared between the four Compute Processors.
Five of these z15 processors—each consisting of four Compute Processors and one System Controller—constitutes a "drawer." Four drawers come together in a single z15-powered mainframe.
Although the concept of multiple processors to a drawer and multiple drawers to a system remains, the architecture inside Telum itself is radically different—and considerably simplified.
Telum architecture
Telum is somewhat simpler at first glance than z15 was—it's an eight-core processor built on Samsung's 7nm process, with two processors combined on each package (similar to AMD's chiplet approach for Ryzen). There is no separate System Controller processor—all of Telum's processors are identical.
From here, four Telum CPU packages combine to make one four-socket "drawer," and four of those drawers go into a single mainframe system. This provides 256 total cores on 32 CPUs. Each core runs at a base clockrate over 5 GHz—providing more predictable and consistent latency for real-time transactions than a lower base with higher turbo rate would.
Pockets full of cache
Doing away with the central System Processor on each package meant redesigning Telum's cache, as well—the enormous 960MiB L4 cache is gone, as well as the per-die shared L3 cache. In Telum, each individual core has a private 32MiB L2 cache—and that's it. There is no hardware L3 or L4 cache at all.
This is where things get deeply weird—while each Telum core's 32MiB L2 cache is technically private, it's really only virtually private. When a line from one core's L2 cache is evicted, the processor looks for empty space in the other cores' L2. If it finds some, the evicted L2 cache line from core x is tagged as an L3 cache line and stored in core y's L2.
OK, so we have a virtual, shared up-to-256MiB L3 cache on each Telum processor, composed of the 32MiB "private" L2 cache on each of its eight cores. From here, things go one step further—that 256MiB of shared "virtual L3" on each processor can, in turn, be used as shared "virtual L4" among all processors in a system.
Telum's "virtual L4" works largely the same way its "virtual L3" did in the first place—evicted L3 cache lines from one processor look for a home on a different processor. If another processor in the same Telum system has spare room, the evicted L3 cache line gets retagged as L4 and lives in the virtual L3 on the other processor (which is made up of the "private" L2s of its eight cores) instead.
AnandTech's Ian Cutress goes into more detail on Telum's cache mechanisms. He eventually sums them up by answering "How is this possible?" with a simple "magic."
IBM Telum Processor brings deep learning inference to enterprise workloads
AI inference acceleration
IBM's Christian Jacobi briefly outlines Telum's AI acceleration in this two-minute clip.
Telum also introduces a 6TFLOPS on-die inference accelerator. It's intended to be used for—among other things—real-time fraud detection during financial transactions (as opposed to shortly after the transaction).
In the quest for maximum performance and minimal latency, IBM threads several needles. The new inference accelerator is placed on-die, which allows for lower latency interconnects between the accelerator and CPU cores—but it's not built into the cores themselves, a la Intel's AVX-512 instruction set.
The problem with in-core inference acceleration like Intel's is that it typically limits the AI processing power available to any single core. A Xeon core running an AVX-512 instruction only has the hardware inside its own core available to it, meaning larger inference jobs must be split among multiple Xeon cores to extract the full performance available.
Telum's accelerator is on-die but off-core. This allows a single core to run inference workloads with the might of the entire on-die accelerator, not just the portion built into itself.
In a major refresh of its Z Series chips, IBM is adding on-chip AI acceleration capabilities to allow enterprise customers to perform deep learning inferencing while transactions are taking place to capture business insights and fight fraud in real-time.
IBM is set to unveil the latest Z chip Aug. 23 (Monday) at the annual Hot Chips 33 conference, which is being held virtually due to the ongoing COVID-19 pandemic. The company provided advance details in a media pre-briefing last week.
This will be the first Z chip, used in IBM's System Z mainframes, that won't follow a traditional numeric naming pattern used in the past. Instead of following the previous z15 chip with a z16 moniker, the new processor is being called IBM Telum.
This will be IBM's first processor to include on-chip AI acceleration, according to the company. Designed for customers across a wide variety of uses, including banking, finance, trading, insurance applications and customer interactions, the Telum processors have been in development for the past three years. The first Telum-based systems are planned for release in the first half of 2022.
Previewing IBM Telum Processor
Ross Mauri of IBM
One of the major strengths of the new Telum chips is that they are designed to enable applications to run efficiently where their data resides, giving enterprises more flexibility with their most critical workloads, Ross Mauri, the general manager of IBM Z, said in a briefing with reporters on the announcement before Hot Chips.
"From an AI point of view, I have been listening to our clients for several years and they are telling me that they can't run their AI deep learning inferences in their transactions the way they want to," said Mauri. "They really want to bring AI into every transaction. And the types of clients I am talking to are running 1,000, 10,000, 50,000 transactions per second. We are talking high volume, high velocity transactions that are complex, with multiple database reads and writes and full recovery for … transactions in banking, finance, retail, insurance and more."
By integrating mid-transaction AI inferencing into the Telum chips, it will be a huge breakthrough for fraud detection, said Mauri.
"You hear about fraud detection all the time," he said. "Well, I think we are going to be able to move from fraud detection to fraud prevention. I think this is a real game changer when it comes to the business world, and I am really excited about that."
Inside the Telum Architecture
Christian Jacobi of IBM
Christian Jacobi, an IBM distinguished engineer and the chief architect for IBM Z processor design, said that the Telum chip design is specifically optimized to run these kinds of mission critical, heavy duty transaction processing and batch processing workloads, while ensuring top-notch security and availability.
"We have designed this accelerator using the AI core coming from the IBM AI research center," in cooperation with the IBM Research team, the IBM Z chip design team and the IBM Z firmware and software development team, said Jacobi. It is the first IBM chip created using technology from the IBM Research AI hardware center.
"The goal for this processor and the AI accelerator was to enable embedding AI with super low latency directly into the transactions without needing to send the data off-platform, which brings all sorts of latency inconsistencies and security concerns," said Jacobi. "Sending data over a network, oftentimes personal and private data, requires cryptography and auditing of security standards that creates a lot of complexity in an enterprise environment. We have designed this accelerator to operate directly on the data using virtual address mechanisms and the data protection mechanisms that naturally apply to any other thing on the IBM Z processor."
To achieve the low latencies, IBM engineers directly connected the accelerator to the on-chip cache infrastructure, which can directly access model data and transaction data from the caches, he explained. "It enables low batch counts so that we do not have to wait for multiple transactions to arrive of the same model type. All of that is geared towards enabling millisecond range inference tasks so that they can be done as part of the transaction processing without impacting their service level agreements … and have the results available to be able to utilize the AI inference result as part of the transaction processing.
The new Telum chips are manufactured for IBM by Samsung using a 7nm Extreme Ultraviolet Lithography (EUV) process.
The chips have a new design compared to IBM's existing z15 chips, according to Jacobi. Today's z15 chips use dual chip modules, but the Telum chips will use a single module.
"Four of those modules will be plugged into one drawer, basically a motherboard with four sockets," he said. "In the prior generations, we used two different chip types – a processor chip and a system control chip that contained a physical L4 cache, and that system control hub chip also acted as a hub whenever two processor chips needed to communicate with each other."
Dropping the system control chip enabled the designers to reduce the latencies, he said. "When one chip needs to talk to another chip, we can implement a flat topology, where each of the eight chips in a drawer has a direct connection to the seven other chips in the drawer. That optimizes latencies for cache interactions and memory access across the drawer."
Jacobi said that the Telum chips will provide a 40 percent performance improvement at the socket level over the company's previous z15 chips, which is boosted in part by its extra cores.
Each Telum processor contains eight processor cores and use a deep super-scalar, out-of-order instruction pipeline. Running with more than 5GHz clock frequency, the redesigned cache and chip-interconnection infrastructure provides 32MB cache per core and can scale to 32 Telum chips. The dual-chip module design contains 22 billion transistors and 19 miles of wire on 17 metal layers.
Introducing the new IBM z15 T02
An Analyst Weighs In
Karl Freund, analyst
Karl Freund, founder and principal analyst of Cambrian AI Research, told EnterpriseAI that the upcoming Telum chips directly target enterprise users with a wide range of useful capabilities.
"In addition to a new cache architecture that will significantly pump-up performance, the Telum processor will provide Z customers with the ability to run real-time AI on the same platform that conducts the mission critical transactions and analyses on which enterprises depend," said Freund. "Until Telum, Z applications could run machine learning on the Z cores, or offload data to another platform for deep learning analysis. The latter introduces unacceptable latencies and significant costs, as well as introducing entirely new and now unnecessary security risk."
For customers, these could be compelling improvements, said Freund.
"I believe Telum provides the biggest reason we have seen in a while for applications to remain in the Z fold," he said. "Honestly, enterprise AI is about to get very real with the Telum processor. After all, the Z is the custodian of some of the most important data in an enterprise. Being able to run deep learning models directly on the Z will unlock tremendous value for Z clients, value that has been hidden until Telum."
What still needs to be determined, he said, is how it all will perform when the development work is completed. "We need to see whether the small accelerator at 'only' 6 TFLOPS can provide adequate processing power to eight very fast cores needing AI services. However, since the data involved here is highly structured numerical floating point or decimal data, instead of images, voice or video, I suspect the accelerator will prove up to the task."
More Details to Come Later
Jacobi said that IBM is not yet providing any additional system performance specifications until the company's engineers complete further Optimizations of its firmware and software stacks that will run on the systems.
"We will provide more performance gains out of Optimization across those layers of the entire stack when we get to that point next year," he said.
"I will add that one of the unique things about our design … is that every core when it does AI … are performing a hybrid – they are performing the complex transaction work including the databases and the business logic, and then switch over to perform AI as part of the transaction," said Jacobi. "When they switch over, they can use the entire capacity of the AI accelerator and not just a sliver that is assigned to the core. It is a dedicated piece of silicon on the chip. And the core can access that piece of silicon and use the entire compute capacity of that AI accelerator. That is important for us to achieve the low latencies that we need to make it all fit within the transaction response budget."
About That Z Series Name Change
The upcoming Telum chips will show up in IBM's Z Series and LinuxONE systems as the main processor chip for both product lines, said IBM's Mauri. They are not superseding the Z Series chips, he said.
So, does that mean that z16, z17 and other future Z chips are no longer on the roadmap?
"No, said Mauri. "It just means that we never named our chips before. We are naming it this time because we are proud of the innovation and breakthroughs and think that it is unique in what it does. And that is the only thing. I think z15 will still be around and there will be future generations, many future generations."
Still to Come: New z/OS
To prepare for the arrival of the Telum chips, IBM has already slated the debut of the next version of its updated z/OS 2.5 operating system for Z Series hardware sometime in September, according to an earlier report by The Register. The mainframe operating system is expected to get more AI and hybrid cloud features, as well as expanded co-processor support, based on a preview of the OS that was unveiled in March.