How big is your simulation, and how fast do you want it done?
When Rescale announced its new high throughput computing (HTC) initiative this summer, some users—rightfully—may have been confused. The August 22 announcement specifically boasted the possibility of running “millions of modeling, simulation or data analysis tasks” at the same time by leveraging HTC and cloud services. This sounds intriguing, but Rescale already focuses on running many simulations quickly using cloud high performance computing (HPC) resources. So, how is HTC any different?
Simulation engineers and small to medium-sized business owners are constantly working with competing constraints and using limited resources in the best way possible. Once these users understand HTC, they will realize that its capabilities are another way to distribute computational workloads and create the best possible outcomes for a simulation department.
The Differences Between HTC and HPC
The differences between HTC and HPC focus primarily on the nature of the study an engineer is running. HTC is built for studies where many small and independent computations are run. If you had a thousand small tasks that each took 10 minutes to complete, then instead of running all these tasks on one machine and requiring 10,000 minutes, you could harness HTC and 1,000 different resources to finish the job in 10 minutes.
Electronic design automation is a big customer for HTC, where several different component sizes, parameters and layouts for printed circuit boards can all be analyzed at the same time to find optimized results. Rescale called out in its August announcement that “semiconductor verification analysis, bioinformatics pipelines and design space exploration” are all areas that can hit the ground running with HTC technology. But it should be noted that artificial intelligence can also benefit here.
HPC, on the other hand, is best reserved for big, dense studies. When a simulation brings together dozens or hundreds of interconnected computations, HPC resources can help. Instead of waiting for your on-premises resources to chug through that big, gnarly study, you can use HPC resources through the cloud and have an answer faster. Several factors are at work here, but when you’re using HPC resources, engineers can imagine that they are working at the supercomputer level that requires large-scale computer clusters. In other words, HPC focuses on the interconnectedness of the components and parameters in a study and runs several calculations at the same time in parallel to produce faster results.
Could an engineer use HTC resources on an HPC project, or the other way around? The possibility exists, but it would be an ill-advised use of expensive resources. Trying to use the smaller-scale resources from a high throughput study to solve a heavy-lifting project suited to HPC would bog down the processors and go slower than needed. On the other hand, taking an HPC supercomputer and solving multiple small-scale HTC level problems could be incredibly expensive and an example of resource overkill.
As HPC and HTC are both useful, a convergence might happen at some point. If most of the studies that a company runs are large and unwieldy, requiring HPC, running several of these studies at the same time could benefit from an HTC-style architecture. Computing power gets faster year after year, and if we had the opportunity to run 1,000 supercomputer-level simulations without breaking the bank or slowing the study to a crawl, there’s no reason to think we’d turn it down.
The Similarities Between HTC and HPC
The big similarities between HTC and HPC are the benefits users receive when accessing these technologies from the cloud. A Hyperion Research study from 2023 titled “Engineering Simulation Workloads and the Rise of the Cloud” showed four major benefits for companies when they move to a cloud-based computing setup:
- The scalable properties of cloud computing mean that when a company has more business or needs more resources for a specific job, the cloud is ready with more capacity.
- There are cost savings when heavy-duty servers are at the cloud service’s location instead of purchased at the customer’s location.
- There is the freedom of having IT policies dictate when to move to the cloud for specific processor-heavy functions. These policies might deal with heightened data security or the need for to have fewer IT professionals on premises when many of the IT functions are dealt with on the cloud side.
- The users get access to specialized hardware or software at the cloud location instead of being limited to the equipment at the home site.
It isn’t hard to see how HPC and HTC can both benefit from these cloud improvements. And it therefore isn’t hard to see why Rescale is excited about its new HTC capabilities.
What Does It All Mean?
The basic goals of simulation are the same now as they were 10 years ago. Engineers are using simulation to understand more about components and systems design and performance. They are seeking to find potential issues or gains early in the development process. They want these results quickly, with a high degree of faith in the outcome, and reassurance that doing the work won’t compromise data or IT systems.
The tools we use for these simulations, however, are much different than they were a decade ago—and are changing all the time. For instance, the rise of AI as a tool for simulation engineers means that we must change not just how we run our studies but also the framework on which we run them. HTC gives engineers and scientists the ability to explore a high number of computations and can then tune a design accordingly. HPC helps to run simulations that require a high degree of computing power for a single result. Both technologies are starting to run through the cloud on servers with more bandwidth than most businesses have in-house.
Both product development and research are full of constant design trade-offs. As a result, engineers try to find the best combination of several variables to come to achieve a result that isn’t necessarily perfect but better than all the other options. HTC and HPC both offer a strong set of tools to help engineers achieve these goals.
