Transaction Processing Performance Council Launches AI and ML Benchmark

TPC’s downloadable kit gives companies the chance to test AI scenarios.

The Transaction Processing Performance Council, a San Francisco-based nonprofit corporation founded in 1985, has developed a new benchmark standard for artificial intelligence (AI), TPCx-AI.  The vendor-neutral benchmark measures the speed and reliability of real-life AI and machine learning (ML) scenarios and data science use cases.

Hamesh Patel, chair of the TPCx-AI committee and principal engineer at Intel Corporation, said TPCx-AI is designed to emulate real-world examples of organizations that use a variety of production-ready data science pipelines.

“[TPCx-AI] is now widely available to anyone who would like to download and run it. We look forward to feedback as industry experts, academics and others interested in benchmarking system performance begin to use it,” said Patel.

Patel added that the TPCx-AI benchmark is the result of collaboration between talented engineers and researchers at leading AI organizations. The TPC members that contributed to the development of TPCx-AI include Intel, Cisco, Dell, HP Enterprise, IBM, Microsoft, Red Hat, TTA and VMware.

TPCx-AI uses a diverse dataset and is designed to be adaptable across a wide range of scale factors. It can evaluate performance for a system under test (SUT) that accomplishes one or more of the following tasks: generating and processing large volumes of data, training preprocessed data to produce realistic ML models, conducting accurate insights for real-world customer scenarios based on the generated models, scaling for large-scale distributed configurations, and allowing for flexibility in configuration changes to meet the demands of the dynamic AI landscape.

The benchmark also measures end-to-end time, the time it takes a system to deliver a complete functional solution from beginning to end. Further, TPCx-AI measures throughput metrics, how many units of information a system can process in a given amount of time, to simulate multiuser environments for a hardware, operating system and data processing system configuration under a controlled, complex, multiuser AI or ML data science workload.

The TPC has classified TPCx-AI as an “Express” class benchmark, meaning that the benchmark is an executable kit that can be deployed and measured. The kit is available for download at the TPC website at: http://tpc.org/tpcx-ai/default5.asp.

The TPC itself has 21 full members, including Actian, Alibaba, AMD, Cisco, Dell, Fujitsu, Hitachi, Huawei, Inspur, Lenovo, Nutanix, Nvidia, Oracle, and Transwarp, along with four associate members: China Academy of Information and Communications Technology (CAICT), Gartner, Imec, and the University of Coimbra.

The TPC has been working to develop benchmark standards for AI since at least 2018. Benchmark standards help industry members to share objective and verifiable performance data with the industry. They benefit the industry by allowing software developers to create systems that are faster, less expensive and more energy efficient.

In the past, the TPC created benchmark standards for transaction processing, decision support systems, and virtualization. More recently, the TPC developed benchmark standards for big data analytics (BDA), the Internet of Things (IoT), database virtualization, and Hyper-converged infrastructure (HCI). Organizations interested in contributing to the TPC’s benchmark development process are encouraged to become members. Additional information is available at the following URL: http://tpc.org/information/about/join5.asp.