Quantum Volume Testing: Setting the Steady Pace to Higher Performing Devices
When it comes to completing the statistical tests and other steps necessary for calculating quantum volume, few people have as much as experience as Dr. Charlie Baldwin.
Baldwin, a lead physicist at Quantinuum, and his team have performed the tests numerous times on three different H-Series quantum computers, which have set six industry records for measured quantum volume since 2020.
Quantum volume is a benchmark developed by IBM in 2019 to measure the overall performance of a quantum computer regardless of the hardware technology. (Quantinuum builds trapped ion systems).
Baldwin’s experience with quantum volume prompted him to share what he’s learned and suggest ways to improve the benchmark in a peer-reviewed paper published this week in Quantum.
“We’ve learned a lot by running these tests and believe there are ways to make quantum volume an even stronger benchmark,” Baldwin said.
We sat down with Baldwin to discuss quantum volume, the paper, and the team’s findings.
How is quantum volume measured? What tests do you run?
Quantum volume is measured by running many randomly constructed circuits on a quantum computer and comparing the outputs to a classical simulation. The circuits are chosen to require random gates and random connectivity to not favor any one architecture. We follow the construction proposed by IBM to build the circuits.
What does quantum volume measure? Why is it important?
In some sense, quantum volume only measures your ability to run the specific set of random quantum volume circuits. That probably doesn’t sound very useful if you have some other application in mind for a quantum computer, but quantum volume is sensitive to many aspects that we believe are key to building more powerful devices.
Quantum computers are often built from the ground up. Different parts—for example, single- and two-qubit gates—have been developed independently over decades of academic research. When these parts are put together in a large quantum circuit, there’re often other errors that creep in and can degrade the overall performance. That’s what makes full-system tests like quantum volume so important; they’re sensitive to these errors.
Increasing quantum volume requires adding more qubits while simultaneously decreasing errors. Our quantum volume results demonstrate all the amazing progress Quantinuum has made at upgrading our trapped-ion systems to include more qubits and identifying and mitigating errors so that users can expect high-fidelity performance on many other algorithms.
You’ve been running quantum volume tests since 2020. What is your biggest takeaway?
I think there’re a couple of things I’ve learned. First, quantum volume isn’t an easy test to run on current machines. While it doesn’t necessarily require a lot of qubits, it does have fairly demanding error requirements. That’s also clear when comparing progress in quantum volume tests across different platforms, which researchers at Los Alamos National Lab did in a recent paper.
Second, I’m always impressed by the continuous and sustained performance progress that our hardware team achieves. And that the progress is actually measurable by using the quantum volume benchmark.
The hardware team has been able to push down many different error sources in the last year while also running customer jobs. This is proven by the quantum volume measurement. For example, H1-2 launched in Fall 2021 with QV=128. But since then, the team has implemented many performance upgrades, recently achieving QV=4096 in about 8 months while also running commercial jobs.
What are the key findings from your paper?
The paper is about four small findings that when put together, we believe, give a clearer view of the quantum volume test.
First, we explored how compiling the quantum volume circuits scales with qubit number and, also proposed using arbitrary angle gates to improve performance—an optimization that many companies are currently exploring.
Second, we studied how quantum volume circuits behave without errors to better relate circuit results to ideal performance.
Third, we ran many numerical simulations to see how the quantum volume test behaved with errors and constructed a method to efficiently estimate performance in larger future systems.
Finally, and I think most importantly, we explored what it takes to meet the quantum volume threshold and what passing it implies about the ability of the quantum computer, especially compared to the requirements for quantum error correction.
What does it take to “pass” the quantum volume threshold?
Passing the threshold for quantum volume is defined by the results of a statistical test on the output of the circuits called the heavy output test. The result of the heavy output test—called the heavy output probability or HOP—must have an uncertainty bar that clears a threshold (2/3).
Originally, IBM constructed a method to estimate that uncertainty based on some assumptions about the distribution and number of samples. They acknowledged that this construction was likely too conservative, meaning it made much larger uncertainty estimates than necessary.
We were able to verify this with simulations and proposed a different method that constructed much tighter uncertainty estimates. We’ve verified the method with numerical simulations. The method allows us to run the test with many fewer circuits while still having the same confidence in the returned estimate.
How do you think the quantum volume test can be improved?
Quantum volume has been criticized for a variety of reasons, but I think there’s still a lot to like about the test. Unlike some other full-system tests, quantum volume has a well-defined procedure, requires challenging circuits, and sets reasonable fidelity requirements.
However, it still has some room for improvement. As machines start to scale up, runtime will become an important dimension to probe. IBM has proposed a metric for measuring run time of quantum volume tests (CLOPS). We also agree that the duration of the computation is important but that there should also be tests that balance run time with fidelity, sometimes called ‘time-to-solution.”
Another aspect that could be improved is filling the gap between when quantum volume is no longer feasible to run—at around 30 qubits—and larger machines. There’s recent work in this area that will be interesting to compare to quantum volume tests.
You presented these findings to IBM researchers who first proposed the benchmark. How was that experience?
It was great to talk to the experts at IBM. They have so much knowledge and experience on running and testing quantum computers. I’ve learned a lot from their previous work and publications.
There is a lot of debate about quantum volume and how long it will be a useful benchmark. What are your thoughts?
The current iteration of quantum volume definitely has an expiration date. It’s limited by our ability to classically simulate the system, so being unable to run quantum volume actually is a goal for quantum computing development. Similarly, quantum volume is a good measuring stick for early development.
Building a large-scale quantum computer is an incredibly challenging task. Like any large project, you break the task up into milestones that you can reach in a reasonable amount of time.
It's like if you want to run a marathon. You wouldn’t start your training by trying to run a marathon on Day 1. You’d build up the distance you run every day at a steady pace. The quantum volume test has been setting our pace of development to steadily reach our goal of building ever higher performing devices.
About Quantinuum
Quantinuum, the world’s largest integrated quantum company, pioneers powerful quantum computers and advanced software solutions. Quantinuum’s technology drives breakthroughs in materials discovery, cybersecurity, and next-gen quantum AI. With over 500 employees, including 370+ scientists and engineers, Quantinuum leads the quantum computing revolution across continents.
For a novel technology to be successful, it must prove that it is both useful and works as described.
Checking that our computers “work as described” is called benchmarking and verification by the experts. We are proud to be leaders in this field, with the most benchmarked quantum processors in the world. We also work with National Laboratories in various countries to develop new benchmarking techniques and standards. Additionally, we have our own team of experts leading the field in benchmarking and verification.
Currently, a lot of verification (i.e. checking that you got the right answer) is done by classical computers – most quantum processors can still be simulated by a classical computer. As we move towards quantum processors that are hard (or impossible) to simulate, this introduces a problem: how can we keep checking that our technology is working correctly without simulating it?
We recently partnered with the UK’s Quantum Software Lab to develop a novel and scalable verification and benchmarking protocol that will help us as we make the transition to quantum processors that cannot be simulated.
This new protocol does not require classical simulation, or the transfer of a qubit between two parties. The team’s “on-chip” verification protocol eliminates the need for a physically separated verifier and makes no assumptions about the processor’s noise. To top it all off, this new protocol is qubit-efficient.
The team’s protocol is application-agnostic, benefiting all users. Further, the protocol is optimized to our QCCD hardware, meaning that we have a path towards verified quantum advantage – as we compute more things that cannot be classically simulated, we will be able to check that what we are doing is right.
Running the protocol on Quantinuum System Model H1, the team ended up performing the largest verified Measurement Based Quantum Computing (MBQC) circuit to date. This was enabled by our System Model H1’s low cross-talk gate zones, mid-circuit measurement and reset, and long coherence times. By performing the largest verified MBQC computation to date, and by verifying computations significantly larger than any others to be verified before, we reaffirm the Quantinuum Systems as best-in-class.
Particle accelerators like the LHC take serious computing power. Often on the bleeding-edge of computing technology, accelerator projects sometimes even drive innovations in computing. In fact, while there is some controversy over exactly where the world wide web was created, it is often attributed to Tim Berners-Lee at CERN, who developed it to meet the demand for automated information-sharing between scientists in universities and institutes around the world.
With annual data generated by accelerators in excess of exabytes (a billion gigabytes), tens of millions of lines of code written to support the experiments, and incredibly demanding hardware requirements, it’s no surprise that the High Energy Physics community is interested in quantum computing, which offers real solutions to some of their hardest problems. Furthermore, the HEP community is well-positioned to support the early stages of technological development: with budgets in the 10s of billions per year and tens of thousands of scientists and engineers working on accelerator and computational physics, this is a ripe industry for quantum computing to tap.
As the authors of this paper stated: “[Quantum Computing] encompasses several defining characteristics that are of particular interest to experimental HEP: the potential for quantum speed-up in processing time, sensitivity to sources of correlations in data, and increased expressivity of quantum systems... Experiments running on high-luminosity accelerators need faster algorithms; identification and reconstruction algorithms need to capture correlations in signals; simulation and inference tools need to express and calculate functions that are classically intractable”
The authors go on to state: “Within the existing data reconstruction and analysis paradigm, access to algorithms that exhibit quantum speed-ups would revolutionize the simulation of large-scale quantum systems and the processing of data from complex experimental set-ups. This would enable a new generation of precision measurements to probe deeper into the nature of the universe. Existing measurements may contain the signatures of underlying quantum correlations or other sources of new physics that are inaccessible to classical analysis techniques. Quantum algorithms that leverage these properties could potentially extract more information from a given dataset than classical algorithms.”
Our scientists have been working with a team at DESY, one of the world’s leading accelerator centers, to bring the power of quantum computing to particle physics. DESY, short for Deutsches Elektronen-Synchrotron, is a national research center for fundamental science located in Hamburg and Zeuthen, where the Center for Quantum Technologies and Applications (CQTA) is based. DESY operates, develops, and constructs particle accelerators used to investigate the structure, dynamics and function of matter, and conducts a broad spectrum of interdisciplinary scientific research. DESY employs about 3,000 staff members from more than 60 nations, and is part of the worldwide computer network to store and analyze the enormous flood of data that is produced by the LHC in Geneva.
In a recent paper, our scientists collaborated with scientists from DESY, the Leiden Institute of Advanced Computer Science (LIACS), and Northeastern University to explore using a generative quantum machine learning model, called a “quantum Boltzmann machine” to untangle data from CERN’s LHC.
The goal was to learn probability distributions relevant to high energy physics better than the corresponding classical models. The data specifically contains “particle jet events”, which describe how colliders collect data about the subatomic particles generated during the experiments.
In some cases the quantum Boltzmann machine was indeed better, compared to a classical Boltzmann machine. The team is analyzed when and why this happens, understanding better how to apply these new quantum tools in this research setting. The team also studied the effect of the data encoding into a quantum state, noting that it can have a decisive effect on the training performance. Especially enticing is that the quantum Boltzmann machine is efficiently trainable, which our scientists showed in a recent paper published in Nature Communications Physics.
Find the Quantinuum team at this year’s SC24 conference from November 17th – 22nd in Atlanta, Georgia. Meet our team at Booth #4351 to discover how Quantinuum is bridging the gap between quantum computing and high-performance compute with leading industry partners.
The Quantinuum team will be participating various events, panels and poster sessions to showcase our quantum computing technologies. Join us at the below sessions:
Monday, Nov 18, 8:00 - 8:25pm, EST
Panel: KAUST booth 1031
Nash Palaniswamy, Quantinuum’s CCO, will join a panel discussion with quantum vendors and KAUST partners to discuss advancements in quantum technology.
Monday, Nov 18, 9:00 - 11:59pm, EST
Beowulf Bash: World of Coca-Cola
This year, we are proudly sponsoring the Beowulf Bash, a unique event organized to bring the HPC community together for a night of unique entertainment! Join us at the event on Monday, November 18th, 9:00pm at the World of Coca-Cola.
Wednesday, Nov 20, 3:30 – 5:00pm, EST
Panel: Educating for a Hybrid Future: Bridging the Gap between High-Performance and Quantum Computing
Vincent Anandraj, Quantinuum’s Director of Global Ecosystem and Strategic Alliances, will moderate this panel which brings together experts from leading supercomputing centers and the quantum computing industry, including PSC, Leibniz Supercomputing Centre, IQM Quantum Computers, NVIDIA, and National Research Foundation.
Thursday, Nov 21, 11:00 – 11:30am, EST
Presentation: Realizing Quantum Kernel Models at Scale with Matrix Product State Simulation
Pablo Andres-Martinez, Research Scientist at Quantinuum, will present research done in collaboration with HSBC, where the team applied quantum methods to fraud detection.