Scientists at Sandia National Labs invent new yardstick for benchmarking performance.
What does a quantum computer have in common with a top draft pick in sports? Both have attracted lots of attention from talent scouts. Quantum computers, experimental machines that can perform some tasks faster than supercomputers, are constantly evaluated, much like young athletes, for their potential to someday become game-changing technology.
Now, scientist-scouts have their first tool to rank a prospective technology’s ability to run realistic tasks, revealing its true potential and limitations.
A new kind of benchmark test, designed at Sandia National Laboratories, predicts how likely it is that a quantum processor will run a specific program without errors.
The so-called mirror-circuit method, published today (December 20, 2021) in Nature Physics, is faster and more accurate than conventional tests, helping scientists develop the technologies that are most likely to lead to the world’s first practical quantum computer, which could greatly accelerate research for medicine, chemistry, physics, agriculture, and national security.
Until now, scientists have been measuring performance on obstacle courses of random operations.
Sandia National Laboratories has designed a faster, more accurate style of test for quantum computers, such as the one pictured here. Credit: Bret Latter, Sandia National Laboratories
But according to the new research, conventional benchmark tests underestimate many 
Sandia National Laboratories computer scientist Tim Proctor is the first author on the Nature Physics paper describing a new way to benchmark quantum computers. Credit: Tim Proctor, Sandia National Laboratories
“It is standard practice in the quantum computing community to use only random, disordered programs to measure performance, and our results show that this is not a good thing to do,” said computer scientist Timothy Proctor, a member of Sandia’s Quantum Performance Laboratory who participated in the research.
The new testing method also saves time, which will help researchers evaluate increasingly sophisticated machines. Most benchmark tests check for errors by running the same set of instructions on a quantum machine and a conventional computer. If there are no errors, the results should match.
However, because quantum computers perform certain calculations much faster than conventional computers, researchers can spend a long time waiting for the regular computers to finish.
With a mirror circuit, however, the output should always be the same as the input or some intentional modification. So instead of waiting, scientists can immediately check the quantum computer’s result.
The research was funded by the Department of Energy’s Office of Science and Sandia’s Laboratory Directed Research and Development program. Sandia is a leading member of the Quantum Systems Accelerator, a Department of Energy national quantum research center.
New method reveals flaws in conventional performance ratings
Proctor and his colleagues found that randomized tests miss or underestimate the compound effects of errors. When an error is compounded it grows worse as the program runs, like a wide receiver who runs the wrong route, straying farther and farther from where they are supposed to be as the play goes on.
By mimicking functional programs, Sandia found final results often had larger discrepancies than randomized tests showed.
“Our benchmarking experiments revealed that the performance of current quantum computers is much more variable on structured programs” than was previously known, Proctor said.
The mirror-circuit method also gives scientists greater insight into how to improve current quantum computers.
“By applying our method to current quantum computers, we were able to learn a lot about the errors that these particular devices suffer — because different types of errors affect different programs a different amount,” Proctor said. “This is the first time these effects have been observed in many-qubit processors. Our method is the first tool for probing these error effects at scale.”
Reference: “Measuring the capabilities of quantum computers” by Timothy Proctor, Kenneth Rudinger, Kevin Young, Erik Nielsen and Robin Blume-Kohout, 20 December 2021, Nature Physics.DOI: 10.1038/s41567-021-01409-7