How to evaluate computers that do not yet exist

To evaluate the performance of a supercomputer, computer scientists turn to a standard tool: a set of LINPACK algorithms that helps to check how a machine can solve problems with a huge number of variables. But for quantum computers, which one day will be able to solve problems inaccessible to ordinary computers, such a standard for measuring performance does not exist.



One of the reasons is that computers, which will have to use the laws of quantum mechanics to speed up certain calculations, are still in a rudimentary state, and the various possible device circuits of such computers are competing with each other. In some of them, the quantum bits, or qubits used for calculation, are enclosed in the spins of a sequence of “captured” ions, while others rely on pieces of superconducting metals that resonate in response to microwave radiation. Comparing the rudimentary architectures is like “as if we went to a nursery to decide which of the babies will become famous basketball players,” says Scott Aaronson, an IT specialist at the University of Texas at Austin.



However, researchers are already making their first attempts to measure the speed of quantum computers. In June 2019, Margaret Martonosi, an IT specialist at Princeton University and her colleagues, presented a comparison of quantum computers from IBM, Rigetti Computing from Berkeley and University of Maryland (UMD) at College Park. The UMD machine running on trapped ions handled most of the 12 test algorithms more accurately than other superconducting computers, and the team announced this at the Phoenix International Computer Architecture Symposium. Christopher Monroe, a physicist at UMD and founder of IonQ, predicts that such comparisons will someday become the standard. “These toy algorithms give us a simple answer - did it work or not?” However, even Martonosi warns that care must be taken in these tests. The analysis even emphasizes how difficult it is to compare quantum computers, and that as a result, developers are free to choose metrics that expose their machines in the best possible light.



Conventional computers work with bits of information encoded in transistors that can turn on and off, indicating zero or one. A qubit can simultaneously denote both zero and one, encoding a state in an ion, the spin of which can be zero, single, or can be in both states at once. Qubits allow the machine to simultaneously process an array of incoming data, instead of doing it sequentially. But the real capabilities of the machine are not realized through this massive parallelism, but through an approach to problems whose solutions can be encoded in quantum waves splashing among qubits. The waves interfere in such a way that the wrong decisions sink, and the right ones float.



A quantum computer can, for example, crack Internet encryption systems based on factorization of large numbers - for a classical computer this is a very difficult task. But solving such problems will require 100,000 qubits, as well as methods for correcting errors in sensitive quantum waves. Researchers say that such machines will not appear for several decades. However, quantum computers with only a few dozen noisy qubits will soon be able to compete with ordinary ones in certain tasks, and developers are already looking for suitable metrics to prove this.

Quantum leap

Rigetti Computing is looking for an application that can give a practical advantage to a quantum computer based on a superconducting chip. Other companies are promoting other metrics to measure progress.



One of the most common metrics is the solution to a problem that is too much for a regular computer, or the so-called. quantum superiority. “It's kind of a 'Hello world!' Project that demonstrates the performance of your quantum computer,” said John Martinis, a Santa Barbara physicist who runs Google’s project to excel on a machine with 72 superconducting qubits.



The task chosen by researchers from Google is extremely abstract. In fact, they program their quantum computer to perform a set of constantly repeating random operations on qubits. Due to quantum interference, a machine must produce certain sequences of zeros and ones with a higher probability than others. If there were no interference, the probability of occurrence of both those and other sequences would be the same. In addition, predicting the exact distribution of work results goes beyond the capabilities of classical computers with an increase in the number of qubits. So if Google researchers can measure this characteristic distribution for their machine of 72 qubits, it will mean that they have achieved quantum superiority by counting something that is not available to a regular computer. However, this cryptic exercise will not open the era of practically useful quantum computers, says Greg Cooperberg, a mathematician at the University of California at Davis. "This is superiority in solving a completely useless task."



Researchers from Rigetti, on the contrary, seek to demonstrate that their quantum computer is capable of performing certain useful tasks more accurately, faster, or cheaper than usual - they called this metric a quantum advantage. “We want to achieve properties that can show us the shortest path to commercial value,” said Chad Rigetti, physicist and founder of the startup. For example, he says, a quantum computer can be ideal for modeling complex interactions of financial assets in a hedge fund.



In September 2018, Rigetti offered $ 1 million to the first user who managed to achieve quantum advantages on his computers, accessible to everyone. The current version uses 16 superconducting qubits. Since factors such as cost are included in the metric, the quantum advantage does not have such a strict definition, says Aram Harrow, a physicist at the Massachusetts Institute of Technology. “But if they’re a little blurry, it’s not a big deal for Ridgetti,” Harrow says.



IBM researchers have identified their metric, quantum volume - it measures the performance of quantum computers without comparison with conventional ones. It includes checking a quantum computer on random calculations, such as what they do at Google. It depends both on the number of qubits and on the number of computational cycles that a machine can withstand before its quantum state is washed out.



Using a machine with 20 superconducting qubits, IBM scientists have reached a quantum volume of 16 units and plan to double it annually, said Jay Gambetta, a physicist at IBM Research Center. Thomas Watson in Yorktown Heights, New York. He says that breakthrough applications will naturally follow. “I don’t think it’s worth marking on something like superiority. We are aware of this when we move towards ever greater achievements. ”



And there’s a direct comparison, like Martonosi’s. In her tests, a 5-qubit ion machine solved all problems correctly in 90% of cases, compared to superconducting qubit machines that solved problems in no more than 50% of cases. This difference reflects the current state of technology, but not their potential, says Martonosi. For example, in a superconducting machine, each qubit interacts only with its neighbors, but each ion in the machine from UMD interacts with all other ions, and this gives it an advantage. But larger machines with ions will no longer have this advantage.



Martonosi says comparisons show a significant improvement in the performance of all quantum computers when they are programmed for the differences in the noise of the qubits and their connectivity. “It works on a wide variety of hardware options,” she says. “And that's very cool.”



Harrow wonders how useful the current metrics will be in the long run. The main difficulty in quantum computing is finding technology that can scale to thousands of qubits, he says. “And these metrics are very loosely related to scaling issues.”