The dog is an innovation. It took place at least 20,000 years ago, when a group of humans domesticated wolves and subsequently began to develop different breeds. The light bulb, wheeled baggage, and the computer are also innovations.

Since prehistoric times, innovation has changed the course of our lives and is, according to science writer Matt Ridley, “The most important fact about the modern world, but one of the least well understood.” Ridley is the author of the new eye-opening book, How Innovation Works: And Why It Flourishes in Freedom. In it, he tells the story of dozens of innovations, to illustrate how that phenomenon is the main cause of the enormous progress humanity has seen in the past few centuries. He derives the following lessons from his study.

Innovation almost always happens gradually and not suddenly. It is “not an individual phenomenon, but a collective, incremental and messy network phenomenon.” Ridley asks, “Who invented the computer?” To answer the question, we would have to go back more than 200 years to the Jacquard loom and then review countless succeeding contributions and innovators. The same can be said about the car or virtually all other innovations, even though we sometimes identify them with individuals like Henry Ford who discovered a way of making the automobile widely accessible to the public.

Innovation is thus collaborative, and the same ideas often independently occur to different individuals at the same time. The telegraph, the thermometer, photography, and the hypodermic needle are examples of simultaneous inventions. Twenty-one people invented the light bulb at about the same time. If Thomas Edison or the Wright brothers would not have existed, we would still enjoy artificial light or the wonder of airplane travel.

Innovation is not the same thing as invention. One can invent something novel, but the people who make a difference are the innovators who figure out the way that that new idea can be useful to society, typically by improving on it and lowering its costs. Innovations exist due to our growing knowledge and a demand for the innovative product. As Ridley observes, “The light bulb emerged inexorably from the combined technologies of the day. It was bound to appear when it did, given the progress of other technologies.”

It’s not possible to plan innovation. Not even the innovators can do so. Innovations more often than not come about because of chance events or unexpected discoveries. Sheer luck explains Alexander Fleming’s discovery of penicillin. The founders of Google did not “set out in search of search engines. The founders of Instagram were trying to make a gaming app. The founders of Twitter were trying to invent a way for people to find podcasts.” Innovation is unpredictable.

That unpredictability also helps explain why the so-called “entrepreneurial state” has not effectively promoted innovation and why we can’t expect it to do so. For example, Ridley explains that contrary to those who advocate in favor of publicly funded innovation, the U.S. government did not intend to create a global internet. Only when the internet “escaped the clutches” of the government – that is, when it was essentially privatized in the 1990s – did the private sector and universities begin to transform the internet into what we use today.

Ridley observes that the widely held view that science leads to technology and innovation—frequently canvassed to justify public subsidies for science—is only partially correct. It is equally true that scientific knowledge is the product of technological improvements and attempts to understand the latter. The first inoculations were conducted without a good understanding as to how and why they worked. Attempts to resolve problems in the yogurt industry contributed to the development of the revolutionary gene-editing method known as CRISPR (which may yet help us find a treatment for COVID-19).

Innovation requires trial and error. It requires the possibility to experiment and to fail. Only then can innovation provide the path to success and human progress. Or, as Ridley puts it, “Innovation is the child of freedom and the parent of prosperity.”