Being Over-Reliant on Software

If you didn't know, I'm a big fan of rocketry. The SmarterEveryDay tour of the United Launch Alliance rocket factory is VERY cool. If you haven't already, check it out.

https://www.youtube.com/watch?v=o0fG_lnVhHw

It's a very interesting video; check it out sometime. It is long though, so for now here are the parts we care about that are pertinent to our conversation: old engineering software was limited and recommending isogrids for the Delta panels (8:30). Newer engineering analysis tools found orthogrids to be a better option for the Vulcan. The Vulcan panels take half as long to manufacture and are stronger (19:28).

The take away:

One piece of software recommends one approach; another piece of software recommends another.

Rocketeers deal with this all the time. RockSim (paid) says a rocket will fly one way, and OpenRocket (free) says it will fly another way. Experienced fliers take both simulations into consideration, along with their experience, to make an estimation.

Let's jump to current events that are affecting the whole planet: SARS-CoV-2.

To be crystal clear: I am NOT arguing that SARS-CoV-2 was lab made. I have no evidence of that, and I doubt you do either. I am only pointing out that the argument "my software didn't tell me to do it that way" is not solid evidence.

There was an early rumor that SARS-CoV-2 was bioengineered and/or escaped from a lab. Some experts are stating that, based on analysis of the genetic code, that is not the case. A quote from Live Science:

"Yet in computer simulations, the mutations in SARS-CoV-2 don't seem to work very well at helping the virus bind to human cells. If scientists had deliberately engineered this virus, they wouldn't have chosen mutations that computer models suggest won't work. But it turns out, nature is smarter than scientists, and the novel coronavirus found a way to mutate that was better — and completely different— from anything scientists could have created, the study found."

And from the original study, published in Nature Medicine: "While the analyses above suggest that SARS-CoV-2 may bind human ACE2 with high affinity, computational analyses predict that the interaction is not ideal and that the RBD sequence is different from those shown in SARS-CoV to be optimal for receptor binding. Thus, the high-affinity binding of the SARS-CoV-2 spike protein to human ACE2 is most likely the result of natural selection on a human or human-like ACE2 that permits another optimal binding solution to arise. This is strong evidence that SARS-CoV-2 is not the product of purposeful manipulation."

The research paper on Nature Medicine adds these arguments against it being manufactured:

"Furthermore, if genetic manipulation had been performed, one of the several reverse-genetic systems available for betacoronaviruses would probably have been used. However, the genetic data irrefutably show that SARS-CoV-2 is not derived from any previously used virus backbone."

Our software wouldn't recommend this approach.

Like people now try to avoid leaving DNA at a crime scene, if someone were engineering a virus and knew what software you were using to determine origin, and didn't want to be caught they would likely avoid using the software they knew the rest of the world used to detect such things. If they didn't the end result would have been a giant red flag that the virus was bioengineered.

In today's age of machine learning and AlphaGo, novel software can be created that is intentionally not told how to think like a human or fed rules from existing software. The software is designed to test possibilities and figure out rules and tactics on its own. Some of what the software comes up with may match what your software is programmed to do - basic scientific realities and such. Like second generation AlphaGo, some of the approaches may be completely new and unknown.

Paired with the right modern technology, a computing machine could test promising solutions in the physical world. Like a cyber attacker who knew someone was running a specific anti-virus, an intelligent bioengineer would test their solution against your software to see what it said. If it was found to set off alarms, it would be redone until it didn't register as bioengineered.

The Nature Medicine paper describes other possible orgins, but admits that "it is currently impossible to prove or disprove the other theories of its origin described here". They also mention a couple of times that an engineered virus would likely have gone through other steps, "but such work has also not previously been described".