Saturday, January 30, 2021

Has Protein Folding Been Solved?

[This is a transcript of the video embedded below.]

Protein folding is one of the biggest, if not THE biggest problem, in biochemistry. It’s become the holy grail of drug development. Some of you may even have folded proteins yourself, at least virtually, with the crowd-science app ‘’Foldit”. But then late last year the headlines proclaimed that Protein Folding was “solved” by artificial intelligence. Was it really solved? And if it was solved, what does that mean? And, erm, what was the protein folding problem again? That’s what we will talk about today.

Proteins are one of the major building blocks of living tissue, for example muscles, which is why you may be familiar with “proteins” as one of the most important nutrients in meat.

But proteins come in a bewildering number of variants and functions. They are everywhere in biology, and are super-important: Proteins can be antibodies that fight against infections, proteins allow organs to communicate between each other, and proteins can repair damaged tissue. Some proteins can perform amazingly complex functions. For example, pumping molecules in and out of cells, or carrying substances along using motions that look much like walking.

But what’s a protein to begin with? Proteins are basically really big molecules. Somewhat more specifically, proteins are chains of smaller molecules called amino acids. But long and loose chains of amino acids are unstable, so proteins fold and curl until they reach a stable, three-dimensional, shape. What is a protein’s stable shape, or stable shapes, if there are several? This is the “protein folding problem”.

Understanding how proteins fold is important because the function of a protein depends on its shape. Some mutations can lead to a change in the amino acid sequence of a protein which causes the protein to fold the wrong way. It can then no longer fulfil its function and the result can be severe illness. There are many diseases which are caused by improperly folded proteins, for example, type two diabetes, Alzheimer’s, Parkinson’s, and also ALS, that’s the disease that Stephen Hawking had.

So, understanding how proteins fold is essential to figuring out how these diseases come about, and how to maybe cure them. But the benefit of understanding protein folding goes beyond that. If we knew how proteins fold, it would generally be much easier to synthetically produce proteins with a desired function.

But protein folding is a hideously difficult problem. What makes it so difficult is that there’s a huge number of ways proteins can fold. The amino acid chains are long and they can fold in many different directions, so the possibilities increase exponentially with the length of the chain.

Cyrus Levinthal estimated in the nineteen-sixties that a typical protein could fold in more than ten to the one-hundred-forty ways. Don’t take this number too seriously though. The number of possible foldings actually depends on the size of the protein. Small proteins may have as “few” as ten to the fifty, while some large ones can have and a mind-blowing ten to the three-hundred possible foldings. That’s almost as many vacua as there are in string theory!

So, just trying out all possible foldings is clearly not feasible. We’d never figure out which one is the most stable one.

The problem is so difficult, you may think it’s unsolvable. But not all is bad. Scientists found out in the nineteen-fifties that when proteins fold under controlled conditions, for example in a test tube, then the shape into which they fold is pretty much determined by the sequence of amino acids. And even in a natural environment, rather than a test tube, this is usually still the case.

Indeed, the Nobel Prize for Chemistry was awarded for this in 1972. Before that, one could have been worried that proteins have a large numbers of stable shapes, but that doesn’t seem to be the case. This is probably because natural selection preferentially made use of large molecules which reliably fold the same way.

There are some exceptions to this. For example prions, like the ones that are responsible for mad cow disease, have several stable shapes. And proteins can change shape if their environment changes, for instance when they encounter certain substances inside a cell. But mostly, the amino acid sequence determines the shape of the protein.

So, the protein folding problem comes down to the question: If you have the amino-acid sequence, can you tell me what’s the most stable shape?

How would one go about solving this problem? There are basically two ways. One is that you can try to come up with a model for why proteins fold one way and not another. You probably won’t be surprised to hear that I had quite a few physicist friends who tried their hands at this. In physics we call that a “top down” approach. The other thing you can do is what we call a “bottom up” approach. This means you observe how a large number of proteins fold and hope to extract regularities from this.

Either way, to get anywhere with protein folding you first of all need examples of how folded proteins look like. One of the most important methods for this is X-ray crystallography. For this, one fires beams of X-rays at crystallized proteins and measures how the rays scatter off. The resulting pattern depends on the position of the different atoms in the molecule, from which one can then infer the three-dimensional shape of the protein. Unfortunately, some proteins take months or even years to crystallize. But a new method has recently much improved the situation by using electron microscopy on deep-frozen proteins. This so-called Cryo-electron microscopy gives much better resolution.

In 1994, to keep track of progress in protein folding predictions, researchers founded an initiative called the Critical Assessment of Protein Structure Prediction, CASP for short. CASP is a competition among different research teams which try to predict how proteins fold. The teams are given a set of amino acid sequences and have to submit which shape they think the protein will fold into.

This competition takes place every two years. It uses protein structures that were just experimentally measured, but have not yet been published, so the competing teams don’t know the right answer. The predictions are then compared with the real shape of the protein, and get a score depending on how well they match. This method for comparing the predicted with the actual three-dimensional shape is called a Global Distance Test, and it’s a percentage. 0% is a total failure, 100% is the high score. In the end, each team gets a complete score that is the average over all their prediction scores.

For the first 20 years, progress in the CASP competition was slow. Then, researchers began putting artificial intelligence on the task. Indeed, in last year’s competition, about half of the teams used artificial intelligence or, more specifically, deep learning. Deep learning uses neural networks. It is software that is trained on large sets of data and learns recognize patterns which it then extrapolates from. I explained this in more detail in an earlier video.

Until some years ago, no one in the CASP competition scored more than 40%. But in the last two installments of the competition, one team has reached remarkable scores. This is DeepMind, a British Company that was acquired by Google in twenty-fourteen. It’s the same company which is also behind the computer program AlphaGo, that in twenty-fifteen was first to beat a professional Go player.

DeepMind’s program for protein folding is called AlphaFold. In twenty-eighteen, AlphaFold got a score of almost 60% in the CASP competition, and in 2020, the update AlphaFold2 reached almost 90%.

The news made big headlines some months ago. Indeed, many news outlets claimed that AlphaFold2 solved the protein folding problem. But did it?

Critics have pointed out that 90% is still a significant failure rate and that some of the most interesting cases are the ones for which AlphaFold2 did not do well, such as complexes of proteins, called oligomers, in which several amino acids are interacting. There is also the general problem with artificial intelligences, which is that they can only learn to extract patterns from data which they’ve been trained on. This means the data has to exist in the first place. If there are entirely new functions that don’t make an appearance in the data set, they may remain undiscovered.

But well. I sense a certain grumpiness here of people who are afraid they’ll be rendered obsolete by software. It’s certainly true that the AlphaFold’s 2020 success won’t be the end of the story. Much needs to be done, and of course one still needs data, meaning measurements, to train artificial intelligence on.

Still I think this is a remarkable achievement and amazing progress. It means that, in the future, protein folding predictions by artificially intelligent software may save scientists much time-consuming and expensive experiments. This could help researchers to develop proteins that have specific functions. Some that are on the wish-list, for example, are proteins to stimulate the immune system to fight cancer, a universal flu vaccine, or proteins that breaking down plastics.


  1. This is a daunting but beautiful problem; glad to see progress is being made. Once AI generates its internal rules, can that then help scientists tease out the physical principles? How do hydrogen bonds interact to minimize the overall energy of a protein? Do vibrational energy levels play a role? How does the pH of the environment affect the timing of the process? It unsettles my prion infected brain.
    Who owns the technology? Frightening.

  2. Nice text. But you didn't mention that this 3 dimensional protein folding is also linked to one of the Millenium problem in mathematic. Solving it for all instances will solve the P=NP conjecture (and make you win 1m$)

  3. For decades, one of my repeated complaints in the field of artificial intelligence has been its lack of anything really new. Most of the seminal theoretical work in the area was done in the 1960s and 1970s, and everything since then has been far more a consequence of adding hardware than new insights. For example, despite the remarkable impact on our daily lives of voice interfaces such as Alexa, the fundamentals of that technology remain based on rather feeble 1960s attempts to imitate natural neurons. Other areas, such as evolutionary methods, have never become influential or significant, despite great hopes.

    The DeepMind/Alpha world is different. I recall feeling a literal tingle when I first read how AlphaGo Zero [1] derived in a matter of days, without human assistance, strategies for playing Go that were unknown to humans after centuries of playing the game.

    With DeepMind, we are finally getting away from merely training machines to do what we as humans did first and into the realm of intelligent devices that can do things we humans will never be able to do.

    This is my field. Artificial intelligence was a significant part of my day job for years and decades. I’ve helped get millions of dollars of funding to universities and small businesses in this area. When I raise novel perspectives on physics in forums such as this one, more often than not, I am merely applying a few of the same ruthlessly ego-indifferent analytical methods that allow machine intelligences to derive results that human minds, for many reasons, tend to gloss over or not even see.

    I am very much an advocate of artificial intelligence. If used well, it can benefit humankind in ways no other technologies can ever do. It is also almost inevitable, a technology that market and social forces will not easily permit to be ignored.

    But there are dangers in power. We need to be careful what we unleash and analyze in advance what its impact may be. If the unthinking application of simple social technologies such as Facebook and Twitter can bring entire nations to or over the edge of genocide and civil war [2] — an all-too-real risk for some of us these days — then imagine for a moment what the thoughtless and unplanned release of truly malicious intelligences with capabilities beyond those of humans might do.

    Thus making sure to keep a bit of terror in the back of our minds is not an altogether bad idea when contemplating such futures. Retaining that “what if” worry might keep us from making what could end up being a terminating error.


    1. re: "With DeepMind, we are finally getting away from merely training machines to do what we as humans did first and into the realm of intelligent devices that can do things we humans will never be able to do."

      I don't see DeepMind doing things that humans haven't or may never do as "intelligence". It just seems like an extension of pattern recognition.

      I have yet to see (but then I don't see everything) these "machine learning" technologies ever explain anything.

      Useful...yes, but without explanation, which drives the scientific method, how will such approaches enable us to model protein folding (or anything else)?

    2. Terry Bollinger: I think there is a distinction to be made between intelligence (being able to solve a given problem) and emotions; and being "malicious" is a person's fault, not the machine's fault. There is no reason to give the machine emotions.

      There is a danger, but it is a danger like the nuclear bomb. The bomb is not malicious, the person that orders the detonation may well be.

      AI are already be used to hack systems with malicious intent, but they are just hammers wielded by evil people.

      And clearly, AlphaGo is not "thinking" in any sense like a person things. A person does not need to play 29 million games to become a grand master. Humans play a few thousand games, and can generalize and extrapolate strategies from single examples. Something Deep "Mind" cannot do. In the end, it is a trial-and-error statistical engine, akin to evolution.

      Certainly impressive, but only a technology to worry about in the wrong human hands.

    3. A^2, yes: DeepMind, like all other artificial intelligence technologies currently in existence, is far indeed from being a general intelligence technology capable of human-like sentience. It is a pattern recognizer, a device that looks at the configuration of a game or protein and maps it to a new structure that takes you closer to your goal, whether winning a game or folding a protein.

      What is unique about DeepMind, and not at all like what has come before, is its ability to discover new patterns without human assistance.

      Most folks have heard of neural nets these days. These are the technologies, scorned for almost two decades (just ask Yann LeCun), behind most of the speech and image recognition applications that transformed computers and the Internet over the past decade or so. They are also vital for robotics, enabling devices such as self-driving cars to recognize where they are and the nature of what is in front of them. Neural nets can operate at speeds far beyond human capability and can be mass-distributed over the entire Internet. Thus a neural net could, in principle, help vast numbers of people who know little about antiques identify items of high value.

      The dirty little secret of all neural nets is that even though folks refer to them as “deep learning” technologies, they don’t learn anything. They only do what humans train them to do, either directly or by providing them with masses of carefully labeled data. A more accurate name for them would be “perception amplifiers,” devices that enable computers and the Internet to interpret raw data in terms of meaningful patterns. The patterns are given to them by humans, who had to do the hard part of figuring out what is essential.

      DeepMind upset this pattern (a pattern of patterns?) by showing that computers can, entirely by themselves, uncover new and goal-relevant patterns never before found by humans — and do it in some cases with rather astonishing speed. Humans took centuries to learn how to play the exceedingly complex game of Go well. AlphaGo Zero replicated all of that learning, on its own, in mere days — and then proceeded quickly into new turf never before seen by human players.

      So yes: None of this is sentience, and all of it is patterns. But the building blocks of sentience begin with patterns, and DeepMind has introduced an entirely new and utterly non-human twist on finding such patterns. That will have impacts, such as in the decades-old and extremely high-value issue of protein folding. But it will not end there. DeepMind is a fresh wind across an old and oddly stagnant field, a wind with repercussions not easily predicted.

    4. "But there are dangers in power. We need to be careful what we unleash and analyze in advance what its impact may be. If the unthinking application of simple social technologies such as Facebook and Twitter can bring entire nations to or over the edge of genocide and civil war [2] — an all-too-real risk for some of us these days "

      .... and so Social Media need a complete overhaul either to moderate or delete them.

    5. Dr. A.M. Castaldo,

      Sorry, I slipped into security dialect. A "malicious virus" for example does great harm if allowed to operate unchecked. While malicious software often amplifies the intent of people, it is unwise to assume emotions even then. For example, the easiest way to kill millions in a pandemic is simply not to care. Minimally emotional sociopaths such as Stalin have often chalked up larger death tolls than less disciplined high-emotion ones such as Hitler.

      On the other hand, high-emotion leaders are more likely to incite pogroms. Oddly, one of the destructive emotions invoked by many pogrom initiators is hygiene or self-cleaning, since that tends to be associated with persuasion-indifferent obsessive compulsive behavior. Hygiene invocation is going on now in the US as some of our larger and more ruthless propaganda outlets relentlessly refer to people seeking asylum as "diseased." The irony is that these same outlets encourage behaviors that vastly amplify COVID-19's deadly impact.

      Regarding computers and emotions, the fully deterministic style of computer information processing leaves no easily discernable pathway by which such devices could either evoke or respond to the still-mysterious physics of qualia, of which emotions are one example.

    6. Terry: "But the building blocks of sentience begin with patterns"

      Begins? What else has human knowledge got going for it besides patterns? I'd happily characterize the entire project of understanding as patterns.

    7. DougOnBlogger,

      >… “What else has human knowledge got going for it besides patterns? I’d happily characterize the entire project of understanding as patterns.”

      While human knowledge does indeed have a close relationship to collecting and validating patterns, human intelligence differs in at least one major way: Patterns don’t ask questions like you just did.


      >… “Social Media need a complete overhaul…”

      Yes. What is scary is how casually all of us walked into a catastrophe. Future prediction is always tricky. When we are presented with powerful new technologies, it becomes even more problematic. The algorithm designers who were seeking to maximize clicks were guilty of being greedy. However, since intense emotions encourage more clicks, they simultaneously incentivized the spreading of the worst possible (and often fantastically invalid) interpretations of every situation. That is unsustainable in the long term, and dark ages are so darned hard to get out of.

    8. Terry Bollinger
      “While human knowledge does indeed have a close relationship to collecting and validating patterns, human intelligence differs in at least one major way: Patterns don’t ask questions like you just did.”

      I am extremely wary of the assertion that sentience and ‘true’ intelligence is something humans have but AI systems self-evidently do not / can never have. Wary primarily because this is an assertion about a quality that we are not fully able to define or understand within ourselves. It also seems to imply that the human brain is not just a neural network but incorporates ‘hidden variables’ that can never be replicated within a machine. That is a step too close to mysticism for me.

    9. Hi MikeS,

      Mysticism is blind faith that "slathering some o' that there software stuff onto a computer" (the actual ancient origin of Houston's Manned Spaceflight Center) is all you need to make everything work.

      Actually figuring out how to write the software, say to implement a working general artificial intelligence, is a lot tougher. Instead of "mysticism," old Missouri jackasses such as myself tend to call that "hard work."

    10. "Most of the seminal theoretical work in the area was done in the 1960s and 1970s, and everything since then has been far more a consequence of adding hardware than new insights."

      I'm interested in learning the basics of AI. Do you know of references that would describe the field at the seminal level you mentioned?

      Robert Clark

    11. Alas, the most straightforward answer is no, but I will look around a bit. These are events I lived through rather than read about, e.g., the above quote is from a couple of decades ago from an old-timer (then!) talking about the lack of progress. I was surprised at first, but I could not show any examples that contradicted his assertion after looking into it.

      The history is complex. Until you asked that, I forgot that there was a period (1980s? 1990s?) when talking about “artificial intelligence” marked you as an old-timer (or sci-fi fan!) who was not up to speed on the most recent tech. The field had diversified into specific subdisciplines, many of which are no longer called artificial intelligence these days.

      But with talky devices in people’s homes, the term has flipped back more to its original meaning of computers that can emulate human intelligence. This once again gives it more of a connotation of “leading-edge work” versus out-of-date, fuddy-duddy abstractionist. It’s fascinating how such language mutates in subtle ways.

      Regarding references, perhaps not too surprisingly, Wikipedia has a nicely comprehensive history piece with plenty of links.[1] That’s a great start. For a more pop intro focusing on human-like emulation (strong robotics flavor), there is also this g2 learning site.[1] I’m sure there are some excellent books out there, but frankly, online resources like Wikipedia are often better for deeper dives.


  4. 1) Begin with a suitably complex, naturally folded, water-soluble protein.
    2) Place it in an unfolding solvent system (chaotropic agents like urea or guanidinium chloride, or detergents like sodium dodecyl sulfate.
    3) It opens, expands, denatures.
    4) Put the solution in a dialysis bag vs. proper aqueous medium to remove the denaturant.
    5) Will the protein properly refold?

    Proteins are concatenated from amino acids as linear chains from reading mRNA. Post-translation (modification and) conformation then occurs.

    The expedient folding model begins with extruding a protein linear chain in suitable aqueous environment, progressively folding it as it appears. Optimize the whole thereafter.

    1. Unlikely to fold back to its shape. Cells have what are called chaperone proteins that can take a misfolded protein and refold into what the cell needs.

    2. The answer is: "sometimes".

    3. Proteins that are created by ribosomes fold as they are constructed one amino acid at a time so it's a sequential process. Parts will be folded inside "knots" where they are no longer pushed around by the electrostatic forces of the surrounding water molecules in the way a free linear protein would. (You can't tweak the inner blocks of a lego spaceship.) An unfolded protein in water gets folded everywhere at once so is likely to scrunch up differently.

      The raw proteins that are built by ribosomes are reworked in further interactions, sliced apart, linked up and also incorporate other bit like trace metal ions so there's a further era of reshaping.

  5. Christian Anfinsen was awarded the Nobel Prize in 1972 for his experimental work on the spontaneous refolding of proteins that indicated that a protein’s conformation is determined by its amino acid sequence. During the year prior to this (1971) I was fortunate enough to share an office with him during his sabbatical visit to the then recently-established Molecular Biophysics Laboratory in Oxford. Chris was a visiting fellow of All Souls College (or “Old Souls” as he usually liked to call it), and I was a still-wet-behind-the-ears young postdoc.

    At that time we knew the structures of just a handful of small proteins, so naturally we spent a lot of time discussing the “protein folding problem”. Chris himself was fairly pessimistic about the prospects, arguing roughly this way: that if there are N proteins in the entire world, then by the time we have solved the structure of (N-1) of them maybe (and only maybe!) might we accurately predict the structure of the Nth.

    I guess this is where AlphaFold2 may be at.

    And this is just the easy part of the problem. Even harder is to predict whether any given polypeptide sequence will fold at all. Folded proteins are only marginally stable, at best. Change a few amino acids, or tweak the temperature or pH a little bit, and the protein won’t fold – even worse, will usually collapse into a sticky aggregated mess.

    It’s all to do with thermodynamics, and I’m not sure that AI has that bit covered yet.

  6. Thank you Sabina for an interesting story! Indeed: the progress of science contrasts so strongly with where the world is heading and gives at least some hope, although possibly illusory. However the exaltation in the meta-scientific media about the 'artificial intelligence' is a bit strange, as if we already understand what intelligence is as such, and can even create artificial one. Strictly speaking, we are talking about the use of a certain programming optimization technique. All the same, the person is responsible for the results of the calculations, and will be so for the foreseeable future, since if something goes wrong, society should understand who 'to put in jail'. Machine intelligence will not feel any discomfort of sitting in a prison, and this is its big problem.

  7. Polypeptides have a large range of shapes and living systems will use some polypeptides in different shapes. Often the phosphorylation of a kinase or other polypeptide changes the shape, which occurs with ATP + K → ADP + KP, K stands for kinase. The free energy of ATP at 76kcal/mole bumps the protein into a higher energy state and a different folded geometry. It is in a way a sort of switch or like a flip-flop logic gate in a computer.

    One can see protein shape in action, well sort of. If you have ever cooked chicken strips and there is a tendon that hits the hot surface, you may notice that it wiggles almost like a worm. The elastic protein is changing it shape in response to the heat. The cooking of an egg is the same, where the shift from proteins in an aqueous state to a firmer white form is a change in protein shape.

    I had an idea, or have one in my crank files, on a possible way to solve the protein folding. The usual attempts at the protein folding problem involve a spatial configuration. The thought occurred to me to look at the Fourier transform of this according to vibrational modes. The amine to carboxyl amino acid ends have the same bonds and then the same set of frequencies. These are in a sense the “probe frequencies” and if excited they would excite other bonds. This would then be a huge Fourier transform problem with phonon modes. It is similar to a prime factorization and I thought maybe a quantum computer could solve this. I never did much with this idea.

    As you say this is an incremental step forward. I think there is a long way yet to go in really solving this. The goal at some point is to design novel proteins that perform functions they are designed for. We may have a way to go on this. As you say evolution does this with a sort of screening for function, and curiously using AI this way is a sort of theoretical version of the same.

    1. Lawrence: An alternative is to DO the transformation, on inputs and results, and then do the same kind of deep learning to predict the output Fourier transform using the input Fourier transform. Then just apply the inverse FFT to the results.

      If you are right, that might be an easier lift for the learning algorithm.

      Normal correlation is linear correlation, or sometimes polynomial correlation, and is of little use in predicting sinusoidal phenomenon. A line fitted through a sine function is just the x-axis, with 0% correlation; but if we fit sin(a+b*x) we can get 100% correlation.

      We can even use both, in series. If we have a sinusoid relative to a line or polynomial curve (like the covid data's weekly variations), then fitting the polynomial first and subtracting it can leave the sinusoid alone as the unexplained variations, to which we can fit a sinusoid.

      In the protein folding situation, there may be many frequencies to consider, and the FFT or a windowed FFT may give us separated data for each, which may be trained on separately.

      I haven't tried it but I suspect the same idea is likely to be true for deep learning applications. A lot depends on how we prepare the data for the learning algorithm, some preparations or transformations reveal relationships that increase the probability of learning.

    2. This might be compared to trying to use FFT to compute Shor's algorithm. It might work, but the problem grows exponentially in size. Quantum computing can overcome some of those limitations.

      These AI deep learning algorithms are neural networks and this approach is to extremize neural net work connections in a way that minimizes their energy. At this time it is evident this is the approach most operative today. I think I have read about using quantum computing to solve the protein folding problem. However, quantum computers are not up to the task yet.

    3. There was a Swedish physicist who was doing something kinda similar to what you describe around 2010-2015, I think his name was Anti Niemi or sth like that, he was cooperating with A. Liwo from UNRES group.

  8. There is solving the overall problem, and then there is applying what has been found to real world problems. It would seem that, even though the overall problem has not been solved, there are many direct uses of what we have obtained - in particular for medicine. Applied technology and applied science do not wait for the overall problem to be solved - scientists charge ahead, especially if they can get funding. Hopefully they will do so with some sense of ethics, but history shows that has not always been the case. I say this in relation to AI as well as medicine (can we 'improve' humans with this knowledge?).

  9. "The goal at some point is to design novel proteins that perform functions they are designed for."

    That may be one goal but it's very far off. Another goal, perhaps closer to realization, is to use the predicted shape of the folded protein to conduct a virtual screen of a library of small molecules, computing which ones are most likely to bind and alter the protein function. Right now, that task is performed in test tubes, with massive "high-throughput screening" systems in which tens, or hundreds of thousands of compounds are thrown at the target protein to see what sticks (literally). Being able to run this process in-silico could really advance drug discovery.

  10. Machine learning is at its core just a fancy and very powerful way of doing a fit, but that underlying fit has no justification. Sabine likes to complain about free parameters, and these machine learning algorithms can have arbitrarily many free parameters. While they might have incredible predictive power, the fundamental problem is that it's almost impossible to know when they fail and thus the results can't be trusted. Since there are infinite ways to fit any data set, a given fit might not tell you anything about a scenario outside that data set. For something like protein folding that might not matter though, if it's right 99% of the time that would still be an incredible boon for bio-medicine and we'd find out through experiments / clinical trials when things don't work.

  11. Some feedback regarding the video editing: I don't enjoy the way it zooms in or out at the beginning of every single new sentence. It's incredibly distracting. I don't know if I'm alone thinking this, but I almost need to avert my gaze to focus on the words that I'm hearing.

  12. Just to remind people who point out that poorly-trained neural-networks produce poor results, and we can never be sure we have found all the relevant data to train them with: human brains have exactly the same problem.

    Also, call it just a parameter fit, but how do you know that's qualitatively different than what your neurons and synapses are doing? (Albeit at a scale factor of roughly 100 billion to a million, and maybe quantity is quality at that level.)

    Finally, for me the function is more important than how it is implemented. An electronic system can detect the color red. Qualia -- schmalia.

    (Do I, a random person on the Internet, know what I'm talking about? Probably not, but that is how I feel based on what I think I know.)

  13. where and when did you learn biology ? were you premed

  14. A 90% sucess rate is pretty remarkable. What we need is *machine explaining* that actually explains to humans the reasoning by which machine learning achieved its high score. I don't expect this to be solved anytime soon, even if machine learning is set on the task ...

    An experiment I'd like to try with AlphaGo is to see how well it does when the board size is increased from 19 x 19 to 29 x29 to 39 x 39 ... to 109 x 109. This isn't an experiment that one can do with humans, but I'm interested in how AlphaGo's playing plateaus. One graphic showed that it did plateau, well above the human scores.

    the other experiment is to try with a higher dimensional board, say by a 19 x 19 x 19 board or even to 19 x 19 x 19 x 19. But this will require modifications to the games playing rules, which the previous experiment doesn't require and so it doesn't generalise as well. Indeed, there may not be a playable game in higher dimensions.

    Personally, I expect human players to improve against AlphaGo by studying AlphaGo's game. After all, although AlphaGo doesn't use a training set of games, from what I've read, the technology did originally arise from studying human games. It seems to be that humans ought to be given a chance to learn by playing against AlphaGo. It's still too early to say how well they will do, but I'm optimistic that they will eventually learn enough to beat it.

    1. Personally, I expect human players to improve against AlphaGo by studying AlphaGo's game. After all, although AlphaGo doesn't use a training set of games, from what I've read, the technology did originally arise from studying human games. It seems to be that humans ought to be given a chance to learn by playing against AlphaGo.

      Unlikely this will happen. When Deep Blue played vs. Kasparov, Deep Blue was given every one of Kasparov's games to study, but Kasparov was denied any of Deep Blue's games to study.

    2. >... "from what I've read, the technology did originally arise from studying human games."

      For the first AlphaGo that is true. For the subsequent AlphaGo Zero it is flatly false. AlphaGo Zero created a virtual reality world in which it began literally with zero knowledge (hence the name) of any human Go strategies. It then began playing against itself within its virtual world, doing do at an astonishing pace and learning as it went. One way if describing its approach is to say it "learned how to learn" as it went, figuring out not just how to play individual moves better, but generalizing its findings into a better understanding of which branches of the almost unfathomably large space of possible moves were most likely to contain powerful new strategies.

      Within mere days it had rediscovered, without any human strategy inputs, every Go strategy that had been developed by humans over centuries of playing the game.

      It then began self-learning new strategies no human had ever devised, again at speeds many, many orders of magnitude faster than humankind.

      >... "It seems to be that humans ought to be given a chance to learn by playing against AlphaGo. It's still too early to say how well they will do, but I'm optimistic that they will eventually learn enough to beat it."

      Give me a break. In this particular case, in which the self-learning is itself happening at incomprehensibly fast speeds, that's like saying "If only we can practice multiplying big numbers enough, I'm sure we can learn to do it better than computers!"

      Sorry about my tone. Watching how that two-bit thug of a dictator over in Russia has used the Internet and his fellow thugs over here to undermine basic sentience in people I've known and cared about for decades, while simultaneously needlessly and maliciously manslaughtering hundreds of thousands of lives via COVID-19, has made me unusually grumpy this month.

  15. Protein biochemist / bioinformatician specializing in protein structure here:

    AI structure prediction may be useful for downstream users, like wet biochemists who investigate how a protein works or interacts, but it gives us absolutely nothing about protein folding.
    What we need to know are pathways a protein takes from completely extended conformation in which it begins its life on the rybosome. We want to know kinetics of these pathways and thermodynamics. We want to be able to predict how change o sequence will affect these pathways.
    AI gives us nothing here. It's just a tea leaves divination (albeit a pretty good one), a black box where you input the sequence and get the final structure. How did the AI deduced this structure? Event this we will not know.
    What we need are (much) better force fields of atomic interactions and (much) better methods to navigate the astronomically huge conformational spaces of proteins. GPGPU has been an extremely usefull thing here and is much underappreciated.

  16. Hello Empischon,
    very interesting and informative comment.
    Greetings Stefan

  17. I'm a former software engineer. I never worked on "AI" and there's obviously been a lot of progress on it since I retired. What I do understand, though, is that while it now uses new data structures and algorithms called "neural networks" to evaluate results against the criteria for success, it's still using more-or-less brute force methods of trial and error to produce those results. These methods are now so much more effective than in the past mainly because they're executed so much faster, on much bigger data sets.

    I see comments about the inability of these systems to report the "reasoning" or the path of "deduction" they followed; why can't they leave some trail of breadcrumbs? But there was no reasoning, or deduction, in the sense we use those words. These things are still pretty mysterious and not reducible to algorithms.

    1. I fully agree. Thanks for pointing this out. It needs to be said given just how much hype surrounds so-called 'machine learning' and 'artificial intelligence'. They're not intelligence as we know it.

      Socrates was once called the wisest man in Athens for admitting what he didn't know and for affirming that a man should know himself. It seems AI doesn't know that it doesn't know, never minding knowimg what it does know and nor can it know itself: even how it is, say, programmed; never mind it's 'deductive' path in playing Go or folding proteins.

  18. Whilst researching somethimg completely unrelated, I stumbled upon an article on Wikipedia about helix folding. They say that the Zimm-Bragg model, which gives a reasonable prediction of fractional helicity, is equivalent to the Ising model, familiar from condensed matter theory! Moreover, there is a refinement called the Lifson-Roig model which can be exactly solved via transfer matrices. These are again used in solving the Ising model and the like.

    This suggests that these models view helix folding as a kind of phase transition! I think this is quite remarkable and at least understandable for anyone conversant with physics as opposed to Alpha Go's opaque modelling. Presumably, DeepMind are keeping their algorithms a closely guarded secret for commercial reasons. Nevertheless, I'd be surprised if the theory that they've used to model their algorithms aren't already in the public domain. Personally, I think it's these people who deserve the credit and not DeepMind itself for 'solving' Go computationally. It seems companies are now in the business of obfuscating where the credit actually belongs - and we are to only admire the gleaming end product.

    Certainly, I'd like to know more about these unsung heroes of machine 'learning'.

    1. Moziber,

      Yep, the deep dark secret is that blinkin' near all of the most interesting advanced research in artificial intelligence over the past decade or so has been intensely based on hard physics concepts and mathematics.

      It all started as a wild hypothesis, "free energy" and such, but quickly developed into a large and remarkably effective research strategy. And again, I'm speaking as someone who was right in the middle of overseeing and selecting a large chunk of the US government funding of such work.

      However, for those of us working in the area this was anything but a secret. It looks obfuscated because all you are seeing now is the glossy marketing face attached to the final results, minus all details. That's not necessarily intentional obfuscation, since those details can confuse even computer scientists and physicists not already familiar with the literature on the topic.

      I've been looking over the sparse information available so far on AlphaFold, and I'll try to make some comments on the actual tech later this week. Oddly, I did not find it as impressive as I thought I would. A couple of folks here have made insightful observations about the need for even more profoundly physics-based approaches, and I think I'm now inclined to agree.

      Also, I'm surprised no one has mentioned quantum computing, at least in the parts of this thread that I've read. As Manin once suggested (but which everyone, including Manin, then immediately forgot), biomolecules are the original and most effective naturally occurring examples of massively parallel quantum computing.

      Just think about it: How else could one molecule figure out folding moves that take humans enough energy to overheat a city block when using conventional computers to calculate the same move?

      That aspect of biomolecules was forgotten almost the moment Manin proposed it (even by Manin!) because everyone became instantly enamored with the then-new and much more human-comprehensible Turing model of computing. That sharp turn and drastic restriction resulted in the current severely constrained view of quantum computing, in which we forcefully herd all quantum behavior into tiny, almost purely classical entities called "qubits".

      Biomolecules, in contrast, use a completely continuous and extremely non-digital version of quantum computing. That is precisely why they can do it so much better.

    2. @Terry Bollinger,

      From what I understand, machine learning began with expert systems modelling knowledge and then with neural networks.

      I don't see that quantum computing heralds anything new in terms of what it can compute since the quantum Turing machine can be related to the classical version. It merely represents a computational speedup of certain algorithms. Of course, costructing one is an immense techonological challenge, and thats where most of the learning is going to be.

  19. Terry,
    I think you may have been the one who made the comment a day or so ago about cooking chicken strips. I spent a good part of that day asking myself why the leftover chicken strips didn't uncook them selves later on in the fridge. But I was stuck on the idea that the heat of cooking just refolded the chicken proteins into a different shape. I finally realized that stuff doesn't uncook in the fridge because the protein gets broken apart.

    Now I am going to go way out on a metaphorical limb. You can think of the American body politic as an over heated protein that got twisted and folded into very unhealthy and ugly shapes over the last four years, but our democracy actually did not totally break apart even when severely tested. Now in cooler political times it begins to unfold, uncook, and return to its more normal and healthy shape. I think we can now likewise all uncook our selves a bit.

  20. Steve, though it was someone else who brought up chicken strips, I like your nicely poetic metaphor. We sure could use some cooler-climate time to enable a bit of quantum political annealing!

  21. The only technical information currently publicly available on the design of AlphaFold2 is pages 22-24 of the CASP14 Abstract Book [1].

    AlphaFold2 begins with three vast databases (UniRef90, BFD, and MGnify clusters) of experimentally determined DNA-to-protein mapping data. This profound reliance on lab-collected data distinguishes AlphaFold2 sharply from AlphaGo Zero and MuZero since it makes it impossible for AlphaFold2 to create and learn from purely virtual training worlds that can operate at computer speeds. This is a common problem with simulation-based AI training methods. It’s not that you cannot create such virtual worlds and then attempt to train AIs or robots in them. Such virtual worlds are overly simplified fantasies that inevitably lack the necessary details of the real world. Consequently, the training results are usually worthless.

    In gaming (e.g., Go), this is not a problem because the virtual universe’s underlying rules — the game rules — are all fully known in advance.

    AlphaFold2’s instead seems to have done better mostly through its use of the attention concept. To quote the abstract: “We found that existing deep-learning architectures overly favor sequence-local interactions and do not sufficiently account for global structural constraints. To remedy this, we have developed a novel, attention-based deep learning architecture to achieve self-consistent structure prediction.”

    Harshall Lamba [2] explains attention by showing how it helps translate languages. I’ll attempt an even simpler explanation: Attention is just a prioritizing input “clues” by the likely impact they will have on the overall situation. Think of driving a motorcycle and seeing both a gnat and a truck heading towards you. To which one do you pay attention? For proteins, this means recognizing constraints such as available space. Bottom-up methods can miss such issues if they do not “pay attention” to emerging constraints.

    Empischon and Jim Birch both alluded to the need for something more profound. I tend to agree. In particular, methods like AlphaFold2 only indirectly attend to “assembly point” ribosome constraints, focusing far more (for now) on the downstream outcomes.

    Unfortunately, focusing on the assembly point requires better prediction of quantum mechanical dynamics than current approximation methods provide. Here again, I think Yuri Manin was onto something important way back in 1980 [3], which is this: Organic molecules are “quantum balanced” in a way that allows them to implement insanely unlikely reactions without slipping into chaos. Some recent work (thanks, Roger D.) may be relevant to that issue [4]. One possibility is that the quantum world is smoother and less complicated than generally recognized, with uncertainty emerging instead from its binding entanglements to the classical world. I like such approaches. They suggest that organic molecules may simply be particularly adept at keeping classical chaos at bay, even (especially) at room temperatures.

    [1] Jumper, J.; Evans, R.; Pritzel, A & others, High Accuracy Protein Structure Prediction Using Deep Learning (AlphaFold2) [Abstract]. In CASP14 Abstract Book, Protein Structure Prediction Center, 2020, 22-24.

    [2] Lamba, H., Intuitive Understanding of Attention Mechanism in Deep Learning. towards data science, 2019.

    [3] Manin, Y. I., Biomolecules as Quantum Computers. From Computable and Uncomputable, Soviet Radio, 1980, 14-15.

    [4] Yan, B. & Sinitsyn, N. A., Recovery of Damaged Information and the Out-of-Time-Ordered Correlator. Physical Review Letters, APS, 2020, 125, 040605.

  22. A good resource for contemplating the effectiveness of neural networks is the documentary film that won a couple of Oscars last year – “My Octopus Teacher.”

    Two thirds of an octopus’s neurons are located in its arms and there is beautiful footage of how that translates into amazingly adaptive and seemingly sentient behavior in what is essentially a mollusk.

    An excellent film for theoretical roomination on the philosophical divide between observer and participant.

  23. This great advancement in protein folding prediction will also have a very strong impact on genomics. We have, at the present time, a huge number of sequenced genomes, containing a huge number of protein-coding genes (there are also genes which do not code for proteins). These gene sequences are also predicted by computer programs. The algorithms used to compare linear protein sequences are not able to compute a significant sequence similarity score if the similarity between sequences is lower than about 25%. Therefore, if a protein sequence has not any significant similarity to any other sequence with a known biochemical function, that sequence is classified as "unknown". About a third of the genes in almost every genome is made of unknown genes. A protein 3D structure tends to be more conserved than its linear sequence. This is because proteins whose sequences have low similarity might have 3D structures with a much higher similarity (a protein function depends on 3D structure and evolutionary pressure works on structures not on linear sequences). Therefore, in silico computation of protein 3D structures coded by unknown genes will increase the identification of their function by comparing them to known genes at the 3D level.


COMMENTS ON THIS BLOG ARE PERMANENTLY CLOSED. You can join the discussion on Patreon.

Note: Only a member of this blog may post a comment.