Garry Gutting's recent post What Do Scientific Studies Show? at the NYT blogs is utterly unremarkable. Or so I thought, being clearly biased because the guy is a professor of philosophy, and I - I'm at the other end of the circle. But then he puts forward a proposal I think is brilliant: A labeling system for scientific news "that made clear a given study’s place in the scientific process", ranging from the speculative idea and preliminary results all the way to established scientific theory. 
I like the idea because it would be an easy way to solve a tension in science news, which is that what's new and exciting, and therefore likely to make headlines, is also often controversial and likely to be refuted later. The solution can't be to not report what's new and exciting, but to find a good way to make clear that, while interesting and promising, this isn't (yet) established scientific consensus.
23andMe has a star rating to indicate how reliable a correlation between a genetic sequence and certain traits/diseases is, based on what has been reported in the scientific literature. (See my earlier blogpost for screenshots showing how that looks like.) They have a white paper laying out the criteria for assessing the scientific status of these correlations. The 23andMe rating serves a similar purpose as the proposed rating for science news. It is handy as a quick orientation, and it is a guide for those who can't or don't want to dig into the scientific literature themselves. It doesn't tell you to disregard results with few stars, just to keep in mind that this might turn out to be a data glitch, and to enjoy or worry with caution.
I think that such a label indicating how established a scientific result or idea is would be easy to use. Writers could just assign it themselves with help from the researchers they have been in contact with while working on a piece. That might not always be very accurate, but undoubtedly bloggers would add their voice. There would most likely be a service popping up to aggregate all ratings on a given topic/press release (probably weighted by the source). I am guessing it would be pretty much self-organized because we're all so very used to these ratings for other purposes.
Do you think such a labeling would be helpful? If so, what criteria would you require for zero to five stars?
I think it is actually a good idea, but there is the danger that funding agencies will base their decisions on the total number of stars the candidate has been responsible for.
ReplyDeleteI think not the establishedness of the science reported about should be rated, but the quality of the popular reports should be reviewed by a star system.
ReplyDeleteFor example criteria could be if the purpose of the report is to deliver the scientific truth and inform the public or if the report is written rather to fuel controversy and flame wars to attract more gullible readers and sell copies, if the reporter was in contact with real experts who know what they are talking about or if he has just interviewed bloggers, philosophers, etc who are not experts but have an agenda to spread their biased and often from a scientific point of view wrong personal opinions about some topics, etc ...
Such a star rating should be needed, to review the purpose and quality of the report. For a rating of the establishedness of a research topic I see no need. In particular since it is cristal clear that by the suggested proposal, any funtamental physics topics like quantum gravity, BSM particle physics, and some cosmological issues, would hardly get a star because it is cutting edge ongoing research, whereas only applied physics related reports would get highly upvoted by the suggested criteria.
...and I agree with Philip Helbig.
ReplyDeleteI think it is difficult to assign a star rating because journalists and scientists are wont to play up speculative implications even if the majority of the work is on very solid ground. For example, I recently read an article about an experiment that aimed to produce a superposition of a macroscopic object in two different positions. The majority of the article is a pretty accurate description of the experiment. However, towards the end of the article, the scientists were quoted as saying that the experiment shows that, according to the many worlds theory, the object was in "two different universes". No comment about whether or not many-worlds is true was included. Now, obviously the many worlds theory is very controversial and, in any case, the statement is based on a misreading of what a "world" is in many worlds theory. What star rating should be given for an article like that?
ReplyDeleteIf it is up to the journalist or the scientists being interviewed to assign a star rating then it is likely that they would go for a high one. The article is about peer reviewed work, the scientists are established and well respected, and the experiment is part of a long line of work by various different groups that establishes "large" superpositions of various types. It is difficult to tell if the journalist has misreported the comments about many worlds by omitting essential caveats or whether the scientists were being misleading by oversimplifying or (just as likely) having a slim grip on the conceptual implications of quantum theory in the first place.
If you ask me, it is nigh on impossible to find a news article about the foundations of quantum theory that does not make a vastly misleading statement somewhere along the line. If it were up to me, I would assign all such articles 0 or 1 stars, because it is arguably more important for the public to understand that quantum theory does not imply a bunch of crazy stuff than it is for them to know the that our ability to manipulate quantum systems has improved by one tiny notch. Given this, I find it hard to believe that a consensus could be achieved on the star rating in my field in particular.
In addition to this, I am opposed to this kind of scheme on principle, just as I am opposed to the existence of bodies like The Science Media Centre that claim to make objective pronouncements about "the science". The ideal of science is that it is based on evidence and, in principle, anyone can go out and check the evidence for themselves to arrive at an informed opinion. Now, in practice this is obviously very difficult as it requires technical knowledge that may take years to acquire. However, I am opposed to anything that gives the impression that science is decided by some committee of experts more than by the evidence, and that arriving at a valid scientific opinion is somehow fundamentally inaccessible to the layman. Putting some kind of rating on an article reinforces this idea in my opinion.
Here is a list of on-line journals.
ReplyDeleteWould you "rate them" given the articles written.....or, just to the given articles?
Best,
Perhaps something with less of an implied "quality rating" than a star system as this is genuinely unclear. A new and exciting and yet still wildly under substantiated breakthrough should be rated as "not as good" as a reiteration of the clearly established. The conept is solid, give readers a clear sense of the solidity of a claim. But don't don't bias them toward disregarding poorly rated articles. Perhaps something closer to the old Hustler porno rating system, from flaccid through fully erect. How hard is the science? That last bit may have been a tad facetious.
ReplyDeleteI think it's a great idea in principle. I tell my students that any science announced as big news is probably going to turn out to be wrong. It's usually news because it contradicts previous theory and experiment, and if it does this, the chances are this one study is the one that's wrong (from 5th forces to superluminal neutrinos)
ReplyDeleteCormac
Dirac told Stern, re proton magnetic moment, that an appalling measurement was merely a day of grinding theory. Nobel Laureate Stern showed the Dirac equation failed. How many stars for Stern's crackpot grant funding proposal? How many stars for Australian crackpot Barry Marshall? Erasto Mpemba? Supersolidity?
ReplyDeleteI propose the vacuum is trace anisotropic toward otherwise identical enantiomorphic atomic mass distributions. Naïve drivel! How many stars before looking? Quantum gravitation and SUSY are empirically non-physical. Both demand vacuum massless boson photon symmetries are vacuum fermionic matter symmetries. How many stars for bosons and fermions being indistinguishable?
ReplyDeleteIt is important that the rating system not be corrupted by personal and subjective influences, especially by the intuitions of the celebrity physicists who gave us string theory and SUSY.
An objective and thoroughly scientific way to rate ideas/theories/models/paradigms would be to rate them on the basis of how many definitive predictions they have both made and had vindicated.
Definitive predictions are made prior to testing, are feasibly tested, are quantitative, are non-adjustable, and are unique to theory being tested/rated.
If this system was adopted, and had a board of governors who insured that the system would not be gamed, what a gift it would be to scientific progress!
Oh Bee, always the one with a finger to the wind of public and scientific opinion before deciding how to feel about a scientific subject. I think I know exactly how you would have felt about Einstein's theory of Special Relativity if you had lived at that time, had the same disposition, and had the same level of scientific knowledge that was appropriate for the time. You would have poo-pooed it because it went against the aether and then done a 180 as soon as you heard Max Planck was interested.
ReplyDelete"An objective and thoroughly scientific way to rate ideas/theories/models/paradigms would be to rate them on the basis of how many definitive predictions they have both made and had vindicated.
ReplyDeleteDefinitive predictions are made prior to testing, are feasibly tested, are quantitative, are non-adjustable, and are unique to theory being tested/rated."
Of course. But when a theory is falsified because it definitively predicted A and B was observed, then everything related to this theory should get 0 stars. Or perhaps for such cases, instead of stars, one could award epicycles for each instance of special pleading and ad-hoc modification of the theory.
Hi Phillip, Nemo,
ReplyDeleteTo my knowledge, funding agencies presently don't use what has become known as "alt metrics". I actually think we'd be better off if they did. Be that as it may, we're not going to improve the issue of alt-metrics by avoiding a rating that's supposed to help communicate science. Besides this, it seems odd to me to put a quite remote concern about researcher's funding prospects ahead of science education for the general public. Best,
B.
Hi Nemo,
ReplyDeleteI disagree with you. I do not think that the quality of the report should be rated. The rating is supposed to give the reader an assessment for how to judge the scientific status of the research he is reading about, not the quality of the report. The rating is supposed to provide exactly what the writing does not get across.
Maybe compare the rating to food labels and the text to the wrapping. Best,
B.
Hi Matt,
ReplyDeleteRegarding your question "What star rating should be given for an article like that?"
I think you're misunderstanding what the rating would be about. You're not rating the article. You're providing a label that summarizes how established the research idea or result is that the article reports on.
Best,
B.
Hi Nick,
ReplyDeleteYes, I see your point. A star rating is too suggestive of quality. I used it as an example primarily because it's something that everybody knows. But you are right, one should use something that's intuitive yet doesn't suggest that a speculative idea is somehow worse than an already well confirmed one, but that it's just at an earlier stage of the scientific process. Best,
B.
Hi Eric,
ReplyDeleteSorry, I don't know what you are trying to say except that you "know exactly how [I] would have felt about Einstein's theory of Special Relativity if you had lived at that time". That's an interesting statement because I don't know myself. Be that as it may, I don't see what it's got to do with this blogpost. Best,
B.
@Bee be sure, that reports topics like quantum gravity for example would get zero star by this rating as it is proposed and you agree with, because the relevant scales are not directly accessible ...
ReplyDeleteWe already have more than enough people who are not experts in these topics, but who nevertheless scream loudly in the popular media, TV, magazines etc, that such and similar research fields should not be allowed and not get fundeded, because they are (in their wrong and biased personal opinion) not testable, falsifiable, and what else. This bad "science journalism" already leads to physics students being scared away from wanting to learn about and study such topics, as you can see for example by this question on Physics SE:
http://physics.stackexchange.com/q/64291/2751
The proposed rating system will do nothing else than add to the bullying of fundamental physics topics in the mass media and convey to the laypeople the conciction that only applied research is good research. Is this really what you want ...?
So I stick to my opinion that it would be far better to review the quality and reliability of the articles (and not the topic), as percieved by experts actually working in these fields, if anything should be rated at all.
... but maybe about such topics that would not get a star by this rating system should not be talked about in the popular media at all to a lay audience? With that I would actually agree, trying to explain things like quantum gravity, BSM particle physics, cosmology, etc to laypeople has done the field no good, on the contrary...
ReplyDeleteHi Nemo,
ReplyDeleteTwo things. First, I guess we seem to agree that using stars is suggestive of quality and one should use something else. In a parallel discussion on fb, I suggested maybe polygons (the rounder, the more 'established').
Second, if you believe that fundamental research would get zero stars (triangles!) generally you're misunderstanding. Maybe you should read Gutting's article? His main concern clearly isn't quantum gravity, but those science topics that are relevant for people to base decisions on. What he had in mind were questions like: What kind of study has been done, how large was the sample, how statistically significant is the result, are there conflicting results, has it been peer reviewed, etc.
I really think pulling upon quantum gravity here is a red herring, but I don't think that quantum gravity papers would generally get a rating of 'speculative'. Even in this field there are ideas that are more established than others, that are backed up by more research than others, some are more controversial than others etc.
Examples. LHC hasn't found any evidence for a lowered Planck scale (graviton emission, black hole production), some MC analysis, peer reviewed, published in JHEP, something like this. It would probably get a rating of high confidence provided they didn't screw up the analysis for some obvious reason. Another example, the BH firewall. Lots of discussion, but highly controversial. This would get a triangle.
Best,
B.
Nemo said:
ReplyDelete"@Bee be sure, that reports topics like quantum gravity for example would get zero star by this rating as it is proposed and you agree with, because the relevant scales are not directly accessible ...
We already have more than enough people who are not experts in these topics, but who nevertheless scream loudly in the popular media, TV, magazines etc, that such and similar research fields should not be allowed and not get fundeded, because they are (in their wrong and biased personal opinion) not testable, falsifiable, and what else. This bad "science journalism" already leads to physics students being scared away from wanting to learn about and study such topics, as you can see for example by this question on Physics SE:
http://physics.stackexchange.com/q/64291/2751"
Agree wholeheartedly. And we should also repeat wholehearted all remarks from Lubos Motl's blog as gospel on other science blogs. And we should never associate that blind uncritical repetition of one man's view as being itself a form of bullying. That would be so unfair and discriminatory...
ReplyDeleteP. Helbig says: "Of course. But when a theory is falsified because it definitively predicted A and B was observed, then everything related to this theory should get 0 stars. Or perhaps for such cases, instead of stars, one could award epicycles for each instance of special pleading and ad-hoc modification of the theory."
I wonder P. Helbig realizes that most of the theories he worships and makes copious excuses for would fare very badly under his proposal?
For example the standard model of cosmology has stumbled badly on the dark matter: cored galactic distributions, orders-of-magnitude too few "subhaloes", not a single "WIMP" in 40 years, homogeneity falsified by latest Planck observations, ...
But hey, what's a few epicycles and 40 years of special pleading when you are in possession of the one true faith?
Define "standard model". Second, how has Planck "falsified homogeneity"?
ReplyDelete
ReplyDeleteP. Helibig: "Define "standard model". Second, how has Planck "falsified homogeneity"?"
Definition: SM = substandard model-building dog that can't hunt.
See published Planck results and analysis by professional astrophysicists.
Robert L. Oldershaw
Discrete Scale Relativity/Fractal Cosmology
Robert, I admire your tenacity in argueing for fractal cosmology. However, I think before anyone would take your theory seriously you will have to incorporate the changing nature of the CC. This is absolutely the top requirement needed in cosmology today.
ReplyDeleteIn the early universe we know the energy density had to be high enough to form the fundamental particles. This is associated with extremely high temperatures. Today we live in a low temperature and a low vacuum energy density regime. This corresponds with an ever decreasing acceleration outward of the universe.
Until the 1a supernova acceleration was observed in 1998 the prevailing foundation of physics was that energy is dual in its nature. Negative energy (as yet undefined) balanced the known positive energy to form a net measurable zero vacuum energy. In this previous scenario an infinite vacuum energy was allowed.
We now know infinite energy and an infinite universe probably does not exist. Instead of the universe being dual in its nature we should start striving for a UNITARY theory in which the total energy of the universe is finite and conserved. This is completely consistent with a very high density universe inflated very rapidly in the beginning with extremely high temperatures. Today that acceleration continues but at an extremely slow rate and with low temperatures. This is consistent with our understanding of solid state physics also, like expanding gases etc. This energy gas can change phase as it cools to form new particles made from existing particles. They don't have to be made from new particles, like Wimps are presumed to be made of.
The main point I'm making is that you have a fancy name for your theory but you are not using known facts that have been discovered in the last 15 years to constrain it. I'm sorry to say that Fractal Cosmology and an infinite energy background for the universe is much like epicycles in view of this new knowledge and our evolving understanding of the unitary energy nature of the universe.
ReplyDeleteHi Eric,
I have a paper at arxiv.org entitled "Towards A Resolution Of The Vacuum Energy Density Crisis". This discusses the vastly disparate energy densities of the microcosm and the macrocosm.
You might be surprised to find that the discrete fractal paradigm offers a rather elegant solution to this very serious problem which has indicated for some time that something is seriously wrong with the standard paradigms of particle physics and cosmology.
Discrete cosmological self-similarity also gives us elegant explanations of the physical content of the fine structure constant and Planck's constant.
In fact it does everything one would ask of a new unified paradigm. All it lacks is a celebrity physicist as its author.
"In fact it does everything one would ask of a new unified paradigm. All it lacks is a celebrity physicist as its author."
ReplyDeleteGet Bee to volunteer!!
ReplyDeleteThe person who succeeds in getting recognition for the new paradigm offered by Discrete Scale Relativity is most likely to be a very young person who has superior analytical skills and will be able to unify GR and QM with sophisticated mathematical modeling based on nonlinear dynamical systems and discrete conformal symmetry.
I also expect DSR to successfully predict the exact nature of the galactic dark matter, whereas the old paradigm will have badly failed this crucial scientific test.
Phillip,
ReplyDeleteRobert is possibly referring to the value of the Hubble constant that Planck has measured to be somewhat lower than the local measurements. Alas, it's a two sigma discrepancy or something. So it's interesting, but saying that homogeneity has been "falsified" is really reading too much into it. Best,
B.
"The main point I'm making is that you have a fancy name for your theory but you are not using known facts that have been discovered in the last 15 years to constrain it."
ReplyDeleteIn particular, as I have pointed out in various places before, a definitive prediction, mentioned in the abstract of the main DSR paper, has been definitively falsified. That's not tenacity, that's deliberately ignoring relevant information.
ReplyDeleteYou are incorrect about your supposed "falsification", but I don't expect you to be sufficiently unbiased to admit that, and perhaps you cannot even imagine that you might be wrong.
The bottom line is that I do not spend much time arguing with highly biased true believers. They change the rules of science to fit their agenda. An excellent example of that is the Dawid (sp?) book making for string theories failures being hotly discussed at Not Even Wrong and vixra log.
Hi Robert,
ReplyDeleteI'm feeling bad about this ganging up on you but it's not personal, at least from me. There's just several things that I think you've got wrong.
A really big one is that string theory is biased towards an infinite energy background, just like your fractal cosmology is. It is also based on a zero based "measurable" energy background, just like standard quantum physics. It is a background dependent theory based on zero measurable energy. That is why it strongly supports super symmetry in which the measurable background is zero.
What I'm saying is that your theory, in which new particles can be formed on larger and larger scales has a lot in common with ST and supersymmetry, at least as far as far as an infinite background is concerned. It probably isn't really in your interest to come down so hard on it. It would be a natural ally to you in that regard.
I don't agree with any of that though. I think in particular that there is a big thing in quantum mechanics that is missing that changes the background. My personal view is that spin angular momentum changes during acceleration but people have misinterpreted quantum mechanical spin as a constant for fermions and bosons. My suspicion is that spin 1/2, 1, etc is really a measure of the combined kinetic plus potential energy of fermions before and after acceleration. That way quantum mechanics would no longer be background dependent, as it should be.
I should add, if it isn't already obvious, that if quantum mechanical spin was a measure of the sum of kinetic and potential energy then that sum would appear as a constant - the Planck value/integer.
ReplyDelete
ReplyDeleteDiscrete Scale Relativity predicted pulsar-planets before they were discovered.
DSR also predicted large numbers of MACHO objects and unbound planetary-mass objects before they were discovered.
DSR offers a more accurate retrodiction of the proton radius than QED.
DSR predicted the unusually low abundance of planets orbiting the lowest mass M-dwarfs before that was discovered.
DSR predicted that the peak of the planetary mass function would occur at about the mass of Neptune before that was discovered.
What I see is biased people riding their personal hobby horses, making a host of unsupported pronouncements about how nature is or is not, and purposely misleading others.
Meanwhile the most promising new paradigm in a very long time is completely ignored.
Pathetic!
/* proposal I think is brilliant: A labeling system for scientific news "that made clear a given study’s place in the scientific process"... */ Isn't it just another attempt for Malleus maleficarum book, which would enable to judge the ideas by their intersubjective parameters (analogous to citation index) instead of their factual content?
ReplyDelete/* ranging from the speculative idea and preliminary results all the way to established scientific theory... */
ReplyDeleteIn another article you're promoting the basic research and just this research wouldn't get a high score in this scale. In particular, given the fact, that so far the quantum gravity theories failed in most of their predictions, I'm afraid, just your research wouldn't get a high score by these criterions. Are you really sure, you still want such a labeling?
Hi Robert,
ReplyDeleteYou misinterpreted me. I wasn't saying fractal cosmology was similar in its conceptual roots to string theory. I was saying that one of its main attributes, an infinite background energy, is also contained in string theory.
Also, you should understand that Peter Woit is constantly encouraging a unitary understanding of mathematical physics. This completely disagrees with your infinite energy background in DSR. That's just a fact. While the conceptual basis for DSR is completely different than string theory its main mathematical basis rests on the same assumption as in conventional quantum physics and in string theory. You just have to accept that.
I, myself, did not understand that was in your theory until a few months ago, when you argued that an infinite universe was ok because that was what most physicists believed. So it does not appear to me that you are at all the iconclast you would like to believe.
ReplyDeleteEric,
Do you understand how a discrete fractal cosmology, i.e., a hierarchical model, avoids Obler's paradox.
Show me you know how this works before I waste more time on your obsession with mass/energy totals, and the fact that infinite values are a non-issue.
I'm not the one hawking a theory. You are the one trying to sell everyone on DSR. You have the burden of proof. I think your haughty demeanor is your worst enemy. Anyone can retreat behind a facade of superiority to avoid answering questions. It convinces nobody.
ReplyDelete
ReplyDeleteI was very skeptical of the firewall pseudo-science, but I must admit I have observed some flaming holes with limited horizons lately.
What, me? Just because I pointed out the mathematical commonality your theory has with standard qft, string theory, and above all, its similarity with the multiverse? A discreetly ordered structure with different scaling that can be added infinitely to the existing but differently scaled order is very little different from the multiverse. Both the multiverse and discreet scaling use the convenient excuse that since you can't see it we can pretend. Both rely explicitly and lazily on infinite energy that doesn't force one to use our brains to constrain the theory. Both are just remnants of laziness and arrogance.
ReplyDeleteEric, Robert: Could you please either come back to the topic of this post and take your bickering elsewhere? Thanks,
ReplyDeleteB.