Robin Hanson's proposal is an interesting scenario which aims to solve problems in scientific research by betting on scientific theories. He starts by summarizing the 'Problems with Academia' and dramatically sets the stage for his suggested cure 'Academia is still largely a medieval guild, with a few powerful elites, many slave-like apprentices, and members who hold a monopoly on the research patronage of princes and the teaching of their sons.' I almost feel sorry for myself.
But he has a point that the current academic system has some weaknesses. I don't want to repeat my take on the problem here (for that see Science and Democracy I, II, and III), so let me just state how I understand him. Hanson claims that the way researchers are awarded and appreciated today favors 'being popular, fashionable, and eloquent, instead of being right'. There is certainly some truth in that, though I don't think the situation is as bad as he makes it seem. Anyway, I was interested enough to learn how he thinks one can improve the situation.
So, he goes on explaining that the process by which 'the academic' receives 'credit' for making claims does not work optimal because academics judge on the quality of their own work, and form a rather closed system in their expertise. He argues that we need a 'specific mechanism' that 'encourages honesty and fair play; the game should be open to anyone to prove him/herself.'
His suggestion for such a mechanism is to bet on the outcome of scientific questions. 'Somewhat like a corn futures market, where one can bet on the future price of corn, here one bets on the future settlement of a present scientific controversy'. This, so he argues, would reward being right over being popular once the question is decided. The betting market would be open for everyone, regardless of education or degree and thus dissolve the boundaries of the rather closed academic system. Since those who eventually were right with their opinion about the scientific question, will win their bet 'Everyone would have a clear incentive to be careful and honest!'. That is, he claims financial profit is the reward that will convert the bad unethical academic to a good and honest guy.
He then investigates several cases and explains how his proposal that he calls 'Idea Futures' would work (see also Wired: Idea Futures).
After this follows a long list of possible objections with answers (which I found really a good idea. I might include something like this in one of my papers as well). Some of the questions came into my mind as well, and I will only pick out these. I don't care very much whether gambling is legal, but one obvious point is that there is no real profit in that game. If the issue to be decided is one of application, there could be real profit but then the 'real thing' to deal with are patents. Hanson's reply to this is 'Being monetarily zero sum does not make betting useless'. No, it doesn't make it useless. But it makes the whole idea far less attractive for anybody interested in making profit - and let us recall that making profit was the motivation he assumes participants have.
He also addresses the point 'Is there enough interest in science questions?' but if he answers it, I don't see how. The rather vague answer is basically 'Idea futures would thrive if it tapped only a small fraction of current interest and effort.' He also points out that it would be important to have the bets properly formulated, and the Idea Futures market therefore needs a certain amount of management and organization to operate smoothly 'Claims should avoid slippery concepts and phrasing which allows many interpretations', a demand that I totally agree on. If such a management would exists, first thing I'd do is require it for all scientific publications... Do we really solve any problem here?
So, here is my opinion: Gambling can not save science. To begin with, science is not lost, and doesn't need to be saved. In fact - excuse me - but I have the impression overall seen it works quite well.
But more importantly, one has to ask exactly which problem Robin Hanson aims at solving. On the long run, there is nature which decides on the value of scientific claims. As I have pointed out in an earlier post, a problem arises in the time between the proposal of a new idea and its verification (or falsification). In the absence of nature as a judge, we have to find other ways to judge on the quality of our work. This is what I call 'the measurement problem'. It is a crucial point because misjudgement in the meantime might mean that the best proposal doesn't survive long enough, or can not be worked out sufficiently, to be verified at all.
This however is not the problem Hanson addresses. Also in his scenario, there is only one ultimate judge that decides who wins the bet, and that is still nature. Though it is not stated explicitly I think he is aware of that: 'Most scientific controversies seem to eventually get resolved enough to settle a bet. [...]. Scientific claims are often defined as claims of "fact" which future evidence could possibly disprove [...]', 'a question about a physical property of a substance, like a bond angle of some new molecule, seems quite resolvable. As a rule, one should prefer questions closer to direct observations'. This means he can not solve 'the measurement problem' on the 'marketplace of ideas'.
Instead, the problem Hanson is concerned with is the time between nature's judgement that promotes one theory to be the best, and the time its inventor receives credits for it. In fact, this is also the problem that he investigates in his examples. I have no doubts that there have been, and still are, sad cases in which the value of a model was only recognized with delay. I do not think however, this is an inherent sickness of academia, but rather the exception than the rule. It is not the main problem that concerns me. As academical research works today, everybody who would learn of a model that explains observations better than the present one, would jump on it and use it. The remaining problem is then one of information distribution, which I believe is, and will further be, significantly improved though the ongoing changes in scientific publishing.
But besides these details, the whole idea fails on a very obvious point: There are undoubtedly scientists who like gambling and betting, as there are in every other profession. But the majority of scientists is not in academia because they want to make profit by investing smartly. If that was their motivation, they had not chosen academia in the first place. I dare to say, the vast majority of scientists would completely ignore the presence of such a betting market (as they did already ignore the proposal altogether). Even if it would possible mean they make profit, they wouldn't be sufficiently interested to pay much attention to it.
Financial profit, which Hanson assumes is the driving motor of everything ('Those who invest wisely would accumulate capital and gain influence, which they could reinvest in discretionary research or in influencing future consensuses.') is just not what drives most scientists. It is probably the case for some, but these are exceptions like those who are eager on 'being popular, fashionable, and eloquent, instead of being right'.
No, the majority of experts wouldn't pay any attention to that betting market. And without experts opinion, the whole system becomes no better than a lottery. Everybody who seriously wanted to make money would chose the stock market. For example, we could ask are neutrinos Dirac or Majorana? That I would think is a question that qualifies for the proposal, and which can be decided examining neutrinoless double-beta decay. But what is going to change for anybody whether or not some physicists or laymen have fun betting on one or the other?
I guess that there would indeed be people - experts and layman - interested in betting on scientific ideas. This is done anyhow, but such a leisure-time activity will not have any backreaction effect on the scientific research itself. It can be quite entertaining though, see e.g. Physicists who fancy a flutter.
The bottomline is: Robin Hanson's proposal is a very interesting toy model, and I give the idea a high creativity and entertainment value. But its a toy model nevertheless. Based on the assumption that maximizing profit it the driving motor for researchers, it is destined to remain an empty farce because it will not attract sufficient experts. These, I am afraid will remain busy drinking coffee with their 'many slave-like apprentices'.
Update: See also Robert Hanson's reply
TAGS: SCIENCE, SCIENCE AND SOCIETY, GAMBLING
"Financial profit, which Hanson assumes is the driving motor of everything ('Those who invest wisely would accumulate capital and gain influence, which they could reinvest in discretionary research or in influencing future consensuses.') is just not what drives most scientists. ".
ReplyDeleteAgain with these silly "free market" ideas! Hanson does not understand scientists, who do not go into Science to become rich, but by and large, because they are curious people who want to understand how Nature works.
"Financial profit, which Hanson assumes is the driving motor of everything ('Those who invest wisely would accumulate capital and gain influence, which they could reinvest in discretionary research or in influencing future consensuses.') is just not what drives most scientists. ".
ReplyDeleteAgain with these silly "free market" ideas to "save" science! Hanson does not understand that scientists do not go into Science to become rich, but by and large, because they are curious people who want to understand how Nature works.
changcho
Dear Bee,
ReplyDeletevery interesting thoughts! While reading I was thinking something like what you then concisely formulate as
Without experts opinion, the whole system becomes no better than a lottery.
Just betting to prove him/herself doesn't make sense in science, at least to me, unless you have some well thought-about argument for your case. Being right just by chance is not the point. Those who are right because they had the correct and crucial ideas are the ones who get citations in later papers, not usually those who made a random guess that came out to be right...
Best, stefan
Hi Stefan, you are right
ReplyDeletewithout experts the whole thing becomes a lottery (life is a lottery) and betting to prove oneself doesn't make sense unless you have a well thought out argument for your case. But trying to prove oneself or 'our' theory is what most of us do in life.
"Hanson claims that the way researchers are awarded and appreciated today favors 'being popular, fashionable, and eloquent, instead of being right'. There is certainly some truth in that."
Bee, that has certainly been the truth throughout history and today.
In most fields, like medicine, it is whatever is the darling of the day that pays the most and reaps the rewards and highest funding for research - of course those in the field need to have some expertise whether it be cancer, heart surgery or brain surgery - but they are not gods - and the patients may die, and any failures or shortcomings are swiftly swept under the carpet ...
Often they are not what society necessarily most needs, but what the people able to pay and are most willing to pay for, determines where funding goes.
After all in Theoretical Physics another theory offering something more promising or 'concrete' than ST or M-Theory, that is where the funding would go. Otherwise ST and M-Theory are just better at PR and securing funding.
In Medical Research we actually spend more on research on possible future cures (no matter how bizarre or absurd) like anti-ageing or immortality, than making available what would improve the conditions of patients today - in some things, basic things which we have still not mastered.
You know like trying to build a roof of the house, but forgetting to check if the foundations are on solid ground or the walls able to support the weight
After all research into finding treatment for genetic anomalies, seems itself an anomaly, but of course the profit is not in curing or preventing genetic anomalies, but in treating and managing disease, and marketing ever more treatment and drugs at greater and greater costs.
Gravity is where it's at. Magnetism and electromagnetism we can manipulate, gravity we cannot - yet
Betting your reputation is a principal component of this idea, Bee. Think about how valuable is your reputation.
ReplyDeleteAargh, Bee, your post is too good - it is uncontroversial to the point that it will receive very little feedback! What is the point of arguing if we all agree ? :)
ReplyDeleteYou said it too well for anybody to feel compelled to try their own hand at it. "Financial profit is not what drives most scientists".
So, now that we all agree, I'll get myself a beer.
Cheers,
T.
I just discovered that the Science News article on this topic is openly available. It was one of the more balanced pieces at the time, when the rest of the worlds press went bonkers (over PAM).
ReplyDeleteHi Amara,
ReplyDeletethanks, this is interesting, I didn't follow this in 2003. There is a nice sentence in this article that describes my objection very nicely Economists have found, for instance, that orange juice futures predict the weather in Florida better than conventional weather forecasts do. Yes. This works because literally everybody has experience with the weather and everybody can say something about it. But in a certain sense, one could say the whole business of science is already predicting the future. One can of course argue that we can trade around our opinions more efficiently, but the circle of those who can make sensible predictions for models isn't going to increase if we 'open' the market. As I said above, I don't think it would do much harm either, but I doubt it would help with the most pressing problems.
Best,
B.
Opening it up, might allow people who actually do have a little bit of knowlege but wouldn't normally participate.
ReplyDeletePlus are you sure that scientists don't like to bet? One of your commenters here is even documented, setting up a bet. Check out the Long Bet, by the same people (Stuart Brand et al), who started the Long Now Foundation.
Hi Tom,
ReplyDeleteIt's my harmony day. I don't have a constant urge for controversy. Though it's tempting sometimes. I just had to insult a couple of big-names and whoops there peaks the visitor number.
Hi Quasar,
In most fields, like medicine, it is whatever is the darling of the day that pays the most and reaps the rewards and highest funding for research
Yes, this is a very unfortunate development, and it is imho completely obvious that this kind of reward system is not beneficiary for enduring progress. That's is why I argue the present 'measure' that says who is today's darling badly needs to be readjusted.
Hi Stefan,
you know, while I was reading this proposal I was reminded of a weird political utopia that I read some while ago. In this book (sorry, forgot name and author), elections were replaced with random choices for the people's representatives.
Though in this case the 'representation' of the public is more truthful in the meaning that the randomly picked politicians (most likely) represent the average, it is totally against the idea of representative democracy. We don't want to elect people just because they are like we are, but because they are 'experts' in how to realize our interests.
In the same sense, science is left to the experts, much as I leave housebuilding to the architect and surgery to the doctor. One can of course argue whether The Shop of Science is too closed, as one can argue for many other professions as well. But imho this should be cured by, and only by, reconsidering the importance of titles and educational degrees. Best,
B.
Dear Amara,
ReplyDeletePlus are you sure that scientists don't like to bet? One of your commenters here is even documented,
Thanks, I have linked to exactly the same article in my post. As I have written: "There are undoubtedly scientists who like gambling and betting, as there are in every other profession." and "I guess that there would indeed be people - experts and layman - interested in betting on scientific ideas."
Opening it up, might allow people who actually do have a little bit of knowlege but wouldn't normally participate.
I am sorry if I did not make my point clear. I have no problem with opening up betting on scientific questions. I would even appreciate it for the general reason that it might draw attention to our efforts! Just that it is not going to solve the problems within our community. This is why I answer the question raised in the title of the proposal with no.
Best,
B.
Betting your reputation is a principal component of this idea, Bee. Think about how valuable is your reputation.
ReplyDeleteDear Amara,
aren't we already betting on our reputation? Am I the only one who has stage fright when hitting the 'submit' button on the arxiv? Our work is attached to our name - whether or not we bet money on it. Best,
B.
Oh sorry, I must apologize. I was reading too fast and missed that sentence and link.
ReplyDeleteOne last suggestion, since my brain shut off some time ago. If you like to escape into science fiction sometimes, some of the Ideas Future stuff has been incorporated into a fantastic novel: Earth Web, by Marc Stiegler. The book is much more than the betting ideas, but you can see some 'every day scenarios (ok, in sci fi) where this idea might be put into practice.
What would Darwin say?
ReplyDeleteDear Bee: I was writing too late last night and didn't finish answering your questions.
ReplyDeleteYes, I get stage fright when I hit the arXiV article-send button too. Yes, that's staking your reputation. I think that your name is one of the most important things that you have in your career. I've spent the last year in a legal snafu with a publisher and photographer who were using my name on text that I didn't write, so I know from my side that I'm willing to fight about it.
However, when one places a bet, I think that that indicates a great confidence on that idea, that is as great or greater than the value that we ourselves place on our reputation. Perhaps you and I have unusually high standards for our reputation which leads to that arXiV stage fright, but maybe not. I know that I would think just as hard on an idea and be just as keenly aware of my reputation, in order to place a bet, as I would to publish that idea in a scientific paper.
Another thing that betting can do is provide 'shades' of confidence in an idea, which is especially useful if there is still little information for that idea. Here is an article for how Idea Futures could be used inside of the corporation.
In the stock (and futures) market there are thousands of perfectly valid and effective ways of making money. The difficult in finding and using one is that there are millions of ways of losing money.
ReplyDeleteThe majority of the people who play these markets do not make money, but the the minority who do make money make so many trades and trade in such large amounts that they are the largest single reason stocks and futures move.
Of course there are people who learn how prices interact with climate (and a bunch of other things). They are not regular people who have the same experience and understanding of the weather as the rest of us. They are very specialized and learned their trade with difficulty, care and occasional disasters. This is one of the thousands of ways.
The first mathematical talent that a trader must possess is the ability to control risk. Even a gambler with an edge against the market will lose all if he bets too large.
The second mathematical talent is the ability to statistically differentiate between luck and skill. Best is to learn in a form that allows hundreds of trades per day. This is how the traders on the floor of the major exchanges learn their trade. Buying MSFT and holding it for 20 years is (a) not trading and (b) is not much statistical evidence of a future ability to beat the market.
Trading is a good way of making lots of money, but for a person interested in science, it is very boring.
More free market nonesense form a worshipper of the greedy cheating class and the corpration
ReplyDeleteThe computer he turns on everyday was a direct result of massively violating free market principles for decades. Browser technology,lithography and many other technologies that gave rise to the PC and interenet revolution are all the result of the massive violation of free market principles.
Thank God for DARPA and other publicly founded high tech centrally planned programs.
Even James Simmoms Hedge fund trading shennangins depended upon decades of massively violating free market principles.
Drive past Renaisance Technologies on 25A and take a look at the technology that reaches towards the sky retrieving market information from around the world.
Another gift from the taxpayer.
No one should be allowed to be billionaire.
Fuck the Free Market. I'm sure God feels the same way
Anon on the Hudson
Dear Amara,
ReplyDeletewhen one places a bet, I think that that indicates a great confidence on that idea, that is as great or greater than the value that we ourselves place on our reputation. Perhaps you and I have unusually high standards for our reputation which leads to that arXiV stage fright, but maybe not. I know that I would think just as hard on an idea and be just as keenly aware of my reputation, in order to place a bet, as I would to publish that idea in a scientific paper.
It seemed to me kind of a crucial point in the proposal that you don't have to bet on your own idea. The reputation factor that comes with the bets is important only if you bet publicly. Do you want to force scientists to publicly give 'shades of confidence' to their own and other models? How many would do that? Does science work this way? In the betting scenario, doesn't your reputation much more depend on how many people support (bet on) your proposal, instead of on you supporting your own idea? And isn't that already done? We aren't betting, but the impact of our publications depends on how many people find the idea promising and start working on it.
Another thing that betting can do is provide 'shades' of confidence in an idea, which is especially useful if there is still little information for that idea.
True. Again, I don't say realizing this idea would be useless, I'm just saying it doesn't work and doesn't solve any problems.
See, if I read scientific papers and try to get an opinion, I don't translate my 'shades of confidence' in 'amounts of money that I bet on it', and I don't want to spend my time doing so. In most cases, it's more that I like one aspect of a model, but think another one might be improved if one tries this, or this, or maybe that... and then I grab my notebook, instead of placing bets on it.
Luanne has asked the right question:
What would Darwin say?
Well, he would tell you that with this rewarding system you are going to grow a generation of scientists who 'fit best' and 'survive' if they are good at betting and gambling (all the points that Carl mentioned above), instead of being creative and careful thinkers. It's a distraction from our priority: doing research. Though I love distractions :-) that shouldn't be part of our job.
Best,
B.
Hi Bee,
ReplyDeleteI cetrainly agree with you that gambling is not able to make an "ideal future", in particular, not able to 'encourages honesty and fair play; the game should be open to anyone to prove him/herself.
The most manifest character of gambling is its high profi with a high risk. This determines that gambling, although requires intelligence to some extent, also relies very much on guts, phsycological skill, especially a good disguise of being honest, and even on luck. It is hardly for a real gambler to be truly honest, although he/she may appear to be.
More importantly, to be wealthy is absolutely not the goal of most serious scientists, but rather to decode the hidden natural truth.
Best, Y.
"However, when one places a bet, I think that that indicates a great confidence on that idea, that is as great or greater than the value that we ourselves place on our reputation. Perhaps you and I have unusually high standards for our reputation which leads to that arXiV stage fright, but maybe not. I know that I would think just as hard on an idea and be just as keenly aware of my reputation, in order to place a bet, as I would to publish that idea in a scientific paper."
ReplyDeleteHi Amara - please understand, as surely you must, that whenever one attempts to publish a paper, one is already indicating "a great confidence on that idea" *without having to have a financial transaction* (as you would in a real bet).
Cheers.
changcho
I don't know how else to state what I mean regarding reputation and betting other than the way that I already wrote (in that quoted paragraph of mine). You can try a gedanken experiment on yourself regarding your degree of confidence on a subject (your own, or others). I cannot add too much more to this topic, except point to another article that might explain a different, useful perspective. This is one by David Brin, another physicist (like Robin Hanson; his second career followed his physics career).
ReplyDeleteA final note: there is a large field of statistics called Bayesian statistics (sorry, a joke, instead, try this), that dovetails directly into this topic too because an estimation of an idea's "truth" is a natural part of the calculation.
One only needs to survey the world's universities - their vast offering of scientific studies, and the value society places on scientific education, to conclude that science is in fine shape.
ReplyDeleteThat said, certain branches of science, notably biology and physics, are under assault by the rising tide of religious fundamentalism in certain developed western countries (we know who they are so I don't need to name names). If the religious fundamentalism people keep their belief to themselves, OK. But they do not. They are active in public promotion, and infiltration of their religious conversion agenda in schools to try to convert the young by discrediting the findings of important scientific theories. It is sad, and is an indication of our times, that many are listening to the teachings of Christian fundamentalism as replacement to scientific knowledge. The current debate on the merits of string theory have played into their hands as yet another example of the failure of science to address deep questions. Scientists should refrain from 'hyping' scientific speculations for economic gains because that will compromise our credibility in the public eyes. We should also minimize further polemics in the string theory debate because it is already being exploited by religious fundamentalism.
I urge scientists to do their part in responsible public debate and education, beyond the limited agenda of most outreach programs.
Alas your post came when I was distracted with several other things. But if you can tolerate delays, I'd like to pursue the conversation. I think you misunderstood several aspects of my proposal. I will start by posting on the topic tomorrow morning over at OvercomingBias.com. Then if your are willing we can (slowly) continue the conversation in whatever forum you prefer.
ReplyDeleteYou object Financial profit ... is just not what drives most scientists, but previously you noted Research funding has to be distributed to the most promising researchers. These decisions have to be made based on something. These decisions ARE made on the basis of SOMETHING. But something is not good enough.
ReplyDeleteCaring about funding is caring about money. I propose that science labs acquire some of their money to fund research from winnings in subsidized betting markets on science topics.
As a real-life capitalist I find this proposal of betting on science as a way of funding to be kind of disturbing. On the one hand it's very similar, on the surface, to the lotteries that fund higher education for the poor (the Tenn. state lottery does this). People bet their money that they will win something in addition to helping provide an education for those who probably wouldn't otherwise afford it. Most probably don't really consider that they are supporting further education: they are just hoping to win more than they've spent.
ReplyDeleteWell, sorry, I've lost my thought now. My 6 year old is home very ill with mono and has distracted me. ;-) But he wants me to spell "China" for some reason, so there it is. :-)
Hi darkmatter,
ReplyDeleteScientists should refrain from 'hyping' scientific speculations for economic gains because that will compromise our credibility in the public eyes. [...] I urge scientists to do their part in responsible public debate and education, beyond the limited agenda of most outreach programs.
Thanks, I couldn't have said it more cleary.
Hi Amara,
:-) I like conspiracies. On some days I am perfectly convinced, that the theory of everything has been found already in the last century. But since scientists didn't want the End of Science to become reality, they decided to keep quiet about it, and all of science is just a game they play. If you get too close to the truth, a guy with sunglasses in a dark suit will show up in your office and tell you: look, we have the theory of everything. you can either join us and play our game, or you'll get a chip implanted in your brain that makes you a life-long researcher of zing-zong theory who will never get anywhere.
Sorry about that distraction. Where was I...
I don't know how else to state what I mean regarding reputation and betting. I think I understand what you are saying, I just do not agree, that placing a bet indicates a great confidence on that idea, that is as great or greater than the value that we ourselves place on our reputation. I do not see how betting would lead to better 'predictions' of the future value of theories. How and why would thinking in terms of bets improve experts judgements on theories, notwithstanding the problem that they most likely wouldn't want to think in these terms?
Best,
B.
Hi Robin,
ReplyDelete"You object Financial profit ... is just not what drives most scientists, but previously you noted Research funding has to be distributed to the most promising researchers. These decisions have to be made based on something. These decisions ARE made on the basis of SOMETHING. But something is not good enough.
Caring about funding is caring about money. I propose that science labs acquire some of their money to fund research from winnings in subsidized betting markets on science topics."
I didn't say scientists don't care about funding. I said they are not driven by financial profit. Caring about funding is unfortunately a necessity, but I doubt acamedics want to pay even more attention to such a market than is already the case. The most severe problem I see is actually the just generally chronically short funding for fundamental research.
You can of course propose that. I just don't think it is going to solve any problem. At least not the ones I am concerned with. I don't see how such a way to judge on the promises of research fields would be any better than what is already done.
Best,
B.
You say, I do not see how betting would lead to better 'predictions' of the future value of theories. How and why would thinking in terms of bets improve experts judgements on theories...?
ReplyDeleteSo far when we have compared the accuracy of betting prices to other kinds of field forecasts, the prices have been at least as accurate. The theoretical reason to expect this is incentives - other forecasting contexts don't reward accuracy as clearly or strongly.
The true incentive of scientists is to understand. Of course, the money is needed to pay the bills, but again, fiancial profit is not the motivator of (most) scientists. In this way, scientists are more like artists: the incentive of a painter is to paint, and the incentive of a writer is to write.
ReplyDeleteReally, financial profit does not drive *everything*...The problem is that most economists assume that it does and then try to apply that principle to situations where it is not appropriate.
changcho
changcho, saying economists assume that people only care about money is about as silly as saying physicists assume only velocities are relevant for predicting object behaviors. No one who has passed an intro course in econ or physics should think either.
ReplyDeleteHi Robin:
ReplyDeleteNobody claims economists assume that people only care about money. But what you are trying to introduce is a one-dimensional measure on scientific ideas that is financial profit. Though this measure has proven useful in many regards, it just is not possible to measure everything in these terms.
What I have tried to communicate is that the opinions scientists hold about their and other theories can not simply be translated into the question: how much money would you bet on it? What your proposal would support are people who are willing and able to express their opinions in such a way. The situation is much more involved; I doubt many academics would think that their opinions can accordingly be expressed in such a way, and would not want to spend time trying to do so. We argue with each other because we can not just say, I give approach A five points more than B.
Besides my objection that scientists would not want to support such a measure, another important point that you have ignored is the requirement to cleanly formulate questions that have a distinct outcome. The main problem that we are facing is not to predict weather theory A or B will better fit with experiment X. The problem is how to invest in order that new theories are found at all!
To give you an example why your proposal fails in important areas, take the current debate around string theory. What can you contribute to that? Do you want people to bet on whether string theory is right or wrong? There are people who claim its not even wrong, so to begin with, how exactly would you address the issue. There is currently no experiment scheduled to test it. How do you decide whether you'll hire a string theorist next year?
I'll give you another example. Take the question whether neutrinos are Dirac or Majorana, and lets assume neutrinoless beta-decay is detected. Now, institute XYZ who predicted the correct outcome gets rewarded for its good forecast and wins some money. The crucial question is then, who will they support with the money? Does betting on majorana mean they are more able to hire promising candidates?
Your proposal might optimize selection among already existent approaches that are testable, or will be testable soon, but for enduring progress it needs more than that.
Best,
B.
To Robin Hanson:
ReplyDeleteWell, perhaps I was oversimplifying. But I do see a lot of economists use the "profit" motive as a sole guiding principle, and who then extrapolate it to situations where it is not applicable. In any case, it is all a matter of realm-of-applicability. Most physicists don't "assume only velocities are relevant for predicting object behaviors ", unless, as is my case, you are interested in the dynamics of small solar system objects (in which case you are also interested in their positions!). My "Laplacian Demon" is the appropriate hardware/software combination; and Laplace was not "silly": his vision was limited because of when he lived. But again, it is all a matter of applicability: if I were interested (from a research p.o.v., because I am interested from a layman's p.o.v) in high energy physics, then I'd care little about positions and velocities, and I would start with the Standard Model as a guiding principle.
In any case, imho I don't think your scheme of betting would work, for the reasons that Bee has already outlined much better than I could hope to. But you are of course most welcome to prove me wrong!
Cheers.
changcho
":-) I like conspiracies. On some days I am perfectly convinced, that the theory of everything has been found already in the last century. But since scientists didn't want the End of Science to become reality, they decided to keep quiet about it, and all of science is just a game they play. If you get too close to the truth, a guy with sunglasses in a dark suit will show up in your office and tell you: look, we have the theory of everything. you can either join us and play our game, or you'll get a chip implanted in your brain that makes you a life-long researcher of zing-zong theory who will never get anywhere."
ReplyDeleteLOL, that was funny. Thanks. But it does sound very much like a religion. ;-)
Dear changcho, Were you aware that Laplace 'bet' on the mass of Saturn?
ReplyDeleteIn the early 1800s, Laplace rediscovered Bayes' work, and he generalized it and applied it to problems in astronomy, celestial mechanics, meteorology, population statistics and other fields. The version of Bayes Theorem that staticistians use today is actually the generalized version due to Laplace.
One particularly nice example of Laplace's Bayesian work was his estimation of the mass of Saturn, given orbital data from various astronomical observatories about the mutual perturbations of Jupiter and Saturn, and using a physical argument that Saturn's mass cannot be so small that it would lose its rings or so large that it would disrupt the Solar System.
Laplace said, in his conclusion, that the mass of Saturn is (1/3512) of the solar mass, and he gave a probability of 11,000 to 1 that the mass of Saturn lies within 1/100 of that value. He should have placed a bet, because over the next 150 years, the accumulation of data changed his estimate for the mass of Saturn by only 0.63%!
Hi Amara - No, I wansn't aware of Laplace's "bet"; interesting. But he was doing what was most natural to him, finding out estimates of the mass of Saturn with the available data to him. Maybe Laplace himself should have placed a bet, though he did not, as far as we know?
ReplyDeleteCheers.
changcho
What you are trying to introduce is a one-dimensional measure on scientific ideas that is financial profit. ... it just is not possible to measure everything in these terms.
ReplyDeleteBetting prices can produce as many dimensions of estimates as one desires.
I doubt many academics would think that their opinions can accordingly be expressed in such a way, and would not want to spend time trying to do so.
As I commented at Overcoming Bias, if a supposed insight does not help you form a better probability estimate on any other topic of interest, it is not clear you have an insight. Can you give an example of such an insight?
Another important point that you have ignored is the requirement to cleanly formulate questions that have a distinct outcome
Some problems are just harder than others, for any institution. My proposal is attractive if it can do better than alternative institutions, even if no institution can do very well. When thinking is muddled, all institutions will do more poorly at rewarding insights; I don't see that betting incentives will do worse that other incentives.
Take the question whether neutrinos are Dirac or Majorana, and lets assume neutrinoless beta-decay is detected. Now, institute XYZ who predicted the correct outcome gets rewarded for its good forecast and wins some money. The crucial question is then, who will they support with the money? Does betting on majorana mean they are more able to hire promising candidates?
The whole point of choosing an institutions with good incentives is that we don't have to answer such questions. Each institute would look at the history of which types of methods and people have gained lots of info at low cost, and make its best guess about which and who to support. We can't do better than to give them the best info and incentives to make such choices well.
Hi Robin,
ReplyDeletethanks for your explanation. Let me try to put it like this. There are all these scientists who read lots of papers, fly around the world and talk on conferences, try to get their ideas down on paper, try to get an opinion about other peoples work, try to include new input into their thoughts. Forming an opinion about other people's work is part of this procedure. But in most cases its not a probability that comes out of it. Do you want to regulate the way scientists think - you are going to kill science, that I can assure you. To begin with, it would make the job extremely unattractive to the most creative and independent people.
Yes, you are of course right that to distribute funding, it is eventually necessary to break down all this information into a single number. Without doubt, this has to be done. It seems to me, you want every scientists to do that. This is where I object. I want researchers to focus on their research, without such additional requests.
Yes, you are also right with this: if you introduce such a betting market, and it is the way to get funding, then people will do it, like they do now pay attention to advertising, because without doubt this presence in the public domain influences funding agencies.
I neither like the one nor the other because I don't think it is the optimal use of scientists skills. It is an distraction, a waste of time and effort. You wrote that the market will eventually be mainly determined by the few traders who know the most. And I assume, these few are then self-funded through their successful trading. That is, one essentially has a group of people, independently funded, who is able to make a good judgement. This is what we need, I think we can agree on that.
Each institute would look at the history of which types of methods and people have gained lots of info at low cost, and make its best guess about which and who to support.
What I was trying to say was that an institute that did well in guessing about topic A and received money need not be doing well in making good guesses about which and who to support with that money. Or do you further want to request that each institute does rely on the betting market to show them where they should invest?
I doubt many academics would think that their opinions can accordingly be expressed in such a way, and would not want to spend time trying to do so.
As I commented at Overcoming Bias, if a supposed insight does not help you form a better probability estimate on any other topic of interest, it is not clear you have an insight. Can you give an example of such an insight?
Plenty. All the insights that are not well enough worked out so you don't find them in my papers. This is what I do for a living: transforming the 'supposed insights' into 'real insight'. Now you might say then its not clear that I have an insight. Right. The difficult part of my job is to find out what of the 'supposed' is 'real'.
I don't mind if somebody wants to assign a probability to my published 'insights'. I just don't want to be the one who has to do that. And again, I doubt many of my colleagues would appreciate if they had to spend their time such.
Best,
B.
Dear Bee:
ReplyDeleteI've read many times here and on overcomingbias that you want researchers to focus on their research, but I think they will always be 'distracted' by the managerial duties. Every line of work requires some kind of financial managing. In many businesses, there is a manager hired for that task. If you're a consultant, then you either do the financial managing yourself, or you hire someone. If you're a scientist, then your government-sponsored (usually) research institute has hired staff to handle the financial managing aspects. They are paid either as a civil servant or employee, or as a hired staff using the money from that overheard that you've added in your budget when you wrote that proposal.
If you're a scientist, and you want to avoid this extra work, then you have to make sure that you have money to pay someone to do that work. Otherwise time spent on financial managing is par for the course for scientists.
And a short note on this that you said:
"you want to regulate the way scientists think"
However, regulating is not in any part of his proposal.
Dear Amara,
ReplyDeleteI think they will always be 'distracted' by the managerial duties
I am afraid you are right. But this distraction should be as small as possible.
However, regulating is not in any part of his proposal.
Scientists need funding. They will have to do whatever is required to get it. As Robin himself stated scientists will have to play that game:
Scientists will do what it takes to get funding. If betting markets control funding, scientists will bet.
This is exactly what I am afraid of. What a proposal like this does is a change in the the environment in which science is done. It will awards people who think along the lines of maximizing profit on this betting market. It will select those who 'fit best' into this category, and make it harder for those who don't. Since I don't see any correlation between being a promising researcher and thinking along these lines, I can not support this strategy.
'Regulating' maybe was not the right word. It might not be the intention but it will be the effect. Similar to how the system now awards good advertisement tactics, the proposed change would award the skill in giving probabilities to existing theoretical ideas.
Best,
B.
"In many businesses, there is a manager hired for that task."
ReplyDeleteScience is NOT a business.
I don't think Hanson will convince many scientists with this scheme.
me: "I think they will always be 'distracted' by the managerial duties"
ReplyDeleteyou: "I am afraid you are right. But this distraction should be as small as possible."
Ok, but managing the finances is a part of daily life for every aspect of our lives; the scientists are not exempt.
One exercise that I encourage every publicly-funded scientist to do is to be a consulatant for a little while. Twelve years ago, I didn't have a choice because I was injured and needed to drastically change my work environment, but what I learned through those years is this:
I learned how many hours I need to work to earn the money to pay my rent and my utility bills. It further translated into how much sofware I needed to write, or how many lines of text into a document or article. I know that you will say that that is too constraining for a scientist to go through that exercise, but, we are publicly paid people, and I think that we owe it to those people, and to ourselves to know what our 'production capability' really is. How much are we really 'worth' ? How much are our ideas worth?
OK, so continuing with your comments.
First, remember what he he said that he is not presenting this as an idea to replace existing institutions, but instead, to provide alternatives to what is being done at the edges of the evaluation procedures.
You said:
"This is exactly what I am afraid of. What a proposal like this does is a change in the the environment in which science is done. "
Science will still be done the way that we do: ideas, collaborations, meetings, proposals, results published in scientific papers, and so on.
and you said:
"It will awards people who think along the lines of maximizing profit on this betting market."
'reward' might be a better word.
I see this as a way to provide _more_ information on the correctness of an idea. If some scientists's institute project manager is choosing the allocation of funds for a project, they have probably sitting on their desk results of that idea's success in published papers, etc. In addition, they have the 'value' of that idea as evaluated by a group of people who are interested in the idea (the Idea Futures value). It's that simple, I think.
P.S. Bee and Stefan: I'm in Heidelberg next week for a workshop and in Munich on the weekend. Is a visit between us possible? Better yet (for me because of the full workshop) would be if a visit to Heidelberg sounds appealing to one (or both? I have no idea of your Easter plans) I'm pretty busy preparing for too much travel and too many presentations, but if a visit seems possible, contact me offline..
ReplyDeleteThere are all these scientists who read lots of papers, fly around the world and talk on conferences, ... Forming an opinion about other people's work is part of this procedure. But in most cases its not a probability that comes out of it. ... you are going to kill science, that I can assure you. ... it would make the job extremely unattractive to the most creative and independent people.
ReplyDeleteWe agree that researchers naturally think about other research and form opinions of it. But this process does not naturally produce answers to questions like what articles should be edited how to be good enough to appear in what journals, or who should be hired where, or what project proposed should be funded. So researchers now have to take on the less fun task of translating their opinions of other research into answers to these questions. I'm just proposing we replace some of these questions with the more natural question of how does this research change probabilities of other important research claims.
Dear Robin,
ReplyDeleteSo researchers now have to take on the less fun task of translating their opinions of other research into answers to these questions. I'm just proposing we replace some of these questions with the more natural question of how does this research change probabilities of other important research claims.
This is not a question of whether researchers have maximal 'fun' or not. The question is whether resources (human and financial) are used optimally, when researchers have to fulfill this task besides doing research. The answer is no.
In a scientific environment that is as complex as today's, it takes a lot of time and effort to achieve an objective overview on the whole field and its dynamics. You can spend your whole day reading recent articles, talking to researchers, trying to get a grip on the latest data, writing emails, reading blogs etc etc. Being up to date is a full time job, not to mention the task of projecting all this information in a one-parameter set. If you introduce a system that rewards people for doing so, you redirect brain capacity in a way that is not optimal for progress.
Besides this, as I have repeatedly mentioned, you are missing the point that it is more important to find PEOPLE who are promising candidates to support than to find which project is currently the hottest. You decide on a project, then you can hire people to work on it. Fine - this is one part of the story. But you can't plan discoveries. If you want a system that guarantees enduring progress, you need it to support people, not already existent approaches.
Also, you have to assure that the financial support is stable and not subject to fluctuations. We can't let every 3 sigma bump have an impact on funding.
Dear Amara
'reward' might be a better word.
Sorry, typo, you are right. I meant 'reward' not 'award'.
Best,
B.
No gambling cannot save science unless the casinos begin to make donations to scientific research.
ReplyDeleteI take it that the idea here is in part to "out" now-silent dissent in science, breaking up some of the (apparent) monopolies, earlier; before, say 30 years have passed.
ReplyDeleteMoney provides some incentive to record your dissent, or express your approval of the herd's standard view.
Had researchers been able to sincerely record their gut apprehensions about string theory long ago, that might have been helpful.
Automatic surveys could do the same - but they can be gamed unless the respondents are forced to put some money on the line, or multiple voting somehow eliminated.