tag:blogger.com,1999:blog-22973357.post3188002578019196319..comments2023-09-27T07:44:19.769-04:00Comments on Sabine Hossenfelder: Backreaction: The Reproducibility Crisis: An Interview with Prof. Dorothy BishopSabine Hossenfelderhttp://www.blogger.com/profile/06151209308084588985noreply@blogger.comBlogger25125tag:blogger.com,1999:blog-22973357.post-10763456678637856172020-02-21T10:11:41.489-05:002020-02-21T10:11:41.489-05:00Dear Stor,
When I say "anyone" I mean e...Dear Stor,<br /><br />When I say "anyone" I mean even someone with a very weak knowledge in mathematics. <br /><br />I live in a country (which is not France, where I did my PhD) where it is not unusual for my colleagues to write the PhD thesis of their students. All of it. <br /><br />Some of my colleagues, with a PhD in mathematics (statistics) do not know what a limit means. Here we Bourbakihttps://www.blogger.com/profile/14739779159703428622noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-71230556394025031222020-02-21T07:02:08.910-05:002020-02-21T07:02:08.910-05:00Bourbaki,
you said that nowadays anyone with mone...Bourbaki,<br /><br />you said that nowadays anyone with money can get a PhD. All I was saying in my comment is that you don't even need to have money. Anyone can get a PhD money or not. I didn't have any money, but I did get my PhD in pure math in a US university.storhttps://www.blogger.com/profile/08963157059325167335noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-64148596049576202512020-02-20T10:40:52.002-05:002020-02-20T10:40:52.002-05:00What Maths PhD are you talking about? Pure Maths c...What Maths PhD are you talking about? Pure Maths certainly are since they are rare: small number of candidates for a small number of grants. Applied Maths are not: lots of candidates compared to the number of grants. <br /><br />That is what I see in Europe. For example: it is quite easy to get a grant (or have it payed for by the financial institution (Bank etc...) where you work) in Bourbakihttps://www.blogger.com/profile/14739779159703428622noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-55737885139946167702020-02-20T08:49:28.238-05:002020-02-20T08:49:28.238-05:00Bourbaki,
You don't need money. Most math PhD...Bourbaki,<br /><br />You don't need money. Most math PhDs are supported by the school.storhttps://www.blogger.com/profile/08963157059325167335noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-19701809358058036852020-02-20T02:55:04.738-05:002020-02-20T02:55:04.738-05:00Dear Lance,
"How does someone get a PhD in w...Dear Lance,<br /><br />"How does someone get a PhD in whatever branch of knowledge without knowing which research approaches, methodologies and methods are pertinent to that subject and having been properly trained in the applicable research methods!!?"<br /><br />Things were done that way in the "old times". You learned how to do research doing research. That was the point ofBourbakihttps://www.blogger.com/profile/14739779159703428622noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-74349045176488073632020-02-19T18:15:07.308-05:002020-02-19T18:15:07.308-05:00How does someone get a PhD in whatever branch of k...How does someone get a PhD in whatever branch of knowledge without knowing which research approaches, methodologies and methods are pertinent to that subject and having been properly trained in the applicable research methods!!? It appears to me that the standard of awarding PhDs needs to be improved so that any PhD student shows mastery of the applicable research methodologies etc. Yes the Lancehttps://www.blogger.com/profile/16736955327507738874noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-1792842207043832142020-02-18T16:48:14.002-05:002020-02-18T16:48:14.002-05:00The worse thing with scientific studies is that th...The worse thing with scientific studies is that they can only control for so many variables. Lets say that we publish that sunlight causes melanoma. Everybody puts sunscreen on and 40 years later we find out that the chemicals from the sunscreen cause 10 other kinds of cancer and not getting enough sun causes 10 other kinds of cancer. Studies rarely can be good enough to make decisions about Michael John Sarnowskihttps://www.blogger.com/profile/00528454593064091302noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-5606923137293814832020-02-18T15:36:26.655-05:002020-02-18T15:36:26.655-05:00Thank you for your link from Gelman's article....Thank you for your link from Gelman's article. <br /><br />I found it useful and helps delineate some of the issues discussed above.Lockleyhttps://www.blogger.com/profile/08530123861664618224noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-42098258493649238152020-02-18T07:11:59.346-05:002020-02-18T07:11:59.346-05:00In machine learning one runs into the issue that t...In machine learning one runs into the issue that training data and model validation data ought to be distinct, but a work-around is k-fold cross-validation. https://towardsdatascience.com/cross-validation-in-machine-learning-72924a69872fArunhttps://www.blogger.com/profile/03451666670728177970noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-20112402170774984802020-02-17T18:16:36.322-05:002020-02-17T18:16:36.322-05:00That would require a new data set that is develope...That would require a new data set that is developed in the same way as the data set used in the initial study (otherwise an inability to replicate might be due to data issues). That might possible with medical and physics experiments, if the researcher can get additional funding to produce a new, independent data set for a rerun of the experiment.<br />. As with much of social science, I work t marvellhttps://www.blogger.com/profile/06867407646383818288noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-17633272765559851562020-02-17T16:40:14.936-05:002020-02-17T16:40:14.936-05:00That really depends on how expensive it is to repe...That really depends on how expensive it is to repeat the study. Peter Shorhttps://www.blogger.com/profile/13823970640202949073noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-10775848954575078072020-02-17T15:22:11.978-05:002020-02-17T15:22:11.978-05:00Andrew Gelman has also written extensively on this...Andrew Gelman has also written extensively on this topic on his <a href="https://statmodeling.stat.columbia.edu/" rel="nofollow">blog</a>. One additional "horseman" I think he would add is the binary thinking promoted by null-hypothesis statistical testing (NHST): results are either "significant" and the hypothesis can be considered verified, or "not significant" andKevin S. Van Hornhttps://www.blogger.com/profile/07591228017757297031noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-5864604132398856852020-02-17T15:12:42.413-05:002020-02-17T15:12:42.413-05:00Hi Sabine,
Recently, the presentations at Metasci...Hi Sabine,<br /><br />Recently, the presentations at Metascience 2019 have been made available.<br /><br />Glad you interviewed Professor Bishop. Her presentation is available here, entitled : "The role of cognitive biases in sustaining bad science"<br /><br />https://www.metascience2019.org/presentations/dorothy-bishop/<br /><br />CheersRKhttps://www.blogger.com/profile/05623076709623363318noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-32956179854940763722020-02-17T09:19:16.806-05:002020-02-17T09:19:16.806-05:00For a recent quantitative analysis of how often ex...For a recent quantitative analysis of how often experimental reports of the synthesis of a new material are repeated in the scientific literature, see this recent PNAS paper: https://www.pnas.org/content/117/2/877.short<br /><br />The process of writing this paper involved some hard-working students in my group analyzing thousands of individual papers.Prof. Shollhttps://www.blogger.com/profile/01478272124748674635noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-56634705920429857952020-02-16T16:44:24.955-05:002020-02-16T16:44:24.955-05:00Bourbaki,
I see a complementarity between traditio...Bourbaki,<br />I see a complementarity between traditional review articles and preprints as one possible way of proceeding. <br />Some notes about this are here, with a focus on arXiv but perhaps extensible to other areas: https://www.niso.org/niso-io/2020/01/standards-and-role-preprints-scholarly-communication<br />I'm hoping to do more writing on this topic.<br />Brian Simbolihttps://www.blogger.com/profile/02495506054549317802noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-56590222704480928822020-02-16T16:22:07.678-05:002020-02-16T16:22:07.678-05:00When I started my PhD in solvent extraction forty ...When I started my PhD in solvent extraction forty odd years ago, my supervisor got me to do an undergraduate experiment extracting copper. I could not get the accepted answer of 2.0 for the slope. Kept getting 1.8. Tried for three months … different spectrometers etc. Eventually gave up and got on with the thesis. A couple of years later, supervising the under graduate class, a student pointed romhttps://www.blogger.com/profile/11911660301033762535noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-78613079619792533202020-02-16T13:55:37.889-05:002020-02-16T13:55:37.889-05:00This replication crisis has been a disaster for so...This replication crisis has been a disaster for social psychology, and medical research, but a boon for the researchers. P-hacking, contrary to the claims made in the interview, is a conscious act and will only be eliminated by constant attention and brutal consequences for the offenders - not a highly probable outcome, at least not less than .05.Cal Billingshttps://www.blogger.com/profile/10125179892814373163noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-31560433517426266282020-02-16T10:23:59.381-05:002020-02-16T10:23:59.381-05:00Great episode. I appreciate the concern about wea...Great episode. I appreciate the concern about weaponization, but science to me is about shining light, so transparency with the public is important.<br /><br />I understand not using data to support a hypothesis not originally contemplated by the experiment, but I assume it is okay to look at data, acknowledge it did not support the original hypothesis, and formulate a new hypothesis from the MarkinDallashttps://www.blogger.com/profile/06200370581574641024noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-33324377916057309522020-02-16T08:46:01.486-05:002020-02-16T08:46:01.486-05:00I observed, as a cognitive behavioural therapist, ...I observed, as a cognitive behavioural therapist, that research papers have either a high P value, but the standard deviation is large, or a small standard deviation, but a low P value. This was a feature back when I started training back in the late 1990s.<br /><br />As for the reproducibility of some of the classic experiments, I observe that people's personalities are a confounding Ashleyhttps://www.blogger.com/profile/13666947574653683678noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-69816626979453182020-02-16T01:05:10.587-05:002020-02-16T01:05:10.587-05:00Bourbaki,
First let me say that I don't think...Bourbaki,<br /><br />First let me say that I don't think it's an issue that in some research areas it's easier to churn out papers than in others. I say this both from personal experience and from my experience with bibliometric analysis. The field-dependence of publication-pace is very easy to normalize to and it's something that people tend to get intuitively right (though of Sabine Hossenfelderhttps://www.blogger.com/profile/06151209308084588985noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-55584724875069305442020-02-16T00:49:21.015-05:002020-02-16T00:49:21.015-05:00t marvell,
I don't see the concern you raise....t marvell,<br /><br />I don't see the concern you raise. If a pre-registered study stumbles across something unexpected yet relevant, there is nothing preventing researchers from pursuing the matter in their next study. Sabine Hossenfelderhttps://www.blogger.com/profile/06151209308084588985noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-56430703244823815082020-02-15T21:25:55.533-05:002020-02-15T21:25:55.533-05:00As a social scientists, I run across these problem...As a social scientists, I run across these problems constantly, and I agree with everything except the importance of a pre-registered research plan and hypothesis. These in effect mean that the researchers have the conceit to play god, pretending that they know enough to form a plan and hypothesis, and it rules out serendipitious results if they fall outside the researchers' initial mindsets.t marvellhttps://www.blogger.com/profile/06867407646383818288noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-8554450626698551502020-02-15T20:06:56.654-05:002020-02-15T20:06:56.654-05:00Though my gut reaction is that fixing psychology t...Though my gut reaction is that fixing psychology this way will be about like making socialism work, I am happy that this problem is at least now being acknowledged and worked on rather than not. I had to smile about how professor Bishop suggested that certain standard text book cases would be removed. Hopefully so, but apparently that’s not yet displayed by the Stanford Prison Experiment. Philosopher Erichttps://www.blogger.com/profile/11126076811765843302noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-42698328119352778802020-02-15T13:55:54.447-05:002020-02-15T13:55:54.447-05:00Thank you Sabine for this interview,
Reproducibili...Thank you Sabine for this interview,<br />Reproducibility and its 4 horsemen is indeed a growing problem in all areas of science. <br />If I may share my mathematician perspective it is getting increasingly worse. Its bad, really bad. <br />Despite these problems of reproducibility, mathematical areas that have been exploding (in terms of publications numbers) are the ones deriving from Bourbakihttps://www.blogger.com/profile/14739779159703428622noreply@blogger.comtag:blogger.com,1999:blog-22973357.post-39689341888472679672020-02-15T13:11:35.438-05:002020-02-15T13:11:35.438-05:00Here is something from personal experience: about ...Here is something from personal experience: about a decade ago I decided to get involved in biochemistry. First project was to work with a researcher who had gotten great results from a wunderkind grad student. We were simply going to replicate his experiment before doing a grant proposal based on his results. His work resulted in a published paper, by the way. And we couldn't do it. We foundRick Lubbockhttps://www.blogger.com/profile/01070357751075383393noreply@blogger.com