> Most will refuse to publish replications, negative studies, or anything they deem unimportant, even if the study was conducted correctly.
I think this was really caused by the rise of bureaucracy in academia. Bureaucrats favorite thing is a measurement, especially when they don't understand its meaning. There's always been a drive for novelty in academia, it's just at the very core of the game. But we placed far too much focus on this, despite the foundation of science being replication. We made a trade, foundation for (the illusion of) progress. It's like trying to build a skyscraper higher and higher without concern for the ground it stands on. Doesn't take a genius to tell you that building is going to come crashing down. But proponents say "it hasn't yet! If it was going to fall it would have already" while critics are actually saying "we can't tell you when it'll fall, but there's some concerning cracks and we're worried it'll collapse and we won't even be able to tell we're in a pile of rubble."
I don't know what the solution is, but I do know that our fear of people wasting money and creating fraudulent studies has only resulted in wasting money and fraudulent studies. We've removed the verification system while creating strong incentives to cheat (punish or perish, right?).
I think one thing we do need to recognize is that in the grand scheme of things, academia isn't very expensive. A small percentage of a large number is still a large number. Even if half of academics were frauds it would be a small percentage of waste, and pale in comparison to more common waste, fraud, and abuse of government funds.
From what I can tell, the US spent $60bn for University R&D in 2023[0] (less than 1% of US Federal expenditures). But in that same time there was $400bn in waste and fraud through Covid relief funds [1]. With $280bn being straight up fraud. That alone is more than 4x of all academic research funding!!!
I'm unconvinced most in academia are motivated by money or prestige, as it's a terrible way to achieve those things. But I am convinced people are likely to commit fraud when their livelihoods are at stake or when they can believe that a small lie now will allow them to continue doing their work. So as I see it, the publish or perish paradigm only promotes the former. The lack of replication only allows, and even normalizes, the latter. The stress for novelty only makes academics try to write more like business people, trying to sell their product in some perverse rat race.
So I think we have to be a bit honest here. Even if we were to naively make this space essentially unregulated it couldn't be the pinnacle of waste, fraud, and abuse that many claim it is. But I doubt even letting scientists be entirely free from publication requirements that you'd find much waste, fraud, and abuse. Science has a naturally regulating structure. It was literally created to be that way! We got to where we are in through this self regulating system because scientists love to argue about who is right and the process of science is meant to do exactly that. Was there waste and fraud in the past? Yes. I don't think it's entirely avoidable, it'll never be $0 of waste money. But the system was undoubtably successful. And those that took advantage of the system were better at fooling the public than they were their fellow scientists. Which is something I think we've still failed to catch onto
You either have something documented and quantified and measured and objective criteria tickboxes and deal with this style of failure mode, or you rely on subjective judgment and assessment and accept the failure mode of bias, nepotism, old boy's clubs etc. Of course the ideal case is to rely on the unbureaucratic informal wise and impartial judgment of some hypothetical perfect humans you can fully trust and rely on, and they always decide fully on merits etc. without having to follow any rigid criteria and checkboxes and numbers on hiring and promotion etc. But people are not perfect and society largely decided to go the bureaucratic way to ensure equal opportunities and to reduce bias through this kind of transparency.
> You either have something documented and quantified and measured and objective criteria tickboxes and deal with this style of failure mode, or you rely on subjective judgment and assessment and accept the failure mode of bias, nepotism, old boy's clubs etc
My argument is that our current pursuit of the former only reinforces the existence of the latter.
You have a fundamental flaw in your argument, one that illustrates a common, yet fundamental, misunderstanding of science. There is no "objective" thing to measure, there are only proxies. I actually recently stumbled on a short by Adam Savage that I think captures this[0], although I think he's a bit wrong too. Regardless of precision we are always using a proxy. A tape measure does not define a meter, it only serves as a reference to compare with. A reference where not only the human makes error when reading, but that the reference itself has error[1]. So there are no direct measurements, there are only measurements by proxy.
You may have heard someone say "science doesn't prove things, it disproves them", and that's in part a consequence to this. Our measurements are meaningless without an understanding of their uncertainty (both quantifiable and unquantifiable!) as well as the assumptions they are made under.
I'm not trying to be pedantic here, I think this precision in understanding matters to the conversation. My argument is that by discounting those errors that they accumulate. We've had a pretty good run. This current system has only really started to be practiced in the 60s and 70's. So 50 years is a lot of time for error to accumulate. 50 years is a lot of time for small, seemingly insignificant, and easy to dismiss errors to accumulate into large, intangible, and complex problems.
There's something that I guess is more subtle in my argument: science is self-correcting. I don't mean "science" as the category of pursuits that seek truths about the world around us, but I mean "science" as a systematic approach to obtaining knowledge. A key reason this self-correction happens is due to replication. But in reality that is a consequence of how we pin down truth itself. We seek causal structures. More specifically, we seek counterfactual models. Assuming honest practitioners, failures of reproduction happen for primarily for one of two reasons: 1) ambiguity of communication between the original experimenters and those replicating or 2) a variation in conditions. 2) is actually quite common and tells us something new about that causal structure. In practice it is extremely difficult, if not impossible, to exactly replicate the conditions of the original experiment, so even with successful replication we gain information about the robustness of the results.
But why am I talking about all this? Because without the explicit acknowledgement of these limitations we seem to easily forget them. We are often treating substantially more subjective measures (such as impact or novelty) as far more objective than we would treat even physical measurements. It should be absolutely no surprise that things like impact are at best extremely difficult to measure. Even with a time machine we may not accurately measure the impact of a work for decades, or more. Ironically, a major reason for a work's impact to be found only after decades (or centuries) is the belief that at its time it had no impact, and was a dead end. You'd be amazed at how common this actually is. It's where jokes similar to how everything is named after the second person to discover something, the first being Euler[2]. But science is self-correcting. Even if a discovery of Euler's was lost, it is only a matter of time before someone (independently) rediscovers it.
I'm talking about this because there is no perfect system. Because a measurement without the acknowledge of its uncertainty is far less accurate than a measurement with. I'm talking about this because we will always have errors and the existence of them is not a reason to dismiss things. Instead we have to compare and contrast both the benefits and limits of competing ideas. We are only doing ourselves a disservice by pretending the limits don't exist. And if we mindlessly pursue objective measurements we'll only end up finding we've metric hacked our way into reading tea leaves. As we advance in any subject the minutia always ends up being the critical element (see [0]) and so the problem is it doesn't matter if we're 90% "objective" and 10% reading the tea leaves. Not when the decisions are made differentiating the 10%. In reality we're not even good at measuring that 90% when it comes to determining how productive academics are[3-5]
[5] See the two links in this comment as further evidence. They are about relatively recent Nobel works that faced frequent rejections https://news.ycombinator.com/item?id=47340733
Someone has to pay for all this. That someone is most often not a scientist themselves. They don't have a that vague intuitive research taste that scientists have. Beyond fairly trivial levels of technical correctness, the value of research lies in its narrative implications, its interestingness, its surprise factor etc. These are not objective and are often more about aesthetics and taste than popularly understood. Why does the research matter? To whom does it matter? Are those people important? Do they control resources?
There's a cost in either direction. You can't ignore the the costs of reading the tea leaves while acknowledging the costs of unnecessary work. Both have costs.
>> Instead we have to compare and contrast both the benefits and limits of competing ideas. We are only doing ourselves a disservice by pretending the limits don't exist.
> But in that same time there was $400bn in waste and fraud through Covid relief funds [1].
The cost of academic fraud should also include the indirect costs of bad decision making.
The Covid relief funds were only needed because politicians implemented extremely aggressive policies based on unproven epidemiological models built on fraudulent practices. I investigated all this extensively at the time and it was really sad/shocking how non-existent intellectual standards are in the field of epidemiology. The models were trash RNGs that couldn't have been validated even if they'd tried, which they never had because the field doesn't consider validation to be necessary to get a paper published. So the models made wildly wrong predictions based on untested, buggy, non-replicable models, which then led to lockdowns, which led to economic catastrophe, which led to the relief programme. All of the fraud in that programme - really the entire cost of it - should be laid at the feet of academic fraud.
I don't know what the solution is, but I do know that our fear of people wasting money and creating fraudulent studies has only resulted in wasting money and fraudulent studies. We've removed the verification system while creating strong incentives to cheat (punish or perish, right?).
I think one thing we do need to recognize is that in the grand scheme of things, academia isn't very expensive. A small percentage of a large number is still a large number. Even if half of academics were frauds it would be a small percentage of waste, and pale in comparison to more common waste, fraud, and abuse of government funds.
From what I can tell, the US spent $60bn for University R&D in 2023[0] (less than 1% of US Federal expenditures). But in that same time there was $400bn in waste and fraud through Covid relief funds [1]. With $280bn being straight up fraud. That alone is more than 4x of all academic research funding!!!
I'm unconvinced most in academia are motivated by money or prestige, as it's a terrible way to achieve those things. But I am convinced people are likely to commit fraud when their livelihoods are at stake or when they can believe that a small lie now will allow them to continue doing their work. So as I see it, the publish or perish paradigm only promotes the former. The lack of replication only allows, and even normalizes, the latter. The stress for novelty only makes academics try to write more like business people, trying to sell their product in some perverse rat race.
So I think we have to be a bit honest here. Even if we were to naively make this space essentially unregulated it couldn't be the pinnacle of waste, fraud, and abuse that many claim it is. But I doubt even letting scientists be entirely free from publication requirements that you'd find much waste, fraud, and abuse. Science has a naturally regulating structure. It was literally created to be that way! We got to where we are in through this self regulating system because scientists love to argue about who is right and the process of science is meant to do exactly that. Was there waste and fraud in the past? Yes. I don't think it's entirely avoidable, it'll never be $0 of waste money. But the system was undoubtably successful. And those that took advantage of the system were better at fooling the public than they were their fellow scientists. Which is something I think we've still failed to catch onto
[0] https://usafacts.org/articles/what-do-universities-do-with-t...
[1] https://apnews.com/article/pandemic-fraud-waste-billions-sma...