Should I trust a "peer reviewed" article?
I just read a paper published last month in the "Energies" journal of MDPI.
MDPi are Swiss registered publishers of open access, peer reviewed, scientific papers in a variety of fields; they were founded and still run by a PRC national. The editor in chief of "Energies", one of the many journals that MDPI publish, is an Italian professor in Rome. MDPI have a very mixed reputation, including their peer review process, although on the whole have been rated as being academic by the Norwegian Scientific Index, that is, NSI have rated a minority of the journals / published papers as crap.
The paper in question was published by a consortium of authors stated to be affiliated with 3 universities in Russia. The data presented in the paper is of interest to me, but I also found a few red flags in the paper.
One illustration purportedly shows the hardware for one type of process but is clearly for another process.
One producer that is a data source is mentioned at least twice for being engaged in a manufacturing process that is different from what they use. (I know this first hand.) Furthermore, the paper refers to the producer by their original name which was replaced at least 10 years ago.
Another producer that is another data source is mentioned as using a completely different process.
There are reasons to suspect that there are national interests at play, but I won't get into it because it would just be speculation.
As mentioned, the data is of commercial interest to me, that is, I would be interested in including it in our external communication particularly because they tell a completely different story compared to the loosely substantiated claims made in a previous paper that was sponsored by large corporates in one specific sub-sector.
Both my head and gut tell me to ignore the paper.
Based on the generalities above, what would you think?
Re: Should I trust a "peer reviewed" article?
MDPI is the wikipedia of scientific journals. IMO, it is overly "open-sourced" with too many opportunities for people to edit. But that's just me.
Re: Should I trust a "peer reviewed" article?
Maybe I'm a fossil, but I'm skeptical of any open access journal that doesn't appear in print. I've experienced some less than rigorous review at some of those outfits.
Someone will likely object to my categorical skepticism by noting that some of the most important research in the world has appeared in PLOS One. Noted. I still default to skepticism.
Re: Should I trust a "peer reviewed" article?
it is the lowest quality of 'peer' review so treat it as pre-prints.
Re: Should I trust a "peer reviewed" article?
Yeah, some healthy dose of scepticism is likely to be in order...
Re: Should I trust a "peer reviewed" article?
You have enough reservations about what you're reading to be skeptical. When your "Bat Senses" start tingling, listen to them!
Can you pass it by some of your industry colleagues and see what they say? How about industry competitors who you are friendly with?
Re: Should I trust a "peer reviewed" article?
Quote:
Originally Posted by
Peter Polack
You have enough reservations about what you're reading to be skeptical. When your "Bat Senses" start tingling, listen to them!
Can you pass it by some of your industry colleagues and see what they say? How about industry competitors who you are friendly with?
Sensible suggestions. However, the reality is that the data is not publicly available as most bits are held by private companies and to compile an analysis would require collecting data from further afield, viz., their energy sources, etc. There have been blatantly biased papers published in the past, and the challenge has been not only to collect the data but also to have transparency and consistency in the methodology as well as independent verification of both data and analysis. In other words, it's a bit complicated as well as complex.
The errors I spotted would be obvious to anyone else in the trade, and they would make most people question whether the authors know what they're talking about. At least one trade journalist has referred to it in an article without any reservations, so I sent an email to the journalist asking about it.
Re: Should I trust a "peer reviewed" article?
If using it the way you propose would imply some level of endorsement, that would make the decision for me. If you think the idea in the article has merit, there’s probably a way to bring it up for discussion without appearing to endorse the article.
Re: Should I trust a "peer reviewed" article?
Quote:
Originally Posted by
72gmc
If using it the way you propose would imply some level of endorsement, that would make the decision for me. If you think the idea in the article has merit, there’s probably a way to bring it up for discussion without appearing to endorse the article.
Not trying to endorse the paper, quite the opposite: I would be interested in using the data presented in the paper to back up a statement I wouldn't mind making. My current view is that unfortunately, it's not robust enough.
Re: Should I trust a "peer reviewed" article?
Anyone ever read Didier Raoult’s original hydroxychloroquine article?
In an nutshell, the treatment group had patients “lost to follow up”, whereas the control group did not. The reason they were lost to FU was because they either died or were transferred to the ICU (? he couldn’t follow them in the ICU). No one was “lost to follow up” in the control group, and none died or were sent to the ICU. And look where that garbage led us.
Always read the methods and details.
Re: Should I trust a "peer reviewed" article?
Quote:
Originally Posted by
caleb
Maybe I'm a fossil, but I'm skeptical of any open access journal that doesn't appear in print. I've experienced some less than rigorous review at some of those outfits.
Someone will likely object to my categorical skepticism by noting that some of the most important research in the world has appeared in PLOS One. Noted. I still default to skepticism.
In a different field and all but hell, even The Lancet published Wakefield et al (1988) which took over 10 years to retract.
Re: Should I trust a "peer reviewed" article?
Undoubtedly you've heard of the current replication crisis in psychology research, peer-reviewed status notwithstanding. I can't help but think it's common across other fields of study as well, it's just that nobody's really looked (psychology and other social science studies tend to be inexpensive and easy to replicate, as opposed to, say, biology or physics or engineering). So, I guess it comes down to what "trust" means...
Re: Should I trust a "peer reviewed" article?
I think much of this can be attributed to the 'publish or perish' mentality and the requirement to attract grant funding by the universities/colleges for an academic to even hope to retain their job. If you don't have even the slightest hope of getting an article into Lancet, or NEJM or Science (or the corresponding premier journal in your field) then what is left? Depending on where your research falls you may be forced to submit to these privately operated quasi-peer journal aggregators, even if they are 'pay-to-publish'. It still makes a line item on someone's CV. And that may be all it takes to satisfy the 7 Senior Vice President's of Academic Research and the other 14 Assistant Vice Presidents a large university might have.
Don't even get me started on the transition from scholar led (PhD) to the professional managerial (MBA) class that seems to have overtaken university administration.
Re: Should I trust a "peer reviewed" article?
Quote:
Originally Posted by
Bingissimo
... Don't even get me started on the transition from scholar led (PhD) to the professional managerial (MBA) class that seems to have overtaken university administration.
Oh, please do start...
Agreed that the publish or perish mentality drives the publication of a lot of drek research. I see this in my field, in journals that are ostensibly "peer-reviewed" but a deeper dive reveals incomplete vetting of incoming articles at best, or, at worst, research that's really not "peer-reviewed" in the traditional sense at all.
If you're on faculty and can't get published in an APA journal, well, PLOS Psychology (or something even more questionable), here we come. It's much better than not being published at all.
Re: Should I trust a "peer reviewed" article?
Quote:
Originally Posted by
Chik
Sensible suggestions. However, the reality is that the data is not publicly available as most bits are held by private companies and to compile an analysis would require collecting data from further afield, viz., their energy sources, etc. There have been blatantly biased papers published in the past, and the challenge has been not only to collect the data but also to have transparency and consistency in the methodology as well as independent verification of both data and analysis. In other words, it's a bit complicated as well as complex.
The errors I spotted would be obvious to anyone else in the trade, and they would make most people question whether the authors know what they're talking about. At least one trade journalist has referred to it in an article without any reservations, so I sent an email to the journalist asking about it.
If the real data is hidden, that mean it can't be reviewed thorously imho. So it means what they call a "peer review process" is no more than hitting the like button in a social media.
