User Tag List

Results 1 to 15 of 15

Thread: Should I trust a "peer reviewed" article?

  1. #1
    Join Date
    Jun 2015
    Location
    Belgium
    Posts
    2,277
    Post Thanks / Like
    Mentioned
    15 Post(s)
    Tagged
    0 Thread(s)

    Default Should I trust a "peer reviewed" article?

    I just read a paper published last month in the "Energies" journal of MDPI.

    MDPi are Swiss registered publishers of open access, peer reviewed, scientific papers in a variety of fields; they were founded and still run by a PRC national. The editor in chief of "Energies", one of the many journals that MDPI publish, is an Italian professor in Rome. MDPI have a very mixed reputation, including their peer review process, although on the whole have been rated as being academic by the Norwegian Scientific Index, that is, NSI have rated a minority of the journals / published papers as crap.

    The paper in question was published by a consortium of authors stated to be affiliated with 3 universities in Russia. The data presented in the paper is of interest to me, but I also found a few red flags in the paper.

    One illustration purportedly shows the hardware for one type of process but is clearly for another process.

    One producer that is a data source is mentioned at least twice for being engaged in a manufacturing process that is different from what they use. (I know this first hand.) Furthermore, the paper refers to the producer by their original name which was replaced at least 10 years ago.

    Another producer that is another data source is mentioned as using a completely different process.

    There are reasons to suspect that there are national interests at play, but I won't get into it because it would just be speculation.

    As mentioned, the data is of commercial interest to me, that is, I would be interested in including it in our external communication particularly because they tell a completely different story compared to the loosely substantiated claims made in a previous paper that was sponsored by large corporates in one specific sub-sector.

    Both my head and gut tell me to ignore the paper.

    Based on the generalities above, what would you think?
    Chikashi Miyamoto

  2. #2
    Join Date
    Oct 2008
    Location
    Northwest AZ
    Posts
    6,035
    Post Thanks / Like
    Mentioned
    19 Post(s)
    Tagged
    1 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    MDPI is the wikipedia of scientific journals. IMO, it is overly "open-sourced" with too many opportunities for people to edit. But that's just me.
    Retired Sailor, Marine dad, semi-professional cyclist, fly fisherman, and Indian School STEM teacher.
    Assistant Operating Officer at Farm Soap homemade soaps. www.farmsoap.com

  3. #3
    Join Date
    Jul 2008
    Location
    NY & MN
    Posts
    5,433
    Post Thanks / Like
    Mentioned
    11 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    Maybe I'm a fossil, but I'm skeptical of any open access journal that doesn't appear in print. I've experienced some less than rigorous review at some of those outfits.

    Someone will likely object to my categorical skepticism by noting that some of the most important research in the world has appeared in PLOS One. Noted. I still default to skepticism.

  4. #4
    Join Date
    Sep 2010
    Posts
    4,820
    Post Thanks / Like
    Mentioned
    16 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    it is the lowest quality of 'peer' review so treat it as pre-prints.

  5. #5
    Join Date
    Jun 2015
    Location
    Belgium
    Posts
    2,277
    Post Thanks / Like
    Mentioned
    15 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    Yeah, some healthy dose of scepticism is likely to be in order...
    Chikashi Miyamoto

  6. #6
    Join Date
    Mar 2009
    Location
    Meriden CT
    Posts
    1,663
    Post Thanks / Like
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    You have enough reservations about what you're reading to be skeptical. When your "Bat Senses" start tingling, listen to them!

    Can you pass it by some of your industry colleagues and see what they say? How about industry competitors who you are friendly with?

  7. #7
    Join Date
    Jun 2015
    Location
    Belgium
    Posts
    2,277
    Post Thanks / Like
    Mentioned
    15 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    Quote Originally Posted by Peter Polack View Post
    You have enough reservations about what you're reading to be skeptical. When your "Bat Senses" start tingling, listen to them!

    Can you pass it by some of your industry colleagues and see what they say? How about industry competitors who you are friendly with?
    Sensible suggestions. However, the reality is that the data is not publicly available as most bits are held by private companies and to compile an analysis would require collecting data from further afield, viz., their energy sources, etc. There have been blatantly biased papers published in the past, and the challenge has been not only to collect the data but also to have transparency and consistency in the methodology as well as independent verification of both data and analysis. In other words, it's a bit complicated as well as complex.

    The errors I spotted would be obvious to anyone else in the trade, and they would make most people question whether the authors know what they're talking about. At least one trade journalist has referred to it in an article without any reservations, so I sent an email to the journalist asking about it.
    Chikashi Miyamoto

  8. #8
    Join Date
    Jun 2008
    Location
    Seattle, WA
    Posts
    6,894
    Post Thanks / Like
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    If using it the way you propose would imply some level of endorsement, that would make the decision for me. If you think the idea in the article has merit, there’s probably a way to bring it up for discussion without appearing to endorse the article.

  9. #9
    Join Date
    Jun 2015
    Location
    Belgium
    Posts
    2,277
    Post Thanks / Like
    Mentioned
    15 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    Quote Originally Posted by 72gmc View Post
    If using it the way you propose would imply some level of endorsement, that would make the decision for me. If you think the idea in the article has merit, there’s probably a way to bring it up for discussion without appearing to endorse the article.
    Not trying to endorse the paper, quite the opposite: I would be interested in using the data presented in the paper to back up a statement I wouldn't mind making. My current view is that unfortunately, it's not robust enough.
    Chikashi Miyamoto

  10. #10
    Join Date
    Mar 2019
    Posts
    666
    Post Thanks / Like
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    Anyone ever read Didier Raoult’s original hydroxychloroquine article?

    In an nutshell, the treatment group had patients “lost to follow up”, whereas the control group did not. The reason they were lost to FU was because they either died or were transferred to the ICU (? he couldn’t follow them in the ICU). No one was “lost to follow up” in the control group, and none died or were sent to the ICU. And look where that garbage led us.

    Always read the methods and details.

  11. #11
    Join Date
    Oct 2017
    Posts
    84
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    Quote Originally Posted by caleb View Post
    Maybe I'm a fossil, but I'm skeptical of any open access journal that doesn't appear in print. I've experienced some less than rigorous review at some of those outfits.

    Someone will likely object to my categorical skepticism by noting that some of the most important research in the world has appeared in PLOS One. Noted. I still default to skepticism.
    In a different field and all but hell, even The Lancet published Wakefield et al (1988) which took over 10 years to retract.

  12. #12
    Join Date
    Nov 2010
    Location
    Concord, NH
    Posts
    2,417
    Post Thanks / Like
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    Undoubtedly you've heard of the current replication crisis in psychology research, peer-reviewed status notwithstanding. I can't help but think it's common across other fields of study as well, it's just that nobody's really looked (psychology and other social science studies tend to be inexpensive and easy to replicate, as opposed to, say, biology or physics or engineering). So, I guess it comes down to what "trust" means...

  13. #13
    Join Date
    Oct 2017
    Posts
    84
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    I think much of this can be attributed to the 'publish or perish' mentality and the requirement to attract grant funding by the universities/colleges for an academic to even hope to retain their job. If you don't have even the slightest hope of getting an article into Lancet, or NEJM or Science (or the corresponding premier journal in your field) then what is left? Depending on where your research falls you may be forced to submit to these privately operated quasi-peer journal aggregators, even if they are 'pay-to-publish'. It still makes a line item on someone's CV. And that may be all it takes to satisfy the 7 Senior Vice President's of Academic Research and the other 14 Assistant Vice Presidents a large university might have.

    Don't even get me started on the transition from scholar led (PhD) to the professional managerial (MBA) class that seems to have overtaken university administration.

  14. #14
    Join Date
    Nov 2010
    Location
    Concord, NH
    Posts
    2,417
    Post Thanks / Like
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    Quote Originally Posted by Bingissimo View Post
    ... Don't even get me started on the transition from scholar led (PhD) to the professional managerial (MBA) class that seems to have overtaken university administration.
    Oh, please do start...

    Agreed that the publish or perish mentality drives the publication of a lot of drek research. I see this in my field, in journals that are ostensibly "peer-reviewed" but a deeper dive reveals incomplete vetting of incoming articles at best, or, at worst, research that's really not "peer-reviewed" in the traditional sense at all.
    If you're on faculty and can't get published in an APA journal, well, PLOS Psychology (or something even more questionable), here we come. It's much better than not being published at all.

  15. #15
    Join Date
    Oct 2012
    Posts
    10,977
    Post Thanks / Like
    Mentioned
    12 Post(s)
    Tagged
    0 Thread(s)

    Default Re: Should I trust a "peer reviewed" article?

    Quote Originally Posted by Chik View Post
    Sensible suggestions. However, the reality is that the data is not publicly available as most bits are held by private companies and to compile an analysis would require collecting data from further afield, viz., their energy sources, etc. There have been blatantly biased papers published in the past, and the challenge has been not only to collect the data but also to have transparency and consistency in the methodology as well as independent verification of both data and analysis. In other words, it's a bit complicated as well as complex.

    The errors I spotted would be obvious to anyone else in the trade, and they would make most people question whether the authors know what they're talking about. At least one trade journalist has referred to it in an article without any reservations, so I sent an email to the journalist asking about it.
    If the real data is hidden, that mean it can't be reviewed thorously imho. So it means what they call a "peer review process" is no more than hitting the like button in a social media.
    --
    T h o m a s

Similar Threads

  1. --"Trust.." one of the ..
    By ron l edmiston in forum The OT
    Replies: 0
    Last Post: 08-17-2014, 08:24 AM

Bookmarks

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •