Jack Bradshaw

Frank Batini’s article (The role of academics in influencing the perceived threat from climate change) highlights how data can be misused to achieve an end.

The ‘peer review’ process is intended to ensure that scientific findings are credible and justifiable. But with the frenzy to publish for academic standing and the proliferation of journals wanting to accommodate them, the peer review process often fails under the pressure. Editors often have difficulty finding unpaid reviewers who are willing to give the considerable time and effort to thoroughly review these papers. Some may even be intimidated by the number of authors – 38 authors of one of the papers below tops the list so far, no doubt all accruing more academic brownie points.

The consequences of sloppy science and inadequate peer review can have consequences well beyond an academic exercise, as demonstrated by the cascading misuse of data shown here.

In 2011, 92,000 ha of the northern jarrah forest in Western Australia was surveyed from the air and it was estimated that 1.5% (1,350 ha) of the forest had ‘collapsed’. Field surveys found that 74% of measured stems over 1cm diameter were dying or recently killed and there was a 26% mortality of trees on these sites after 6 months. These sites were mostly shallow soils around rocky outcrops (Matusick et al. 2013).

In a broader study of climate impacts, Ruthrof et al. (2018)  claimed that there was a 48.6% mortality of the forest trees and extrapolated it to a larger area of the northern jarrah forest from Perth to Jarrahdale (more than 300,000 ha), quoting the paper above as the sole source of the data. A casual Sunday drive through the forest will clearly show that this is an absurd statement.

In a paper by Hartmann et al. (2022), 16,000 ha of the northern jarrah forest (now extended to about 900,000 ha) was said to have experienced high levels of tree die-off with up to 74% of tree crowns dying or recently dead quoting the same Matusick et al. (2013) source. It was also claimed that there was an even greater dieback on rocky sites.

Bergstrom et al. (2021), while acknowledging that die-off is associated with drier sites in the forests of the south west of WA, nevertheless used the Matusick et al. (2013) data as evidence of the collapse of 1M ha of ‘Mediterranean forests and woodlands of south western Australia’. Bergstrom later said that “an extreme heatwave in 2010-2011 has ravaged land ecosystems and devastated forests and woodlands in Western Australia”. 

The end result of this series of peer reviewed papers is that the IPCC in its 6th assessment report has declared that the northern jarrah forest is now at “key risk of transition or collapse (high level of confidence)”. All of this based on an extrapolation and selective terminology of tree mortality on 1,350 ha of shallow soils around rocky outcrops that have a long history of drought death and recovery.

This declaration is now being used by some in the public arena as justification for the closure of the native forest timber industry.

It has become common to hide behind ‘peer review’ implying that it equates to the truth which cannot be questioned. It is clearly not. They also argue that disagreements should be aired in the journal in which the paper was published. However journals do not generally encourage criticism of papers because it reflects on their reviewing and damages their reputation. Furthermore the original authors are always left with the last word. The Brandolini principle is a further disincentive to engage so the problem persists.

Australian Rural & Regional News asked a few further questions of the author, Jack Bradshaw, who responded as follows:

ARR.News:

• Do you have personal experience of the peer review process that you could relate?
• If one disagrees with a paper, why not put together one with an alternative argument? Why could there not be/ are there not a range of papers with different views on the subject?
• With regard to journals’ peer review processes, could it be that that a person with no actual on the ground experience of the WA forest could be a peer reviewer, for instance?

Jack Bradshaw:

The journal process is that if you find fault with a paper (ie is not the same as disagreement) then you submit a comment to the editor of the journal. They then send the comment to the authors for their response. The problem is that journals do not like errors being pointed out because it reflects badly on their review process and their reputation. The other issue is that even if the comment is published the authors have the final say in their defence and there is no opportunity for real discussion.

One comment on methodology of a paper that Frank Batini and I submitted to a journal was not accepted because our data and comment was not peer reviewed – which of course neither the original paper is until it is reviewed by the journal reviewers. We were not sent the response from the authors by the journal but got it from the authors a year or so later.

I have subsequently sent a comment on methodology of a different paper to a different journal and I am still waiting for a decision on whether or not they will publish it. It is now more than 2 months and I expect that they will delay for long enough so that it becomes irrelevant. This is another issue – even if the original paper is shown to be in error it is still ‘on the books’ and continues to be quoted, while the comments rarely are.

The rules of one journal are, for instance:

Comments should “provide a significant and useful addition to the scientific literature and on-going scholarly discussion (i.e. not simply identifying error(s) in the original published article) and be of interest, not only to specialists in the field, but to … broader readership.”

“Once the Comment and Reply articles have been published, this represents the final formal published discussion in …, i.e. further Comment and/or Reply articles responding to the original Comment and Reply articles will not be considered for publication.”

I was a Panel Editor for a journal myself for several years. Some of the issues are that:

• Reviewers are not paid and they are pressured to do the job quickly to speed up publication. It is often hard to find reviewers willing to do it and as a consequence the quality of review can vary from very comprehensive to very sketchy.
• Reviewers are usually chosen from those who have published on a similar subject before. That does not make them expert – with the modern practice of naming multiple authors, it is very difficult to know who is expert and who is a hanger-on unless you know them. It also has a tendency to use academics to the exclusion of practitioners.
• In this day of extreme specialty reviewers can be too narrow in their knowledge and unable to see the issue in a broader context.
• If reviewers or even editors are in the same ‘camp’ as the authors then you are likely to get ideology, not science.

A reviewer’s job is to look at the methodology, see if other relevant literature has been considered, check if quotations from other resources are accurate, and see if the conclusions are logical. It is not necessary to agree with the author, just that the way it is done and the conclusions are defensible. For example you would not necessarily have to have on-ground experience of WA forest to do a good review – but you do need to understand enough about the subject and the context to do a critical analysis. The problem that some academics have is that they become too focused and do not see how their experience fits into the broader picture – whether they are local or otherwise.

There are many papers with different views on a subject which is something of an indictment of science itself. Science is meant be objective but the older you get the more you realise it is influenced by ideology, bias etc that is common to all human endeavour. Peer review ideally would minimise this but it is not working as it should. If you Google ‘failures of peer review’ you will find a plethora of papers that point out the problems, some of them quite alarming. There are suggested ways to improve but I think it is fair to say it is getting worse rather than better.

References

– Bergstrom DM, Wienecke BC, van den Hoff J, Hughes L, Lindenmayer DB, Ainsworth TD, Baker CM, Bland L, Bowman DMJS, Brooks ST et al. 2021. Combating ecosystem collapse from the tropics to the Antarctic. Global Change Biology. 27(9):1692-1703.
– Hartmann H, Bastos A, Das A, J. , Esquivel-Muelbert A, Hammond WM, Martínez-Vilalta J, McDowell NG, Powers JS, Pugh TAM, Ruthrof KX et al. 2022. Climate Change Risks to Global Forest Health: Emergence of Unexpected Events of Elevated Tree Mortality Worldwide. Annual Review of Plant Biology. 73(1):673-702.
– Matusick G, Ruthrof KX, Brouwers NC, Dell B, Hardy GSJ. 2013. Sudden forest canopy collapse corresponding with extreme drought and heat in a mediterranean-type eucalypt forest in southwestern Australia. European Journal of Forest Research. 132(3):497-510.
– Ruthrof K, Breshears D, Fontaine J, Froend R, Matusick G, Kala J, Miller B, Mitchell P, Wilson S, van Keulen M et al. 2018. Subcontinental heat wave triggers terrestrial and marine, multi-taxa responses. Scientific Reports. 8.

Jack Bradshaw is a retired forester, based in Western Australia.