Details
| Original language | English |
|---|---|
| Pages (from-to) | 75-82 |
| Number of pages | 8 |
| Journal | Environmental Science and Policy |
| Volume | 64 |
| Publication status | Published - 1 Oct 2016 |
| Externally published | Yes |
Abstract
Given the proliferation of primary research articles, the importance of reliable environmental evidence reviews for informing policy and management decisions is increasing. Although conducting reviews is an efficient method of synthesising the fragmented primary evidence base, reviews that are of poor methodological reliability have the potential to misinform by not accurately reflecting the available evidence base. To assess the current value of evidence reviews for decision-making we appraised a systematic sample of articles published in early 2015 (N = 92) using the Collaboration for Environmental Evidence Synthesis Assessment Tool (CEESAT). CEESAT assesses the methodology of policy-relevant evidence reviews according to elements important for objectivity, transparency and comprehensiveness. Overall, reviews performed poorly with a median score of 2.5/39 and a modal score of zero (range 0–30, mean 5.8), and low scores were ubiquitous across subject areas. In general, reviews that applied meta-analytical techniques achieved higher scores than narrative syntheses (median 18.3 and 2.0 respectively), as a result of the latter consistently failing to adequately report methodology or how conclusions were drawn. However, some narrative syntheses achieved high scores, illustrating that the reliability of reviews should be assessed on a case-by-case basis. Given the potential importance of reviews for informing management and policy, as well as research, it is vital that overall methodological reliability is improved. Although the increasing number of systematic reviews and meta-analyses highlight that some progress is being made, our findings suggest little or no improvement in the last decade. To motivate progress, we recommend that an annual assessment of the methodological reliability of evidence reviews be conducted. To better serve the environmental policy and management communities we identify a requirement for independent critical appraisal of review methodology thus enabling decision-makers to select reviews that are most likely to accurately reflect the evidence base.
Keywords
- CEESAT, Decision-making, Evidence syntheses, Evidence-base, Review evaluation, Review methodology
ASJC Scopus subject areas
- Social Sciences(all)
- Geography, Planning and Development
- Environmental Science(all)
- Management, Monitoring, Policy and Law
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
In: Environmental Science and Policy, Vol. 64, 01.10.2016, p. 75-82.
Research output: Contribution to journal › Article › Research › peer review
}
TY - JOUR
T1 - The reliability of evidence review methodology in environmental science and conservation
AU - O'Leary, Bethan C.
AU - Kvist, Kristian
AU - Bayliss, Helen R.
AU - Derroire, Géraldine
AU - Healey, John R.
AU - Hughes, Kathryn
AU - Kleinschroth, Fritz
AU - Sciberras, Marija
AU - Woodcock, Paul
AU - Pullin, Andrew S.
N1 - Publisher Copyright: © 2016 Elsevier Ltd
PY - 2016/10/1
Y1 - 2016/10/1
N2 - Given the proliferation of primary research articles, the importance of reliable environmental evidence reviews for informing policy and management decisions is increasing. Although conducting reviews is an efficient method of synthesising the fragmented primary evidence base, reviews that are of poor methodological reliability have the potential to misinform by not accurately reflecting the available evidence base. To assess the current value of evidence reviews for decision-making we appraised a systematic sample of articles published in early 2015 (N = 92) using the Collaboration for Environmental Evidence Synthesis Assessment Tool (CEESAT). CEESAT assesses the methodology of policy-relevant evidence reviews according to elements important for objectivity, transparency and comprehensiveness. Overall, reviews performed poorly with a median score of 2.5/39 and a modal score of zero (range 0–30, mean 5.8), and low scores were ubiquitous across subject areas. In general, reviews that applied meta-analytical techniques achieved higher scores than narrative syntheses (median 18.3 and 2.0 respectively), as a result of the latter consistently failing to adequately report methodology or how conclusions were drawn. However, some narrative syntheses achieved high scores, illustrating that the reliability of reviews should be assessed on a case-by-case basis. Given the potential importance of reviews for informing management and policy, as well as research, it is vital that overall methodological reliability is improved. Although the increasing number of systematic reviews and meta-analyses highlight that some progress is being made, our findings suggest little or no improvement in the last decade. To motivate progress, we recommend that an annual assessment of the methodological reliability of evidence reviews be conducted. To better serve the environmental policy and management communities we identify a requirement for independent critical appraisal of review methodology thus enabling decision-makers to select reviews that are most likely to accurately reflect the evidence base.
AB - Given the proliferation of primary research articles, the importance of reliable environmental evidence reviews for informing policy and management decisions is increasing. Although conducting reviews is an efficient method of synthesising the fragmented primary evidence base, reviews that are of poor methodological reliability have the potential to misinform by not accurately reflecting the available evidence base. To assess the current value of evidence reviews for decision-making we appraised a systematic sample of articles published in early 2015 (N = 92) using the Collaboration for Environmental Evidence Synthesis Assessment Tool (CEESAT). CEESAT assesses the methodology of policy-relevant evidence reviews according to elements important for objectivity, transparency and comprehensiveness. Overall, reviews performed poorly with a median score of 2.5/39 and a modal score of zero (range 0–30, mean 5.8), and low scores were ubiquitous across subject areas. In general, reviews that applied meta-analytical techniques achieved higher scores than narrative syntheses (median 18.3 and 2.0 respectively), as a result of the latter consistently failing to adequately report methodology or how conclusions were drawn. However, some narrative syntheses achieved high scores, illustrating that the reliability of reviews should be assessed on a case-by-case basis. Given the potential importance of reviews for informing management and policy, as well as research, it is vital that overall methodological reliability is improved. Although the increasing number of systematic reviews and meta-analyses highlight that some progress is being made, our findings suggest little or no improvement in the last decade. To motivate progress, we recommend that an annual assessment of the methodological reliability of evidence reviews be conducted. To better serve the environmental policy and management communities we identify a requirement for independent critical appraisal of review methodology thus enabling decision-makers to select reviews that are most likely to accurately reflect the evidence base.
KW - CEESAT
KW - Decision-making
KW - Evidence syntheses
KW - Evidence-base
KW - Review evaluation
KW - Review methodology
UR - http://www.scopus.com/inward/record.url?scp=84976627451&partnerID=8YFLogxK
U2 - 10.1016/j.envsci.2016.06.012
DO - 10.1016/j.envsci.2016.06.012
M3 - Article
AN - SCOPUS:84976627451
VL - 64
SP - 75
EP - 82
JO - Environmental Science and Policy
JF - Environmental Science and Policy
SN - 1462-9011
ER -