An alternative to the hand searching gold standard: validating methodological search filters using relative recall
Date
2006
Authors
Sampson, Margaret
Zhang, Li
Morrison, Andra
Barrowman, Nicholas J
Clifford, Tammy J
Platt, Robert
Klassen, Terry
Moher, David
Journal Title
Journal ISSN
Volume Title
Publisher
BioMed Central
ORCID
Type
Article
Refereed Paper
Refereed Paper
Degree Level
Abstract
Background
Search filters or hedges play an important role in evidence-based medicine but their development depends on the availability of a "gold standard" - a reference standard against which to establish the performance of the filter. We demonstrate the feasibility of using relative recall of included studies from multiple systematic reviews to validate methodological search filters as an alternative to validation against a gold standard formed through hand searching.
Methods
We identified 105 Cochrane reviews that used the Highly Sensitive Search Strategy (HSSS), included randomized or quasi-randomized controlled trials, and reported their included studies. We measured the ability of two published and one novel variant of the HSSS to retrieve the MEDLINE-index studies included in these reviews.
Results
The systematic reviews were comprehensive in their searches. 72% of included primary studies were indexed in MEDLINE. Relative recall of the three strategies ranged from .98 to .91 across all reviews and more comprehensive strategies showed higher recall.
Conclusion
An approach using relative recall instead of a hand searching gold standard proved feasible and produced recall figures that were congruent with previously published figures for the HSSS. This technique would permit validation of a methodological filter using a collection of approximately 100 studies of the chosen design drawn from the included studies of multiple systematic reviews that used comprehensive search strategies.
Description
Margaret Sampson: msampson@uottawa.ca; Li Zhang: li.zhang@usask.ca; Andra Morrison: andram@cadth.ca; Nicholas J Barrowman: nbarrowman@cheo.on.ca; Tammy J Clifford: tammyc@cadth.ca; Robert W Platt: rplatt@po-box.mcgill.ca; Terry P Klassen: terry.klassen@ualberta.ca; David Moher: dmoher@uottawa.ca
Keywords
Citation
BMC Medical Research Methodology 6:33 (18 July 2006)
Degree
Department
Program
Advisor
Committee
Citation
BMC Medical Research Methodology 6:33 (18 July 2006)
Part Of
item.page.relation.ispartofseries
DOI
10.1186/1471-2288-6-33