On 2016 Mar 03, Liam McKeever commented:

Ms. Allard,

Thank you for your thoughtful comments. Using [tiab] would certainly increase the specificity of the search without much loss to sensitivity. The technique of this tutorial assumes that the article indexed for MEDLINE has been properly indexed, which will not always be a correct assumption. While I consider this assumption an acceptable and explainable loss in sensitivity in exchange for what is usually a considerable increase in specificity, the technique you are describing may actually be a perfect compromise.

Sincerely,

Liam McKeever,MS,RDN

On 2016 Feb 29, Rhonda J Allard commented:

While I see the point you make about being sure your search strategies take into account both Medline and non-Medline articles, I don't believe you need to create two searches to accomplish this goal. The following search strategy (in my opinion), is as effective:

fasting AND (alternate day* OR alternating day*) -- it retrieves 120 articles

Notes: 1) The term fasting will map to Fasting [Mesh] and also search "fasting" in all fields. This will retrieve both Medline and non-Medline articles. 2) Anytime you use double quotes for phrase searching or the asterisk for truncation you are turning off the "Automatic Term Mapping" and searching all of PubMed.

If the searcher is concerned with the amount of irrelevant articles they might retrieve when using keywords, another trick is to limit keywords to the title or abstract using [tiab], see below:

military retrieves 127,746 results (it maps to Military Personnel[Mesh] and searches for "military" in All Fields) whereas (Military Personnel[MH] OR military[tiab]) retrieves 56,624 results. This second search limits the search results to where the topic is the military. The term military can appear in the Author Affiliation field or as part of another MeSH term.

On 2016 Feb 25, Liam McKeever commented:

Mr. Bramer,

 Your point here is well taken but really represents a calculated shortcoming of that particular search, not the technique. The search in our paper was only designed to demonstrate the different aspects of the techniques in the paper. We kept it short on purpose for the publication. Obviously, in performing a review, a thorough analysis of any and all possible relevant MeSH terms must be considered. Generally, in practice, this technique becomes a somewhat iterative process where search results are checked against articles of known relevance. Early in the process, an article is often found missing from the list. Any article that does not show up in the list is analysed and the search strategy is revised. I then use a Boolean 'NOT' to subtract the search that has already been scanned by the reviewers from the new search. In this way, no one has to duplicate their efforts and the search parameters remain as tight as possible. Generally, the search process is continued right up to publication and adapts as necessary along the way. This is just a different way to go about the process. 
 As for your techniques having a 2-3% specificity. Do you realize that means only 2-3% of the citations irrelevant to your search are properly classified as such? Did you maybe mean to indicate a 97-98% specificity? If so, I do not see how you would manage a specificity that high without sacrificing sensitivity.If you have a citation that demonstrates the validity of a such a technique, I would be very interested to see it.

Liam McKeever, MS, RDN

On 2016 Feb 25, Wichor Bramer commented:

Dear Mr McKeever,

Thank you for your detailed response to my remarks. Let me respond in return to some of your answers.

2

I was indeed referring that a thorough systematic review search ORs tiab terms with MeSH terms, not ANDs it as you do in your final search strategy. It depends on the goal of your research whether you focus on sensitivity of specificity. If you state you want to perform an exhaustive search strategy (such as is needed for systematic reviews), you should aim for sensitivity (without loosing too much specificity of course, but in the literature for SR searches a specificity of 2-3% is very normal). In your search strategy you will find relevant Medline articles on alternate day fasting only if they are also indexed with the MeSH terms you added. However you will miss important articles that have other relevant MeSH terms such as Varady KA, 2011, which has the MeSH terms Diet, Reducing, Weight Loss and Obesity/therapy. Hence you use MeSH terms to restrict your free text searches, which is not an improvement.

4

I was not objecting complicated searches in general, believe me, my SR searches are far more complicated than the one you show here (see for example: Malfliet A, 2015). However, I object unnecessary complicatedness that does not improve the search results. If I can get better results, only adding a few articles but not missing the above mentioned relevant article, by simply searching for:

("Alternate Day Fasting" OR "fasting on alternating days" OR "alternate-day fasting") NOT (animals[mh] NOT humans[mh])

I don't understand why making it much more complicated with all these steps is a payoff in thoroughness, transparence and manipulability.

Adding to that a new point of critique: are you aware that your search relies on automatic term mapping as well? If a search results page in PubMed says: Quoted phrase not found that means PubMed will try to do automatic term mapping. If you checked search details you would see that the phrase "fasting on alternating days" is reportedly not found, and replaced by

(("fasting"[MeSH Terms] OR "fasting"[All Fields]) AND alternating[All Fields] AND days[All Fields])

meaning that the user is not in control and the formula is neither robust nor reproducible nor transparent.

Sincerely, Wichor Bramer

Information specialist Erasmus MC (involved as search coordinator in hundreds of review per year)

On 2015 Nov 11, Liam McKeever commented:

Mr. Bramer,

Thank you for your critique of our work. We will respond to your comments in order in which they were written.

1. Regarding the likelihood that someone would rely heavily on a "related citations" feature to perform a review, the initial impetus for writing the paper came from such comments from seasoned researchers indicating they rely heavily on this technique. After reading the paper, this is often the first point people argue. Obviously, I would not expect a librarian to rely on such inadequate techniques, but it is happening in the field and so we addressed it. Remember that there are many forms of review where the methods of review are not made explicit, such as in grant writing. Here, researchers are more likely to fall back on the methods with which they are most familiar. For that audience, these comments were appropriate.

2. You state that there are robust search strategies which utilize both MeSH Terms and free text terms and that this has the added benefit of culling articles which may have been improperly categorized by the MEDLINE indexers. If you are referring to restricting (with Boolean AND) a MeSH search to only those folders which include certain text words in the [all fields], our strategy incorporates this. If you are instead describing duplicating your MEDLINE search by using free text words connected to the MeSH search with a Boolean OR, that is something entirely different.<br> It is true that re-searching the MEDLINE index with free text words opens the door to locating a possibly misplaced citation, but it comes with a steep cost. Let us define some terms. The sensitivity of search is the ability of that search to identify a truly relevant article. The specificity of a search is its ability to reject truly irrelevant articles. There is always a tradeoff between sensitivity and specificity. We would argue that a good search maximizes both sensitivity and specificity. The MEDLINE MeSH-term indexing method is both sensitive and highly specific. The reason for this that MEDLINE indexers read every article and catalog it according to the meaning of the article as opposed to a simple word count. To combine a MeSH-based search with a text-based version of the same search negates the entire purpose of having MEDLINE indexers. This would create a search that is highly sensitive but not very specific. Search strategies low in specificity may bypass the human error of the MEDLINE indexers, but possibly increase the human error of the scientists performing the review due to the unnecessary increase in volume of irrelevant citations.

3. It is for this reason that we must separate the search between the MEDLINE and nonMEDLINE database. We use the robust MeSH search techniques to search the MEDLINE database and restrict the less robust “free text” based techniques to search only the ~10% of PubMed that has not been indexed for MEDLINE. This creates a search that is both sensitive and specific. Failing to separate the searches decreases the specificity of your search by increasing the percentage of irrelevant citations.

4. You ask why this search needs to be so complicated. All searches are complicated. They only appear uncomplicated when you type in something simple and allow PubMed to do the thinking for you. The search on Alternate Day Fasting was chosen because it was simple enough to allow the reader to see the steps involved. Those steps are designed to put the control back in the hands of the user and refrain from relying too heavily on algorithms and automated term mapping. This requires some work up front, but the payoff is a thorough, reproducible formula that is transparent and easily manipulated as project needs change.

Sincerely, Liam McKeever, MS, RDN

On 2015 Nov 10, Wichor Bramer commented:

This article does not really give a good example of how to create search strategies. The title of the article states that it describes a method for exhaustive searches for literature reviews. I will discuss the contents of the article likewise.

The solutions described here as common are not common practice in a sense that they are performed as described by researchers writing a systematic review. Hardly any researcher will think that related citations is the proper way to do an exhaustive search. Robust search strategies can be built in both medline and non-medline part of PubMed with one search strategy using both mesh terms and free text terms, which is very common practice among both researchers and information specialists. That way not only the most recent articles are retrieved by the free text terms, but also articles where MeSH terms are incorrectly assigned will be found.

There is no need to limit the MEDLINE component to MeSH only searches, as text search might find extra relevant articles, nor is there a need to limit non-Medline part to text only searches, as this will not retrieve any articles, so no irrelevant articles as well.

The ultimate search strategy presented here is very complicated:

((("Fasting"[MeSH] OR "Obesity/diet therapy"[MeSH] OR "Weight Loss/physiology"[Mesh]) AND "Humans"[MeSH]) AND ("alternate day fasting" OR "fasting on alternating days" OR "alternate-day fasting")) OR (("Alternate Day Fasting" OR "fasting on alternating days" OR "alternate-day fasting") NOT medline[sb])

Why shoud it be so complicated? With this search articles on alternate day fasting are only retrieved if they also have one of the MeSH terms shown. Is that necessary? Each article found with this method has one of the phrases ("alternate day fasting" OR "fasting on alternating days" OR "alternate-day fasting") in the text. In that case one can just search for those phrases, and not much more. You don’t need 32 steps for that.

That search is not systematic either, but the described method does not provide a step by step approach to create a systematic search. The authors should have consulted with an information specialist before writing an article on a topic like this.

On 2015 Aug 12, Liam McKeever commented:

Dear Ms. Gluck,

Thank you for your comments. Training in the use of multiple databases was not the point of this paper. This paper was written in response to a recognition that many of the methods researchers use to perform their systematic reviews are not in fact systematic. A systematic review is meant to bring scientific methods into the process of writing a review. This means the methods of the review must be reproducible. Currently, many reviews that attempt to be truly systematic employ only the MEDLINE database because of its organized system of medical subject headings. If they do this correctly, they can perform an exhaustive search of the MEDLINE database. Our paper provided a technique for taking this systematic approach into an exhaustive search of both the MEDLINE and the PubMed databases, leading to a master formula, which could then be picked apart and improved upon by the scientific community. The techniques translate well to other databases and have recently been translated to EMBASE.

The argument that a complete systematic review should include an attempt to collect all relevant articles from multiple databases is well taken and commonly accepted. Preventing publication bias however is a much bigger picture than including multiple databases in a search strategy and was beyond the scope of this paper. To get all the null findings necessary to overcome publication bias would also mean including studies that either never entered or did not survive the peer review process. While such attempts should be made, a more achievable goal would be the thorough analysis of the publication bias present in a review where the search methodology is both explicit and reproducible.

The selection of appropriate databases for a systematic review, as you implied, varies greatly by profession. It was therefore also not in the scope of this paper. I do think there is some value in considering what it actually means than not all databases contain all journals. I find it highly unlikely that a bio-medically relevant journal would not be indexed in MEDLINE simply because they neglected to apply. It is much more likely that they applied and were rejected. Just as a systematic review has inclusion criteria at the level of the articles selected, databases have inclusion criteria at the level of the journals selected for cataloging. The degree of research quality and scope of topic areas are considered and determined to either meet or not meet the standards of the database. When we select a database for a systematic review, we are defining our inclusion criteria at the level of the journal. With this in mind, assuming adequate search methods were used and provided a thorough analysis of publication bias has been performed, one could make the argument that a properly selected major database pairing, like MEDLINE and PubMed may be acceptable for a systematic review. From a scientific methods perspective, we feel what is most important is that the inclusion criteria at all levels are explicit and that is what this paper attempts to facilitate.

Sincerely, Liam McKeever, MS, RDN

On 2015 Aug 06, Jeannine Gluck commented:

The authors give the impression that an exhaustive and comprehensive search can be carried out in a single database. Not so. While PubMed/MEDLINE clearly covers a very large number of publications, no database is comprehensive. EMBASE, for one example, has better coverage of the European and pharmaceutical literature. There are many subject-specific databases, any one of which may be relevant for a search in a particular aspect of the health sciences. Researchers looking at educational or sociological aspects of their field would do well to consult resources in those disciplines.

Workshops on systematic reviews emphasize, first and foremost, the requirement that multiple sources be consulted in order to minimize bias. This article was not about systematic reviews, per se. Still, any researchers calling their search "exhaustive" would do well to heed this advice. Had the authors included a librarian on their team, this serious oversight would not likely have happened.

On 2015 Aug 06, Jeannine Gluck commented:

The authors give the impression that an exhaustive and comprehensive search can be carried out in a single database. Not so. While PubMed/MEDLINE clearly covers a very large number of publications, no database is comprehensive. EMBASE, for one example, has better coverage of the European and pharmaceutical literature. There are many subject-specific databases, any one of which may be relevant for a search in a particular aspect of the health sciences. Researchers looking at educational or sociological aspects of their field would do well to consult resources in those disciplines.

Workshops on systematic reviews emphasize, first and foremost, the requirement that multiple sources be consulted in order to minimize bias. This article was not about systematic reviews, per se. Still, any researchers calling their search "exhaustive" would do well to heed this advice. Had the authors included a librarian on their team, this serious oversight would not likely have happened.

On 2015 Aug 12, Liam McKeever commented:

Dear Ms. Gluck,

Thank you for your comments. Training in the use of multiple databases was not the point of this paper. This paper was written in response to a recognition that many of the methods researchers use to perform their systematic reviews are not in fact systematic. A systematic review is meant to bring scientific methods into the process of writing a review. This means the methods of the review must be reproducible. Currently, many reviews that attempt to be truly systematic employ only the MEDLINE database because of its organized system of medical subject headings. If they do this correctly, they can perform an exhaustive search of the MEDLINE database. Our paper provided a technique for taking this systematic approach into an exhaustive search of both the MEDLINE and the PubMed databases, leading to a master formula, which could then be picked apart and improved upon by the scientific community. The techniques translate well to other databases and have recently been translated to EMBASE.

The argument that a complete systematic review should include an attempt to collect all relevant articles from multiple databases is well taken and commonly accepted. Preventing publication bias however is a much bigger picture than including multiple databases in a search strategy and was beyond the scope of this paper. To get all the null findings necessary to overcome publication bias would also mean including studies that either never entered or did not survive the peer review process. While such attempts should be made, a more achievable goal would be the thorough analysis of the publication bias present in a review where the search methodology is both explicit and reproducible.

The selection of appropriate databases for a systematic review, as you implied, varies greatly by profession. It was therefore also not in the scope of this paper. I do think there is some value in considering what it actually means than not all databases contain all journals. I find it highly unlikely that a bio-medically relevant journal would not be indexed in MEDLINE simply because they neglected to apply. It is much more likely that they applied and were rejected. Just as a systematic review has inclusion criteria at the level of the articles selected, databases have inclusion criteria at the level of the journals selected for cataloging. The degree of research quality and scope of topic areas are considered and determined to either meet or not meet the standards of the database. When we select a database for a systematic review, we are defining our inclusion criteria at the level of the journal. With this in mind, assuming adequate search methods were used and provided a thorough analysis of publication bias has been performed, one could make the argument that a properly selected major database pairing, like MEDLINE and PubMed may be acceptable for a systematic review. From a scientific methods perspective, we feel what is most important is that the inclusion criteria at all levels are explicit and that is what this paper attempts to facilitate.

Sincerely, Liam McKeever, MS, RDN

On 2015 Nov 10, Wichor Bramer commented:

This article does not really give a good example of how to create search strategies. The title of the article states that it describes a method for exhaustive searches for literature reviews. I will discuss the contents of the article likewise.

The solutions described here as common are not common practice in a sense that they are performed as described by researchers writing a systematic review. Hardly any researcher will think that related citations is the proper way to do an exhaustive search. Robust search strategies can be built in both medline and non-medline part of PubMed with one search strategy using both mesh terms and free text terms, which is very common practice among both researchers and information specialists. That way not only the most recent articles are retrieved by the free text terms, but also articles where MeSH terms are incorrectly assigned will be found.

There is no need to limit the MEDLINE component to MeSH only searches, as text search might find extra relevant articles, nor is there a need to limit non-Medline part to text only searches, as this will not retrieve any articles, so no irrelevant articles as well.

The ultimate search strategy presented here is very complicated:

((("Fasting"[MeSH] OR "Obesity/diet therapy"[MeSH] OR "Weight Loss/physiology"[Mesh]) AND "Humans"[MeSH]) AND ("alternate day fasting" OR "fasting on alternating days" OR "alternate-day fasting")) OR (("Alternate Day Fasting" OR "fasting on alternating days" OR "alternate-day fasting") NOT medline[sb])

Why shoud it be so complicated? With this search articles on alternate day fasting are only retrieved if they also have one of the MeSH terms shown. Is that necessary? Each article found with this method has one of the phrases ("alternate day fasting" OR "fasting on alternating days" OR "alternate-day fasting") in the text. In that case one can just search for those phrases, and not much more. You don’t need 32 steps for that.

That search is not systematic either, but the described method does not provide a step by step approach to create a systematic search. The authors should have consulted with an information specialist before writing an article on a topic like this.

On 2015 Nov 11, Liam McKeever commented:

Mr. Bramer,

Thank you for your critique of our work. We will respond to your comments in order in which they were written.

1. Regarding the likelihood that someone would rely heavily on a "related citations" feature to perform a review, the initial impetus for writing the paper came from such comments from seasoned researchers indicating they rely heavily on this technique. After reading the paper, this is often the first point people argue. Obviously, I would not expect a librarian to rely on such inadequate techniques, but it is happening in the field and so we addressed it. Remember that there are many forms of review where the methods of review are not made explicit, such as in grant writing. Here, researchers are more likely to fall back on the methods with which they are most familiar. For that audience, these comments were appropriate.

2. You state that there are robust search strategies which utilize both MeSH Terms and free text terms and that this has the added benefit of culling articles which may have been improperly categorized by the MEDLINE indexers. If you are referring to restricting (with Boolean AND) a MeSH search to only those folders which include certain text words in the [all fields], our strategy incorporates this. If you are instead describing duplicating your MEDLINE search by using free text words connected to the MeSH search with a Boolean OR, that is something entirely different.<br> It is true that re-searching the MEDLINE index with free text words opens the door to locating a possibly misplaced citation, but it comes with a steep cost. Let us define some terms. The sensitivity of search is the ability of that search to identify a truly relevant article. The specificity of a search is its ability to reject truly irrelevant articles. There is always a tradeoff between sensitivity and specificity. We would argue that a good search maximizes both sensitivity and specificity. The MEDLINE MeSH-term indexing method is both sensitive and highly specific. The reason for this that MEDLINE indexers read every article and catalog it according to the meaning of the article as opposed to a simple word count. To combine a MeSH-based search with a text-based version of the same search negates the entire purpose of having MEDLINE indexers. This would create a search that is highly sensitive but not very specific. Search strategies low in specificity may bypass the human error of the MEDLINE indexers, but possibly increase the human error of the scientists performing the review due to the unnecessary increase in volume of irrelevant citations.

3. It is for this reason that we must separate the search between the MEDLINE and nonMEDLINE database. We use the robust MeSH search techniques to search the MEDLINE database and restrict the less robust “free text” based techniques to search only the ~10% of PubMed that has not been indexed for MEDLINE. This creates a search that is both sensitive and specific. Failing to separate the searches decreases the specificity of your search by increasing the percentage of irrelevant citations.

4. You ask why this search needs to be so complicated. All searches are complicated. They only appear uncomplicated when you type in something simple and allow PubMed to do the thinking for you. The search on Alternate Day Fasting was chosen because it was simple enough to allow the reader to see the steps involved. Those steps are designed to put the control back in the hands of the user and refrain from relying too heavily on algorithms and automated term mapping. This requires some work up front, but the payoff is a thorough, reproducible formula that is transparent and easily manipulated as project needs change.

Sincerely, Liam McKeever, MS, RDN

On 2016 Feb 29, Rhonda J Allard commented:

While I see the point you make about being sure your search strategies take into account both Medline and non-Medline articles, I don't believe you need to create two searches to accomplish this goal. The following search strategy (in my opinion), is as effective:

fasting AND (alternate day* OR alternating day*) -- it retrieves 120 articles

Notes: 1) The term fasting will map to Fasting [Mesh] and also search "fasting" in all fields. This will retrieve both Medline and non-Medline articles. 2) Anytime you use double quotes for phrase searching or the asterisk for truncation you are turning off the "Automatic Term Mapping" and searching all of PubMed.

If the searcher is concerned with the amount of irrelevant articles they might retrieve when using keywords, another trick is to limit keywords to the title or abstract using [tiab], see below:

military retrieves 127,746 results (it maps to Military Personnel[Mesh] and searches for "military" in All Fields) whereas (Military Personnel[MH] OR military[tiab]) retrieves 56,624 results. This second search limits the search results to where the topic is the military. The term military can appear in the Author Affiliation field or as part of another MeSH term.