On 2016 Aug 22, Anthony Jorm commented:

The Royal Australian and New Zealand College of Psychiatrists (RANZCP) has recently published clinical practice guidelines on schizophrenia Galletly C, 2016, eating disorders Hay P, 2014 and mood disorders Malhi GS, 2015. These guidelines contain a mixture of evidence-based recommendations where there were relevant intervention studies, and consensus-based recommendations where relevant studies did not exist. The consensus-based recommendations comprised a substantial proportion of the guidelines for mood disorders (59%) and schizophrenia (46%), but less so for eating disorders (10%), indicating that expert consensus is an important source of guidance on best practice in psychiatry.

Given the substantial contribution of expert consensus to these guidelines, it is important that the methods for establishing this consensus are adequate. The Australian National Health and Medical Research Council (NHMRC) has published requirements for development of clinical practice guidelines, but these do not give much guidance on how this should be done, simply mandating that “The method used to arrive at consensus-based recommendations or points (e.g. voting or formal methods, such as Delphi) is documented”. (National Health and Medical Research Council. Procedures and requirements for meeting the 2011 NHMRC standard for clinical practice guidelines. Melbourne: National Health and Medical Research Council; 2011.)

Another potential source of criteria for evaluating the quality of methods for developing consensus-based recommendations comes from research on ‘wisdom of crowds’ Lorenz J, 2011 Kattan MW, 2016 Baumeister RF, 2016. Based on such research, Surowiecki has proposed four conditions necessary for a group to make good decisions (Surowiecki J. The wisdom of crowds: why the many are smarter than the few. London: Abacus; 2004.): 1. Diversity of expertise. A heterogeneous group of experts will produce better quality decisions than a homogeneous one. For guidelines developers, this may mean that the experts should come from a range of relevant disciplines, including consumer experts where appropriate. 2. Independence. The experts must be able to make their decisions independently, so that they are not influenced by others. For guidelines developers, this means that voting on consensus-based recommendations is carried out privately so that strong individuals cannot dominate the group. 3. Decentralization. Expertise is held by autonomous individuals working in a decentralized way. For guidelines developers, it is important to specify what sources of information the experts had available to them. 4. Aggregation. There is a mechanism for coordinating and aggregating the group’s expertise. For guideline developers, this could involve an independent person who runs the voting and gives feedback to the group.

If we take these four conditions as appropriate for judging the quality of methods for developing consensus-based recommendations, how well do the RANZCP guidelines meet them?

An indication of diversity of expertise is the disciplinary composition of the guideline working groups. There was limited diversity for all working groups, with non-psychiatrists comprising 3 of the 8 members for eating disorders, 4 out of the 12 members for mood disorders and 2 out of the 10 members for schizophrenia working group. There were no consumer or carer members of any of the working groups. While the mood disorder and schizophrenia guidelines included consensus-based recommendations s for indigenous peoples, it is not stated whether any of the working groups included indigenous members.

The mood disorders and schizophrenia working groups did not appear to involve independent decision making. Both groups had discussions until consensus was reached. The eating disorders guidelines did not give relevant information about whether there was independence.

All three guidelines state that consensus-based recommendations were based on collective clinical and research knowledge and experience. The eating disorder guidelines additionally state that level IV articles were considered where higher-level evidence was lacking and this informed the consensus-based recommendations.

After drafting, all guidelines had input from a broader group of expert advisers with a wide diversity of expertise. However, it is not clear whether these advisers had the potential to persuade working group members to change consensus-based recommendations.

Where the guidelines included consensus-based recommendations relevant to indigenous peoples, it is not clear what sources of cultural expertise these were based on.

None of the guidelines state how judgements were aggregated to determine consensus. The mood disorders guidelines state that agreement on consensus-based recommendations was “in most cases unanimous but allowed one committee member to abstain”. The other guidelines did not define what constituted consensus.

In conclusion, there are major weaknesses in the procedures used to determine consensus-based recommendations for all three guidelines. These are lack of independence in decision making by experts, a lack of a formal mechanism for aggregating judgments, and a lack of diversity of expertise, particular in areas where consumers and carers could contribute and where cultural expertise is relevant.

While NHMRC gives quite detailed guidance on how to develop evidence-based recommendations, there is little guidance on best practice for developing consensus-based recommendations. While many of these weaknesses would be overcome by using formal consensus methods such as the Delphi process, there is a need for NHMRC and similar agencies to produce more rigorous quality standards for development of consensus-based recommendations.

On 2016 Aug 22, Anthony Jorm commented:

The Royal Australian and New Zealand College of Psychiatrists (RANZCP) has recently published clinical practice guidelines on schizophrenia Galletly C, 2016, eating disorders Hay P, 2014 and mood disorders Malhi GS, 2015. These guidelines contain a mixture of evidence-based recommendations where there were relevant intervention studies, and consensus-based recommendations where relevant studies did not exist. The consensus-based recommendations comprised a substantial proportion of the guidelines for mood disorders (59%) and schizophrenia (46%), but less so for eating disorders (10%), indicating that expert consensus is an important source of guidance on best practice in psychiatry.

Given the substantial contribution of expert consensus to these guidelines, it is important that the methods for establishing this consensus are adequate. The Australian National Health and Medical Research Council (NHMRC) has published requirements for development of clinical practice guidelines, but these do not give much guidance on how this should be done, simply mandating that “The method used to arrive at consensus-based recommendations or points (e.g. voting or formal methods, such as Delphi) is documented”. (National Health and Medical Research Council. Procedures and requirements for meeting the 2011 NHMRC standard for clinical practice guidelines. Melbourne: National Health and Medical Research Council; 2011.)

Another potential source of criteria for evaluating the quality of methods for developing consensus-based recommendations comes from research on ‘wisdom of crowds’ Lorenz J, 2011 Kattan MW, 2016 Baumeister RF, 2016. Based on such research, Surowiecki has proposed four conditions necessary for a group to make good decisions (Surowiecki J. The wisdom of crowds: why the many are smarter than the few. London: Abacus; 2004.): 1. Diversity of expertise. A heterogeneous group of experts will produce better quality decisions than a homogeneous one. For guidelines developers, this may mean that the experts should come from a range of relevant disciplines, including consumer experts where appropriate. 2. Independence. The experts must be able to make their decisions independently, so that they are not influenced by others. For guidelines developers, this means that voting on consensus-based recommendations is carried out privately so that strong individuals cannot dominate the group. 3. Decentralization. Expertise is held by autonomous individuals working in a decentralized way. For guidelines developers, it is important to specify what sources of information the experts had available to them. 4. Aggregation. There is a mechanism for coordinating and aggregating the group’s expertise. For guideline developers, this could involve an independent person who runs the voting and gives feedback to the group.

If we take these four conditions as appropriate for judging the quality of methods for developing consensus-based recommendations, how well do the RANZCP guidelines meet them?

An indication of diversity of expertise is the disciplinary composition of the guideline working groups. There was limited diversity for all working groups, with non-psychiatrists comprising 3 of the 8 members for eating disorders, 4 out of the 12 members for mood disorders and 2 out of the 10 members for schizophrenia working group. There were no consumer or carer members of any of the working groups. While the mood disorder and schizophrenia guidelines included consensus-based recommendations s for indigenous peoples, it is not stated whether any of the working groups included indigenous members.

The mood disorders and schizophrenia working groups did not appear to involve independent decision making. Both groups had discussions until consensus was reached. The eating disorders guidelines did not give relevant information about whether there was independence.

All three guidelines state that consensus-based recommendations were based on collective clinical and research knowledge and experience. The eating disorder guidelines additionally state that level IV articles were considered where higher-level evidence was lacking and this informed the consensus-based recommendations.

After drafting, all guidelines had input from a broader group of expert advisers with a wide diversity of expertise. However, it is not clear whether these advisers had the potential to persuade working group members to change consensus-based recommendations.

Where the guidelines included consensus-based recommendations relevant to indigenous peoples, it is not clear what sources of cultural expertise these were based on.

None of the guidelines state how judgements were aggregated to determine consensus. The mood disorders guidelines state that agreement on consensus-based recommendations was “in most cases unanimous but allowed one committee member to abstain”. The other guidelines did not define what constituted consensus.

In conclusion, there are major weaknesses in the procedures used to determine consensus-based recommendations for all three guidelines. These are lack of independence in decision making by experts, a lack of a formal mechanism for aggregating judgments, and a lack of diversity of expertise, particular in areas where consumers and carers could contribute and where cultural expertise is relevant.

While NHMRC gives quite detailed guidance on how to develop evidence-based recommendations, there is little guidance on best practice for developing consensus-based recommendations. While many of these weaknesses would be overcome by using formal consensus methods such as the Delphi process, there is a need for NHMRC and similar agencies to produce more rigorous quality standards for development of consensus-based recommendations.