On 2014 May 22, Jacob Puliyel commented:

Post-marketing surveillance cannot properly estimate risks

The authors must be congratulated for conducting a study whose scope was so vast. However it must be recognized that RCTs are the best method to look for risks and benefits. Post-marketing surveillance is relied on to detect rare adverse events. It has severe limitations in quantifying the magnitude of risks. The present study highlights the problem well.

1 The authors estimate the risk of intussusceptions (IS) in two window periods after immunization. Previously surveillance had found that the incidence of IS was significantly increased in the first 3 weeks and more after the first dose than after the second dose. But that does not imply the risk of IS is limited to this window period alone. Only long term RCTs can really identify the full risk of IS after immunization and can categorically confirm that there is no risk in other periods.

2 For estimating reduction of diarrhea the authors use the hospital discharge diagnosis coded rota virus (A08.0) or acute gastroenteritis excluding rota virus (A01-A09, K32 excluding A08.0) multiplied by proportion of the cases estimated to be rota virus.

I am not sure how coding is done in the National Morbidity Database of the Australian Institute of Health and Welfare, but in many countries, diarrhea where rota virus is identified is coded as rota virus diarrhea, even where other pathogenic organisms are identified alongside and rota virus may not be the cause of the episode of diarrhea. The new rota virus vaccine strain from India, the monovalent human-bovine (116E) rotavirus, was cultured from an asymptomatic neonate in India. http://www.ncbi.nlm.nih.gov/pubmed/24629994. Some strains of rota virus could be a harmless commensals. The harmless rota virus may be protecting the individual imparting natural immunity without need for vaccines. Just because an organism is identified in a child with diarrhea, it does not necessarily imply that the organism is the cause of the diarrhea.

Furthermore, there are obvious problems in assuming that a fixed proportion of all diarrhea which are 'not coded as rota virus,' are due to rota virus.

Given these drawbacks, the estimate of 6500 rota virus hospitalizations avoided in Australia may not be accurate.

The judgment on acceptability of the risk-benefit equation pivots on the trade off between the putative 6500 rota virus admissions avoided on the one hand and the 14 cases of IS caused by the vaccine on the other, and these figures may not be reliable in the first place.

3 In their conclusions in the Abstract the authors say the "balance of risks and benefits at population level was highly favorable – a finding likely to extend to other settings despite varying incidence of IS and potentially higher morbidity and mortality from gastroenteritis and IS." This conclusion is clearly debatable.

The morbidity and mortality from gastroenteritis may be high in developing countries, but the morbidity and mortality for IS is disproportionately higher. Most developing countries can manage dehydration but facilities for surgical and radiological management of IS may not be available in large areas, making the risk unacceptable in accordance with the principle of primum non nocere.

4 Another factor the authors do not consider in their conclusion (that the findings are likely to extend to other countries,) is the varying vaccine efficacy seen in different countries. While the efficacy is nearly 90% in Western countries it is barely 50% in the tropics. Although the authors do not refer to the study by Madhi et al http://www.ncbi.nlm.nih.gov/pubmed/20107214 it is often quoted in this context. Severe rota virus gastroenteritis (SRVGE) was more common in Malawi than South Africa (13.1 vs. 5.4) and even though efficacy was lower in Malawi (49.4% vs. 76.9%) more cases of SRVGE were prevented by vaccination (6.7 vs. 4.2) in Malawi. This is often given as the justification for using the vaccine (with such low efficacy) in poor tropical countries.

This does not apply to all nations in the tropics. Although the incidence of gastroenteritis is high in India, the incidence of rota virus diarrhea was even less than South Africa. The incidence SRVGE in the unvaccinated in India was 3.4% compared to 13.1 in Malawi and 5.4 in South Africa. The absolute risk reduction (ARR) by vaccination was tiny in India (1.7). This is much lower than the benefit in Malawi (6.7) and even South Africa (4.2). It raises questions about the need for the vaccine in countries like India using the ‘disease burden’ argument. http://www.ncbi.nlm.nih.gov/pubmed/24629994#cm24629994_3808. Clearly each country needs to evaluate local risks and benefits and a blanket recommendation for all countries for the vaccine is perhaps not appropriate.

5 In conclusion: This study clearly shows the problems of relying on post marketing surveillance to evaluate harms. After unusual adverse events have been identified during post marketing surveillance, (if they are serious in nature) the vaccine must be withdrawn and reassessed in RCTs of sufficient size.

6 The data from the small 2-year follow-up RCT of the Indian vaccine 116E (4500 children received the new rota virus vaccine) will help understand if the '3 week window' is adequate to identify all adverse events. This follow-up paper is awaited. The preliminary report is available http://www.ncbi.nlm.nih.gov/pubmed/24629994#cm24629994. From the initial data it appears that the incidence of IS may be higher with this vaccine and so it may be a good vaccine to study the ‘window period’ of increased risk of IS.

On 2014 May 22, Jacob Puliyel commented:

Post-marketing surveillance cannot properly estimate risks

The authors must be congratulated for conducting a study whose scope was so vast. However it must be recognized that RCTs are the best method to look for risks and benefits. Post-marketing surveillance is relied on to detect rare adverse events. It has severe limitations in quantifying the magnitude of risks. The present study highlights the problem well.

1 The authors estimate the risk of intussusceptions (IS) in two window periods after immunization. Previously surveillance had found that the incidence of IS was significantly increased in the first 3 weeks and more after the first dose than after the second dose. But that does not imply the risk of IS is limited to this window period alone. Only long term RCTs can really identify the full risk of IS after immunization and can categorically confirm that there is no risk in other periods.

2 For estimating reduction of diarrhea the authors use the hospital discharge diagnosis coded rota virus (A08.0) or acute gastroenteritis excluding rota virus (A01-A09, K32 excluding A08.0) multiplied by proportion of the cases estimated to be rota virus.

I am not sure how coding is done in the National Morbidity Database of the Australian Institute of Health and Welfare, but in many countries, diarrhea where rota virus is identified is coded as rota virus diarrhea, even where other pathogenic organisms are identified alongside and rota virus may not be the cause of the episode of diarrhea. The new rota virus vaccine strain from India, the monovalent human-bovine (116E) rotavirus, was cultured from an asymptomatic neonate in India. http://www.ncbi.nlm.nih.gov/pubmed/24629994. Some strains of rota virus could be a harmless commensals. The harmless rota virus may be protecting the individual imparting natural immunity without need for vaccines. Just because an organism is identified in a child with diarrhea, it does not necessarily imply that the organism is the cause of the diarrhea.

Furthermore, there are obvious problems in assuming that a fixed proportion of all diarrhea which are 'not coded as rota virus,' are due to rota virus.

Given these drawbacks, the estimate of 6500 rota virus hospitalizations avoided in Australia may not be accurate.

The judgment on acceptability of the risk-benefit equation pivots on the trade off between the putative 6500 rota virus admissions avoided on the one hand and the 14 cases of IS caused by the vaccine on the other, and these figures may not be reliable in the first place.

3 In their conclusions in the Abstract the authors say the "balance of risks and benefits at population level was highly favorable – a finding likely to extend to other settings despite varying incidence of IS and potentially higher morbidity and mortality from gastroenteritis and IS." This conclusion is clearly debatable.

The morbidity and mortality from gastroenteritis may be high in developing countries, but the morbidity and mortality for IS is disproportionately higher. Most developing countries can manage dehydration but facilities for surgical and radiological management of IS may not be available in large areas, making the risk unacceptable in accordance with the principle of primum non nocere.

4 Another factor the authors do not consider in their conclusion (that the findings are likely to extend to other countries,) is the varying vaccine efficacy seen in different countries. While the efficacy is nearly 90% in Western countries it is barely 50% in the tropics. Although the authors do not refer to the study by Madhi et al http://www.ncbi.nlm.nih.gov/pubmed/20107214 it is often quoted in this context. Severe rota virus gastroenteritis (SRVGE) was more common in Malawi than South Africa (13.1 vs. 5.4) and even though efficacy was lower in Malawi (49.4% vs. 76.9%) more cases of SRVGE were prevented by vaccination (6.7 vs. 4.2) in Malawi. This is often given as the justification for using the vaccine (with such low efficacy) in poor tropical countries.

This does not apply to all nations in the tropics. Although the incidence of gastroenteritis is high in India, the incidence of rota virus diarrhea was even less than South Africa. The incidence SRVGE in the unvaccinated in India was 3.4% compared to 13.1 in Malawi and 5.4 in South Africa. The absolute risk reduction (ARR) by vaccination was tiny in India (1.7). This is much lower than the benefit in Malawi (6.7) and even South Africa (4.2). It raises questions about the need for the vaccine in countries like India using the ‘disease burden’ argument. http://www.ncbi.nlm.nih.gov/pubmed/24629994#cm24629994_3808. Clearly each country needs to evaluate local risks and benefits and a blanket recommendation for all countries for the vaccine is perhaps not appropriate.

5 In conclusion: This study clearly shows the problems of relying on post marketing surveillance to evaluate harms. After unusual adverse events have been identified during post marketing surveillance, (if they are serious in nature) the vaccine must be withdrawn and reassessed in RCTs of sufficient size.

6 The data from the small 2-year follow-up RCT of the Indian vaccine 116E (4500 children received the new rota virus vaccine) will help understand if the '3 week window' is adequate to identify all adverse events. This follow-up paper is awaited. The preliminary report is available http://www.ncbi.nlm.nih.gov/pubmed/24629994#cm24629994. From the initial data it appears that the incidence of IS may be higher with this vaccine and so it may be a good vaccine to study the ‘window period’ of increased risk of IS.