On 2015 Sep 22, Kenneth Witwer commented:

Dr. Dweep's response is appreciated but does not engage our suggestions or explain why the miRWalk validated target module was:

1) extensively erroneous (1.0, as we pointed out I believe in 2011),

2) greatly expanded and again erroneous (2.0 at the start of 2015, apparently due to a database error, and as we shared with the authors earlier this year),

3) and finally what appears to be a mirror of miRTarBase, at least according to our results.

Instability and a lack of clear information on versions and methods causes confusion, especially when scientists take these data at face value. I have reviewed submitted manuscripts that used miRWalk "validated" results as input for further experiments or analysis, even when these results were, unbeknownst to the authors, completely erroneous.

Dr. Dweep suggests that our analysis was deficient because of the terms we used for our 50 selected genes. This would indeed be important had we attempted to gauge the comprehensiveness of miRWalk or other databases (we expressly did not), or if we had used one set of terms for our miRWalk query, and another set for querying all the other databases (we did not, unless forced to do so by an interface). Which gene terms we used in our comparison of databases, then, is irrelevant.

Dr. Dweep's second point is that our analysis focused only on a small portion of the miRWalk database, the validated target module. Should we have ignored perceived problems with such modules, simply because the file sizes for these data are smaller than the sum of all predictive or correlative information on miRNAs in any given database, or on the internet, or whatever other set of information we might consider?

Finally, Dr. Dweep refers to supplemental methods that explain, albeit in vague terms that do not allow reproduction of the results, how validated targets are gathered using text searches and four databases. This does not explain why the results we downloaded from miRWalk2.0 in the period of April-August 2015 were exactly the same as found in another database, miRTarBase (last updated two years ago), down to the sorting of the hits, nor does it explain the drastic fluctuations in numbers of validated hits over the years, almost all of which we examined were erroneous. Thus, a miRWalk validated target module user in January, 2015, would have received a completely different set of results compared with the same query a year earlier or six months later. As we have suggested to Dr. Dweep, one might simply to link to miRTarBase in cases where miRWalk2.0 does not provide additional information, and provide more extensive versioning information or even an automated email update to users when mistakes are found or major changes implemented.

We agree that validated target databases have potential uses and hope that our findings are somehow helpful, as a cautionary note even if they are not used to improve databases.

On 2015 Sep 22, Harsh Dweep commented:

We congratulate Dr. Witwer for the publication of his results in Drug Development Research journal. This publication is based on analyses of only one of the many key features of miRWalk2.0 comparing it with 50 genes by considering GeneCard terminology. Here, we would like to mention that the gene synonymous information is neither comprehensively maintained by Genecard nor MeSH (and other databases). For example, a total of 14, 22 and 23 synonyms are documented in NCBI-gene, MeSH and GeneCards, respectively, for the Clusterin (CLU) gene. Only 5 synonyms are common between GeneCards (as used by Witwer et al based on) and MeSH. However, the information of PubMed is relying on the terms stored in MeSH. By considering only GeneCards for evaluation (text-mining), a large amount of information on synonyms as well as their related articles can be missed. In addition, an alias of a gene can be commonly found in other genes, for example, VH alias is common for 36 different genes. These comments reflect only part of the problems related to text mining.

Moreover, Witwer addresses only 0.008% of the information contained in miRWalk2.0.

Additionally, it is clearly mentioned in the supplementary information of miRWalk2.0 article (page 11) that “information on miRNA interactions associated with pathways, diseases, OMIM disorders, HPOs, organs, cell lines and proteins involved in miRNA processing, is extracted by an automated text-mining search in the titles and abstracts of PubMed articles. In a next step, this information was combined with experimentally verified miRNA-target interactions downloaded from four databases”.

On 2015 Sep 03, Kenneth Witwer commented:

We recently published a small comparison of several validated miRNA-target databases (Lee YJ, 2015)--that is, catalogs of miRNA-target interactions with published experimental evidence. A "validated target" module is one part of miRWalk2.0, so this database and its predecessor (1.0) were included in our study. We queried 50 genes at different times. 82 miRNA-target interactions were returned by miRWalk1.0, 5468 by miRWalk2.0 in January, 2015, and 91 by miRWalk2.0 in May, June, and August, 2015, with only 5 from the original 82. As of August, 2015, the final set of 91 interactions was identical to that returned by miRTarBase (Hsu SD, 2014, Hsu SD, 2011), down to the sort order. Although miRTarBase is cited as one of numerous sources of information for miRWalk output, it was not clear from the methods that it would be the only source for the genes we queried. Experimental validation databases have the potential to provide useful information, but in light of the stability and accuracy issues we seem to have observed over time, users and reviewers are encouraged to 1) consult multiple sources of information (we found Diana TarBase to be among the most comprehensive and best-curated of those we tested, Vlachos IS, 2015); 2) at the same time be aware that different databases may rely entirely or to some extent on other databases; and 3) check the strength of interaction evidence in the primary literature.

On 2015 Sep 03, Kenneth Witwer commented:

We recently published a small comparison of several validated miRNA-target databases (Lee YJ, 2015)--that is, catalogs of miRNA-target interactions with published experimental evidence. A "validated target" module is one part of miRWalk2.0, so this database and its predecessor (1.0) were included in our study. We queried 50 genes at different times. 82 miRNA-target interactions were returned by miRWalk1.0, 5468 by miRWalk2.0 in January, 2015, and 91 by miRWalk2.0 in May, June, and August, 2015, with only 5 from the original 82. As of August, 2015, the final set of 91 interactions was identical to that returned by miRTarBase (Hsu SD, 2014, Hsu SD, 2011), down to the sort order. Although miRTarBase is cited as one of numerous sources of information for miRWalk output, it was not clear from the methods that it would be the only source for the genes we queried. Experimental validation databases have the potential to provide useful information, but in light of the stability and accuracy issues we seem to have observed over time, users and reviewers are encouraged to 1) consult multiple sources of information (we found Diana TarBase to be among the most comprehensive and best-curated of those we tested, Vlachos IS, 2015); 2) at the same time be aware that different databases may rely entirely or to some extent on other databases; and 3) check the strength of interaction evidence in the primary literature.

On 2015 Sep 22, Harsh Dweep commented:

We congratulate Dr. Witwer for the publication of his results in Drug Development Research journal. This publication is based on analyses of only one of the many key features of miRWalk2.0 comparing it with 50 genes by considering GeneCard terminology. Here, we would like to mention that the gene synonymous information is neither comprehensively maintained by Genecard nor MeSH (and other databases). For example, a total of 14, 22 and 23 synonyms are documented in NCBI-gene, MeSH and GeneCards, respectively, for the Clusterin (CLU) gene. Only 5 synonyms are common between GeneCards (as used by Witwer et al based on) and MeSH. However, the information of PubMed is relying on the terms stored in MeSH. By considering only GeneCards for evaluation (text-mining), a large amount of information on synonyms as well as their related articles can be missed. In addition, an alias of a gene can be commonly found in other genes, for example, VH alias is common for 36 different genes. These comments reflect only part of the problems related to text mining.

Moreover, Witwer addresses only 0.008% of the information contained in miRWalk2.0.

Additionally, it is clearly mentioned in the supplementary information of miRWalk2.0 article (page 11) that “information on miRNA interactions associated with pathways, diseases, OMIM disorders, HPOs, organs, cell lines and proteins involved in miRNA processing, is extracted by an automated text-mining search in the titles and abstracts of PubMed articles. In a next step, this information was combined with experimentally verified miRNA-target interactions downloaded from four databases”.

On 2015 Sep 22, Kenneth Witwer commented:

Dr. Dweep's response is appreciated but does not engage our suggestions or explain why the miRWalk validated target module was:

1) extensively erroneous (1.0, as we pointed out I believe in 2011),

2) greatly expanded and again erroneous (2.0 at the start of 2015, apparently due to a database error, and as we shared with the authors earlier this year),

3) and finally what appears to be a mirror of miRTarBase, at least according to our results.

Instability and a lack of clear information on versions and methods causes confusion, especially when scientists take these data at face value. I have reviewed submitted manuscripts that used miRWalk "validated" results as input for further experiments or analysis, even when these results were, unbeknownst to the authors, completely erroneous.

Dr. Dweep suggests that our analysis was deficient because of the terms we used for our 50 selected genes. This would indeed be important had we attempted to gauge the comprehensiveness of miRWalk or other databases (we expressly did not), or if we had used one set of terms for our miRWalk query, and another set for querying all the other databases (we did not, unless forced to do so by an interface). Which gene terms we used in our comparison of databases, then, is irrelevant.

Dr. Dweep's second point is that our analysis focused only on a small portion of the miRWalk database, the validated target module. Should we have ignored perceived problems with such modules, simply because the file sizes for these data are smaller than the sum of all predictive or correlative information on miRNAs in any given database, or on the internet, or whatever other set of information we might consider?

Finally, Dr. Dweep refers to supplemental methods that explain, albeit in vague terms that do not allow reproduction of the results, how validated targets are gathered using text searches and four databases. This does not explain why the results we downloaded from miRWalk2.0 in the period of April-August 2015 were exactly the same as found in another database, miRTarBase (last updated two years ago), down to the sorting of the hits, nor does it explain the drastic fluctuations in numbers of validated hits over the years, almost all of which we examined were erroneous. Thus, a miRWalk validated target module user in January, 2015, would have received a completely different set of results compared with the same query a year earlier or six months later. As we have suggested to Dr. Dweep, one might simply to link to miRTarBase in cases where miRWalk2.0 does not provide additional information, and provide more extensive versioning information or even an automated email update to users when mistakes are found or major changes implemented.

We agree that validated target databases have potential uses and hope that our findings are somehow helpful, as a cautionary note even if they are not used to improve databases.