Six Factors Considered by Nrepp Reviewers When Evaluating the Quality of Research

In that location are several reasons why an experiment cannot exist replicated.

Independent verification of data is a fundamental principle of scientific enquiry across the disciplines. The cocky-correcting mechanisms of the scientific method depend on the ability of researchers to reproduce the findings of published studies in order to strengthen evidence and build upon existing piece of work. Stanford University medical researcher, John Ioannidis, a prominent scholar on reproducibility in science, has pointed out that the importance of reproducibility does non have to practise with ensuring the 'correctness' of results, but rather with ensuring the transparency of exactly what was done in a given line of research¹.

In theory, researchers should be able to re-create experiments, generate the same results, and get in at the aforementioned conclusions, thus helping to validate and strengthen the original work. However, reality does non e'er encounter these expectations. Too often, scientific findings in biomedical research cannot be reproduced²; consequently, resource and time are wasted, and the credibility of scientific findings are put at adventure. Furthermore, despite recent heightened awareness, there remains a pregnant need to ameliorate educate students and inquiry trainees nearly the lack of reproducibility in life science research and actions that can be taken to improve information technology. Here, we review predominant factors affecting reproducibility and outline efforts to improve the situation.

What is reproducibility?

The phrase 'lack of reproducibility' is understood in the scientific community, but information technology is a rather broad expression that incorporates several aspects. Though a standardized definition has not been fully established, the American Society for Cell Biology^® (ASCB^®) has attempted a multi-tiered approach to defining the term reproducibility by identifying the subtle differences in how the term is perceived throughout the scientific community.

ACSB⁴ has discussed these differences with the following terms: direct replication, which are efforts to reproduce a previously observed consequence by using the aforementioned experimental design and atmospheric condition as the original study; analytic replication, which aims to reproduce a series of scientific findings through a reanalysis of the original data set; systemic replication, which is an attempt to reproduce a published finding under unlike experimental conditions (east.g., in a different civilisation system or animal model); and conceptual replication, where the validity of a phenomenon is evaluated using a different set of experimental weather or methods.

It is generally thought that the improvement of direct replication and analytic replication is most readily addressed through training, policy modifications, and other interventions, while failures in systematic and conceptual replication are more hard to connect to issues with how inquiry was performed as there is more natural variability at play.

The reproducibility trouble

Many studies claim a pregnant result, just their findings cannot be reproduced. This problem has attracted increased attention in recent years, with several studies providing prove that research is often not reproducible. A 2016 Nature survey³, for instance, revealed that in the field of biology alone, over 70% of researchers were unable to reproduce the findings of other scientists and approximately lx% of researchers could not reproduce their own findings.

The lack of reproducibility in scientific enquiry has negative impacts on health, lower scientific output efficiency, slower^{half-dozen , seven} scientific progress, wasted fourth dimension and money, and erodes the public's trust in scientific research. Though many of these problems are difficult to quantify, there have been attempts to calculate fiscal losses. A 2015 meta-analysis⁵ of past studies regarding the cost of non-reproducible research estimated that $28 billion per twelvemonth is spent on preclinical enquiry that is not reproducible. Looking at avoidable waste in biomedical research on the whole, it is estimated that equally much as 85% of expenditure may exist wasted due to factors that similarly contribute to not-reproducible research such as inappropriate study design, failure to adequately address biases, non-publication of studies with disappointing results, and insufficient descriptions of interventions and methods.

Factors contributing to the lack of reproducibility

Failures of reproducibility cannot be traced to a single cause, but in that location are several categories of shortcomings that can explain many of the cases where research cannot be reproduced. Here are some of the most significant categories.

A lack of admission to methodological details, raw data, and research materials.

For scientists to be able to reproduce published piece of work, they must be able to access the original data, protocols, and key research materials. Without these, reproduction is greatly hindered and researchers are forced to reinvent the bike equally they attempt to repeat previous piece of work. The mechanisms and systems for sharing raw unpublished information and inquiry materials, such as data repositories and biorepositories, need to be made robust so that sharing is not an impediment to reproducibility.

Use of misidentified, cantankerous-contaminated, or over-passaged prison cell lines and microorganisms.

Reproducibility can be complicated and/or invalidated by biological materials that cannot be traced back to their original source, are non thoroughly authenticated, or are not properly maintained. For example, if a prison cell line is not identified correctly, or is contaminated with mycoplasma or some other cell blazon, results can be afflicted significantly and their likelihood of replication diminished. There are many cases of studies conducted with misidentified or cantankerous-contaminated cell lines, so results rendered questionable, and conclusions drawn from them are potentially invalid⁸. Improper maintenance of biological materials via long-term serial passaging can also seriously affect genotype and phenotype, which tin make reproducing data difficult. Several studies have demonstrated that serial passaging can atomic number 82 to variations in cistron expression, growth rate, spreading, and migration in prison cell lines^{ix , x}; and changes in physiology, virulence factor production, and antibiotic resistance in microorganisms^{eleven , 12 , 13}.

Disability to manage circuitous datasets

Advancements in engineering have enabled the generation of extensive, circuitous data sets; however, many researchers do non accept the noesis or tools needed for analyzing, interpreting and storing the data correctly. Further, new technologies or methodologies may not even so have established or standardized protocols, so variations and biases tin exist easily introduced, which in turn tin impact the ability to analytically replicate the data.

Poor inquiry practices and experimental design

Among the findings from scholarly efforts examining non-reproducibility is that, in a significant portion of cases, the cause could be traced to poor practices in reporting enquiry results, and poor experimental blueprint^{xiv , 15}. Poorly designed studies without a core set of experimental parameters, whose methodology is not reported clearly, are less probable to be reproducible. If a study is designed without a thorough review of existing testify, or if the efforts to minimize biases are insufficient, reproducibility becomes more than problematic.

Cognitive bias

These refer to the ways that judgement and controlling are afflicted by the individual subjective social context that each person builds effectually them. They are errors made in cognitive processes that are due to personal beliefs or perceptions. Researchers strive for impartiality and effort to avert cognitive bias, just it is often difficult to completely shut out the subtle, subconscious ways that cognitive bias can touch the conduct of inquiry^{16 , 17}. Scientists accept identified dozens of unlike types of cognitive biases, including confirmation bias, selection bias, the bandwagon outcome, cluster illusion, and reporting bias¹⁷. Confirmation bias is the unconscious act of interpreting new bear witness in ways that confirm ane's existing belief system or theories; this blazon of bias impacts how data is gathered, interpreted, and recalled. Selection bias sees researchers cull subjects or data for analysis that is not properly randomized; here, the sample obtained is not truly representative of the whole population. The bandwagon result is the tendency to hold with a position too easily, without sufficient evaluation in society to maintain group harmony; this form of bias may pb to the acceptance of unproven ideas that have gained popularity. Cluster illusion is when patterns are perceived in a pool of random data in which no actual design exists; a bias based on the tendency of the brain to seek out patterns. Reporting bias is when report participants selectively reveal or suppress data in a study according to their ain subconscious drivers; this form of bias may pb to underreporting of negative or undesirable experimental results.

A competitive culture that rewards novel findings and undervalues negative results

The academic inquiry system encourages the rapid publication of novel results. Researchers are rewarded more for publishing novel findings, and non for publishing negative results (e.g., where a correlation was non found)¹⁵. Indeed, there are limited arenas for publishing negative results, which could hone researchers' efforts and avoid repeating work that may be difficult to replicate. Overall, reproducibility in research is hindered by under-reporting of studies that yield results accounted disappointing or insignificant. University hiring and promotion criteria often emphasize publishing in high-impact journals and do not generally reward negative results. As well, a competitive surround for inquiry grants may incentivize researchers to limit reporting of details learned through feel that make experiments work better.

Recommended all-time practices

A number of meaning efforts have been aimed at addressing the lack of reproducibility in scientific research. Private researchers, periodical publishers, funding agencies, and universities accept all made substantial efforts toward identifying potential policy changes aimed at improving reproducibility^{16 , 18 , 19 , 20 , 21}. What has emerged from these efforts is a set of recommended practices and policy prescriptions that are expected to have a large impact.

Training on statistical methods and report design is essential for reproducible enquiry.

Robust sharing of data, materials, software, and other tools.

All of the raw data that underlies any published conclusions should be readily bachelor to boyfriend researchers and reviewers of the published commodity. Depositing the raw data in a publicly available database would reduce the likelihood that researchers would select merely those results that back up a prevailing attitude or confirms previous work. Such sharing would accelerate scientific discoveries, and enable scientists to interact and collaborate at a meaningful level.

Use of authenticated biomaterials

Data integrity and assay reproducibility can exist greatly improved by using authenticated, depression-passage reference materials. Cell lines and microorganisms verified by a multifaceted arroyo that confirms phenotypic and genotypic traits, and a lack of contaminants, are essential tools for inquiry. By starting a set of experiments with traceable and authenticated reference materials, and routinely evaluating biomaterials throughout the research workflow, the resulting data will be more reliable, and more likely to be reproducible.

Training on statistical methods and study blueprint

Experimental reproducibility could be considerably improved if researchers were trained how to properly structure experiments and perform statistical analyses of results. Past strictly adhering to a gear up of best practices in statistical methodology and experimental design, researchers could boost the validity and reproducibility of their work.

Pre-registration of scientific studies

If scientists pre-annals proposed scientific studies (including the approach) prior to initiation of the study, it would allow conscientious scrutiny of all parts of the inquiry process and would discourage the suppression of negative results.

Publish negative data

Many times, 'negative' data that do non back up a hypothesis typically get unpublished as they are not considered high impact or innovative. By publishing negative data, it helps to interpret positive results from related studies and tin assist researchers accommodate their experimental design so that further resource and funding are not wasted²².

Thorough clarification of methods

Information technology is of import that inquiry methodology is thoroughly described to help meliorate reproducibility. Researchers should clearly report fundamental experimental parameters, such every bit whether experiments were blinded, which standards and instruments were used, how many replicates were made, how the results were interpreted, how the statistical analysis was performed, how the randomization was done, and what criteria were used to include or exclude any data.

Ongoing efforts to improve reproducibility

At that place is a varied and influential group of organizations that are already working to better the reproducibility of scientific research. The following is a list of initiatives aimed at supporting one or more aspects of the research reproducibility issue.

American Lodge for Jail cell Biological science (ASCB) - The ASCB Report on Reproducibility

ASCB continues to place methods and all-time practices that would enhance reproducibility in basic research. From its original analysis, the ASCB job force identified and published several recommendations focused on supporting existing efforts and initiating new activities on improve training, reducing competition, sharing data, improving peer reviews, and providing prison cell authentication guidelines.

American Type Culture Collection (ATCC) - Cell and Microbial Authentication Services and Programs

Biological resource centers, such every bit ATCC, provide the research community with standardized, traceable, fully authenticated cell lines and microorganisms to assistance in analysis reproducibility. At ATCC, microbial strains are authenticated and characterized through genotypic, phenotypic, and functional analyses to ostend identity, purity, virulence, and antibiotic resistance. ATCC has likewise taken a atomic number 82 in jail cell line authentication by publishing the voluntary consensus standard, ANSI/ATCC ASN-0002: Authentication of Human Cell Lines: Standardization of STR Profiling, and by performing STR profiling on all human cell lines managed amid its holdings.

Furthermore, ATCC offers online cell line authentication training in partnership with Global Biological Standards Plant, NIH (R25GM116155-03), and Susan Yard. Komen (SPP160007), which focuses on the best practices for receiving, managing, authenticating, culturing, and preserving cell cultures. To farther support cell authentication and reproducibility in the life sciences, ATCC also provides STR profiling and mycoplasma detection testing every bit services to researchers.

National Institutes of Health (NIH) - Rigor and Reproducibility

To assist improve rigor, reproducibility, and transparency in scientific research, the NIH issued a notice in 2015 that informed scientists of revised grant application instructions that focused on improving experimental blueprint, authenticating biological and chemical resources, analyzing and interpreting results, and accurately reporting research findings. These efforts have led to the adoption of similar guidelines by journals beyond numerous scientific disciplines and has resulted in cell line authentication becoming a prerequisite for publication.

Science Commutation & the Eye for Open Science - The Reproducibility Projection: Cancer Biological science

This initiative was designed to provide evidence of reproducibility in cancer enquiry and to identify possible factors that may affect reproducibility. Hither, selected results from high-profile manufactures are independently replicated by unbiased third parties to evaluate if information could be consistently reproduced. For each evaluated study, a registered report delineating the experimental workflow is reviewed and published before experimentation is initiated; later on data collection and analysis, the results are published as a replication study.

Author Policies for Publication

Many peer-reviewed journals have updated their reporting requirements to help improve the reproducibility of published results. The Nature Inquiry journals, for example, have implemented new editorial policies that assist ensure the availability of data, key enquiry materials, computer codes and algorithms, and experimental protocols to other scientists. Researchers must at present complete an editorial policy checklist to ensure compliance with these policies before their manuscript can exist considered for review and publication.

Most people familiar with the upshot of reproducibility concur that these efforts are gaining traction. However, progress will require sustained attention on the issue, also as cooperation and interest from stakeholders across various fields.

The academic research organization encourages the rapid publication of novel results.

Moving forward

Accuracy and reproducibility are essential for fostering robust and apparent research and for promoting scientific advancement. There are predominant factors that have contributed to the lack of reproducibility in life science research. This consequence has come to light in recent years and a number of guidelines and recommendations on achieving reproducibility in the life sciences take emerged, just the practical implementation of these practices may be challenging. It is essential that the scientific community are objective when designing experiments, take responsibility for depicting their results accurately, and thoroughly and precisely depict all methodologies used. Further, funders, publishers, and policy-makers should continue to heighten awareness most the lack of reproducibility and use their position to promote meliorate research practices throughout the life sciences. Past taking action and seeking opportunities for improvement, researchers and cardinal stakeholders tin help improve research practices and the credibility of scientific data.

For more than information on how yous can improve the reproducibility of your research, visit ATCC online.

bathursttooffer.blogspot.com

Source: https://www.nature.com/articles/d42473-019-00004-y

Share this post