Abstract
Objective Timely and comprehensive reporting of clinical trial results build the backbone of evidence-based medicine and responsible research. The proportion of timely disseminated trial results can inform alternative national and international benchmarking of university medical centers (UMCs).
Study Design and Setting For all German UMCs we tracked all registered trials completed between 2009 and 2013. The results and an interactive website benchmark German UMCs regarding their performance in results dissemination.
Results We identified and tracked 2,132 clinical trials. For 1,509 trials, one of the German UMCs took the academic lead. Of these 1,509 “lead trials”, 39% published their results (mostly via journal publications) in a timely manner (<24 months after completion). More than six years after study completion, 26% of all eligible lead trials still had not disseminated results.
Conclusion Despite substantial attention from many stakeholders to the topic, there is still a strong delay or even absence of results dissemination for many trials. German UMCs have several opportunities to improve this situation. Further research should evaluate whether and how a transparent benchmarking of UMC performance in results dissemination helps to increase value and reduce waste in medical research.
Background
The results of clinical trials build the backbone of evidence-based medicine. They inform clinical decision making [1] and health technology assessment [2, 3]. They also inform decision making within ongoing trials and decision making related to the design, review, and funding of new trials [4]. Non-dissemination or delayed dissemination of trial results negatively affects all of these decisionmaking processes [5–8]. For three decades studies investigated and criticized this challenge [9–11]. Since 2008, the Declaration of Helsinki includes the requirements that every study involving human subjects should be prospectively registered and that all results should be made publicly available – irrespective of the results’ direction [12]. The joint statement by the World Health Organization in 2015 defined “timely publication” as “24 months for publication in a peer-reviewed journal (preferably open-access) and 12 months for publication of the key results in the registry’s result section” [8].
The current dissemination of clinical trial results still looks quite different. Recently, Chen et al. analysed the publication of more than 4,000 interventional clinical trials across all 51 US university medical centres (UMCs) that were completed between October 2007 and September 2010 [13]. Only 29% of trials published their results within 24 months after study completion, and only 13% of trials posted their summary results (a tabular summary of the key outcomes) on the clinical trials registry ClinicalTrials.gov. Overall, as of July 2014, 35% of all trials were found to be unpublished [13]. Schmucker et al. conducted a systematic review on similar tracking investigations with a total of 5,112 studies and found that on average 54% of studies are unreported, with a range from 24% - 74% for the individual tracking investigations [14]. A similar range was detected in the systematic review of Dwan et al. [15].
In recent years, journals [16], agencies (the Food and Drug Administration (FDA) and European Medicines Agency (EMA)) [17], ethical guidelines [18], and most recently, funding bodies [19] have all explicitly highlighted the need to reduce biased or delayed publication and developed policies to proactively achieve this objective. UMCs, in contrast, which function as trial sites and host the responsible principle investigators (PIs), have remained surprisingly silent about this issue [20, 21]. A transparent benchmarking for how complete and timely UMCs are in reporting their trial results could incentivize the implementation of more effective UMC policies in this regard. Such benchmarks could also raise public and media awareness about this issue. The “TrialsTracker” project, which includes several different trackers provides automatically updated data for benchmarking activities [22, 23]. While the FDA (https://fdaaa.trialstracker.net) and EU (http://eu.trialstracker.net) TrialsTrackers check if summary results were posted on the ClinicalTrials.gov and the EU clinical trials register (EUCTR), respectively, the original TrialsTracker (https://trialstracker.ebmdatalab.net/) additionally searched for linked results on Pubmed (last update March 2017). TrialsTracker increases its public outreach by presenting results via a publicly accessible website. The FDA and EU TrialsTrackers, however, have two limitations. First, they focus on trials that fall under mandatory reporting rules according to the FDA Amendment Act (FDAAA) or the European Commission guideline 2012/c302/03. Second, their method is restricted to the automated search of registry entries.
In this study, we further develop the concept and practice of benchmarking UMCs in three ways. First, with regard to the sample, we sampled and followed up trials (i.e. all registered interventional clinical studies) that had a completion date between 2009 and 2013 for all German UMCs. In a continuously updated map of all studies on ClinicalTrials.gov Germany is second with 17,945 trials behind France with 21,423 trials [24]. According to the advanced search of the EUCTR Germany has most trials with a EudraCT protocol (n=10,273) followed by United Kingdom (n=8,589), Spain (n=8,536), Italy (n=7,057) and France (n=4,551) [25]. Second, we extended the standard publication search strategies to comprehensive hand searches in Google Scholar. This allow a better understanding of the full picture of available results published outside registry websites and PubMed. Third, with regard to benchmarking, we developed a website, including a Shiny app, that allows the interactive visualization of benchmarking according to the different variables that influence publication measurement, such as time to publication, publication format, sponsor, timing of registration, completion date and others.
Methods
The protocol for this project, including all methodological details for sampling and following up clinical trials for data extraction, and statistical analyses was preregistered with the Open Science Framework (OSF) and continuously updated for amendments (https://osf.io/fh426/). In the following sections, we summarize the methods.
Retrieval of trials
We downloaded the AACT dataset, which aggregates information from ClinicalTrials.gov into a relational database, from http://aact.ctti-clinicaltrials.org/ (version date: April 17, 2017). We further downloaded the dataset from the German Clinical Trials Registry (DRKS) from www.drks.de on July 27, 2017. The delay in DRKS extraction was due to a slight change in the filtering criteria (see detailed methods on OSF). We used an R script to combine all relevant datasets and to extract the trial characteristics.
Inclusion and exclusion of studies
R was also used to restrict the resulting dataset to studies with a primary completion date (AACT, can be planned or actual) or study end date (DRKS, actual) in the years 2009-2013, as well as to exclude observational studies, incomplete entries (missing NCT, affiliation or primary completion date), and duplicates.
For all studies we checked if one or several of the German UMCs contributed to the trial by searching for different versions of the UMC names in the affiliation fields of the extracted dataset. Contributions of a UMC to a trial were either counted as a) “lead” contribution, where the UMC had a mention as responsible party, lead sponsor or principal investigator (AACT) or as primary sponsor (DRKS), or b) “facility” contribution, where the UMC only recruited patients as a facility or collaborator (AACT) or recruitment location (DRKS). One trial can be counted for multiple UMCs if they have different contributions to the trial. However, for the overall results reported in this study, each trial associated to several UMCs is counted only once. After automatic filtering for the UMC names, the correct assignment of trials to UMCs was verified manually.
In the following, we concentrate on the results for the lead trials only. For further results regarding the facility trials see http://s-quest.bihealth.org/intovalue/.
Only studies with the status “Completed”, “Terminated”, “Suspended”, or “Unknown” (or the equivalent DRKS categories; see detailed methods on OSF) were included. Studies from the DRKS sample that also appeared in the AACT sample were identified by searching for NCT IDs as secondary IDs in the DRKS dataset and subsequently removed.
Publication search
For each of the included studies, a results publication was searched independently by two researchers in a 3-step process between July 2017 and December 2017 (see also Figure 1, search strategy). 1) The clinical trial identifier (NCT ID or DRKS ID) was entered on ClinicalTrials.gov/DRKS.de, and the earliest result publication linked in the registry was searched and checked if it was indeed a results publication for the trial. Reviews and other background literature were excluded. 2) The clinical trial identifier was entered on PubMed. 3) Google Scholar and (if no hit was found) Web of Science were searched by subsequently entering the following search terms: clinical trial identifier, official title, brief title (if available), intervention name and principal investigator or primary sponsor. The first two results pages were screened. Publications were matched using a list of explicit criteria (i.e., study design, intervention, and outcomes). All criteria needed to be met to be counted as a match.
If, after all three searches, there was still no result, the study was characterized as “no publication found”. Additionally, the researchers checked if a summary result was posted on clinicaltrials.gov.
In some cases, we identified results resources that were not journal publications (“other” results category). We counted doctoral theses containing the trial results as results publications, but we did not count conference abstracts, posters or presentation slides.
Interrater reliability
For all studies two researchers independently conducted the publication search. Both researchers compared the independently identified results publications. If they identified different publications, we counted the earliest publication. Interrater reliability, which was defined as how often two raters had independently identified the same results publication, was at 78%. In cases with different publications we reached a 100% agreement on which is the earlier publication.
For further information on interrater reliability, data extraction, R-scripts, and statistics (logistic regression and Kaplan-Meier), see the abovementioned protocol registered with OSF (https://osf.io/fh426/).
Results
Demographic data
We identified 2,132 clinical trials via clinicaltrial.gov (n=1,905) and DRKS (n=227) that i) recruited trial participants from at least one German UMC and ii) had their primary completion date (PCD, last visit of last patient for a primary outcome measure) between 2009 and 2013. These trials included 506,876 anticipated participants.
Altogether, 71% (n=1,457) of all trials were counted as lead trial for one of the corresponding German UMCs. Of these 1,457 lead trials, 502 (35%) investigated drugs, and 266 (18%) investigated devices; the rest were “behavioural”, “procedure” or “other” interventions. Only a minority of these lead trials (n=360; 25%) were registered prospectively, and 878 (60%) were registered more than 21 days after the given start date (can be actual or planned, see supplemental Table 1 on OSF) of the trial, with 246 (17%) registered after the completion date (CD). 113 lead trials (8%) included more than 500 anticipated participants. A total of 1,054 trials (72%) were completed, and 136 (9%) were either terminated early or suspended; for 267 trials (18%), the status was unknown. For additional demographic data, see Table 1.
Overall results reporting and the Shiny app website
Because our paper cannot report on all measurement variables that were applied in our study, we developed an interactive website (based on a Shiny app) that allows users to select and combine the measurement variables in which they are most interested and develop a corresponding benchmark for all 36 German UMCs. The website is http://s-quest.bihealth.org/intovalue/. In the following sections, we report the most essential findings of our study.
Of all 1,457 lead trials, we could follow up 1,438 for a minimum of 24 months after the CD. Of those trials, 39% published their results via journal publications, summary results or dissertations within 24 months after the CD. At the level of German UMCs, this publication rate varied from 14% to 62% (Table 2). We found a steady improvement in timely publication from 35% for trials completed in 2009/2010 up to 42% for trials completed in 2012/2013. Figure 2 presents the percentage of unpublished trials over time. By April 2017, summary results were reported in the registry for 91 (7%) of 1,243 lead trials registered with clinicaltrials.gov. We further identified six dissertations, 43 abstracts (among them conference abstracts), and one presentation.
Of the 1,457 lead trials, there was a subgroup of 651 trials that we could follow up for more than six years after the CD. For this subgroup, we found an overall publication rate of 74%, with a variation across universities of 56% to 100%. Altogether, 18,305 participants were planned to be included in the 171 trials that have not published their results. Extrapolated to the full sample of lead trials (18,305 × 1,457 ÷ 651 ÷ 5 years) an average of 8,194 planned participants per year were included in trials from German UMCs that did not disseminate their results after more than six years.
All the results presented above were generated by time-intensive searches, including searches in Google Scholar and Web of Science (see Methods), that were performed independently by two researchers with training in literature searching. When restricting our search efforts to more convenient standards (registry and PubMed; see Methods), we could identify results for only 26% of trials within 24 months after CD (vs. 39% with our extensive search) and for 45% of trials followed up for more than six years (vs. 74% with our extensive search). Thus, we could identify 33% of all timely publications and 39% of all publications with a six-year follow-up period only via the additional search strategies.
Subgroup analyses
The overall publication rates (for more than six years after CD) differed substantially (more than 10%) according to the following factors:
number of participants (71% for trials with 1-100 participants but 94% for trials with >500 participants),
timing of registration (69% for prospectively registered trials but 86% for trials registered after the CD), and
trial status (43% for trials with terminated/suspended/unknown status but 83% for completed trials).
To identify subgroups with substantially different publication rates, we performed an additional exploratory logistic regression analysis (as preregistered, see supplementary file on OSF for more details). We identified the variables ‘mono-vs multicentric’ (OR 1.66, CI 1.39-1.98), ‘lead sponsor’ (industry vs academia, OR 1.67, CI 1.40-2.01), and ‘number of participants’ (OR 1.19 per 500 participants, CI 1.08-1.32) as the variables with the strongest association with publication rates. However, these associations in itself, or in combination are too weak to predict which future studies will be reported in time with great confidence.
Discussion
In this study, we demonstrate that only 39% of all registered clinical trials conducted at one of the 36 German UMCs published their results in a timely manner within 24 months after the trial’s completion date (CD). This rate further decreases to 26% when applying standard search strategies. Six years after the CD and with the most extensive search strategies, 26% of all trials still remain unpublished.
For the following reasons, this high proportion of delayed or omitted result dissemination is unethical and a substantial waste of important research resources. First, the fact that 26% of all clinical trials withhold the knowledge they gained or delay its dissemination negatively impacts i) the design of future, non-redundant translational research and ii) patient-oriented, evidence-based medical decision making. Second, every year, more than 8,000 participants on average were included in lead trials from German UMCs that did not generate any knowledge gain and thus no social value. Social value, however, is the basic ethical principle justifying research that adds burdens and risks to participants. Moreover, most trial participants are patients who already suffer from a disease. Third, administrative efforts to report summary results in the tabular format required by ClinicalTrials.gov are minimal, and this type of results reporting does not prevent more detailed and contextualized result publications in peer-reviewed journals [26]. Despite the ethical rationale only 7% (n=91) of clinical trials conducted at German UMCs (“lead trials”) reported their summary results in clinicaltrials.gov. The recently published EU TrialsTracker that evaluated the compliance with summary results reporting in the EUCTR) confirmed these low reporting rates [22].
In contrast to most other trial tracking activities our search strategy included additional hand searches in Google Scholar that identified many publications that were not indexed at clinicaltrials.gov or PubMed. This step is critical to increase the number of identified publications. Unfortunately, this time-intensive approach only allows static analysis. If, in the future, results information provided at the registry are more complete (as either summary results or linked journal publications), automated tracking could fully take over.
German UMCs have many unique possibilities to improve the current situation. First, UMC staff that coordinates clinical trials could remind and support PIs in timely reporting. UK universities are currently leading the way in this regard [27]. A second option would be to reward those PIs who manage to publish their results in a timely manner and/or report summary results in the registry. At German UMCs, the performance-oriented allocation of funds (“LOM/Leistungsorientierte Mittelvergabe”) currently only rewards aggregated impact factors and third-party funding. A third and harsher option would be to sanction those PIs who do not manage to report at least summary results in the registry within 24 months after CD. A recent update of the Wellcome Trust funding policy for clinical trials demonstrated that funders at least might decide to go this way [28].
While we applied extensive automated methods and hand searches to track each registered trial by two independent researchers our study has still several limitations. Despite the fact that we identified higher publication rates than all former tracking studies our results might still underestimate the true publication rates because we did not search in scientific databases other than PubMed and Web of Science, and we did not contact the responsible parties. Our results might also overestimate the true publications rates for several reasons. First, most included trials (60%) were retrospectively registered. These trials had substantially higher publication rates, which might reflect a registration and reporting bias. Furthermore, we did not include observational clinical studies in our sample. Former tracking studies that sampled at the level of German institutional review boards (IRBs) reported substantially lower publication rates for observational studies [29]. Additionally, our results might overestimate the time to publication because we stopped searching for further publications once we found the first results publication, possibly missing earlier results publications. We also rely on the registry entries being correct. Entries in the DRKS database might be duplicates of entries in clinicaltrials.gov without an appropriate cross-referencing of the NCT ID. Completion dates are entered as expected dates and might not always reflect the actual completion dates. As registries are increasingly consulted as a key resource for health care information and for quality assessment of research practices it is important to improve their quality, accuracy and timeliness [30]. We published a more detailed commentary on how different measurement variables influence the assessment of publication rates elsewhere [31].
We want to highlight that our study only assessed the extent of results reporting. We did not assess the reporting quality e.g. adherence to CONSORT [32]. We also did not assess whether identified papers reported all outcomes as specified in the registered protocol [1]. Finally, we did not compare whether results reported in published papers were consistent with results reported at the registry website [33, 34]. For judgments on the overall quality of clinical trial results reporting these other perspectives should be acknowledged as well.
In summary, the steady improvement in timely publication (within 24 months after CD) over time is promising. In contrast, the very low proportion of trials (7%) that report summary results in the registry is alarming, as most trials thus forego an important opportunity to increase their scientific and social value. Additionally, more recent trials might get published in a timely manner, but old trials still have relevant information that remains unavailable and unused. These results, which are both promising and alarming, should encourage German UMCs and other stakeholders, such as patient and funding organizations, to further improve their efforts and develop policies for the timely publication of trial results for future trials, as well as already finished yet unpublished trials. The publicly available Shiny app (http://s-quest.bihealth.org/intovalue/) might further be used to raise awareness about this element of good scientific practice in the scientific community and in the public.
Competing interest statement
All authors are affiliated with a German UMC in Berlin, Hannover or Freiburg. No further conflicts of interest exist.
Contributorship statement
DS, JM and UD designed the study. SW, NR, KW, HK, SO, CS, SK, UK and BS performed the search. NR, SK and BS performed the statistical analysis. All authors were involved in writing and editing the manuscript.
Funding
Intramural funding. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Software availability
The R script used to as part of this study is available from https://doi.org/10.17605/OSF.IO/FH426 Archived source code at time of publication: https://doi.org/10.17605/OSF.IO/FH426
Licence: MIT License
Data availability
All data underlying the results are available as part of the article and at OSF: Dataset 1: IntoValue. https://doi.org/10.17605/OSF.IO/FH426 15
The data is available under a CC0 Licence. No additional source data are required.
Footnotes
Title and abstract have been changed according to journal requirements. Methods and results have been adapted to improve clarity.