Non-Statistical Key Issues in Conducting Sensible Observational Studies to Resolve Clinical Questions

“Life Is Not Complex. We Are Complex. Life Is Simple, and the Simple Thing Is the Right Thing.” Oscar Wilde

Life is full of questions. Clinical practice is full of clinical questions. Physicians have many questions come across their minds when seeing patients ie 3 questions for every 10 patients they see,¹ although most questions, unfortunately, are left unanswered. To resolve such questions as quickly as possible, the best thing for us is to appropriately conduct clinical research. Some physicians think that observational studies are easier to conduct than randomized controlled trials (RCTs). This is true in some aspects but in other aspects, completely wrong. We have to bear in mind some key issues, which does not necessarily mean complicated multivariate analysis, for conducting sensible observational studies.

Article p 2225

In this issue of the Journal, Minakata et al report the possible association of impaired renal function with poor outcome in patients after coronary artery bypass grafting.² I would like to emphasize that their success may be attributed to the registry of patients well constructed by the clinical questions they had, the patients and outcome they defined, and the variables they selected. Clinical studies based on registries have been increasingly published recently.^3,4 The key issues for registry-based observational studies are discussed in this editorial review.

Purpose of the Study and Clinical Questions

First of all, the most important issue in conducting any clinical research is to have a clear purpose; that is, sensible clinical questions from clinical practice. The quality of clinical research largely depends on the quality of the clinical questions and subsequent research hypotheses. There are roughly 2 types of clinical research (Table). Seeds-driven research examines the efficacy of new drugs for approval by regulatory agencies, whereas needs-driven research is intended to resolve clinical questions. The former needs a very strict study design. Efficacy of drugs should usually be tested in double-blind, RCTs with restrictive criteria for eligible patients and endpoints under the strict regulation and guidance (ICH-GCP) with few exceptions. Observational study design may fit the latter but RCTs also are applicable as pragmatic trials with less restrictive design. Pragmatic RCTs may also be fit for comparisons of strategies of care. For example, intensive control of cardiovascular risk factors such as blood pressure is better to be compared to standard control by a RCT rather than in a cohort study. Therefore, study design should be determined by the purpose of study not by a hierarchical “pyramid of evidence”.

Table. Types of Research

Seeds-driven research

· Evaluation of efficacy of new drugs for approval

· Principally double-blind randomized controlled trials

· Rigid study design

· Restrictive inclusion and exclusion criteria for maximization of efficacy and minimization of adverse event risk

Needs-driven research

· Clinical question based

· Comparison of effectiveness of treatments outside of clinical trial settings

· Comparison of strategies of care

· Evaluation of association between variables and outcomes

· Observational studies and pragmatic randomized controlled trials

· Less restrictive inclusion and exclusion criteria for representatives of clinical practice

Definition of Patients and Outcomes

The target population on which the researchers will focus should be defined clearly according to the purpose of the study. As observational studies usually need a large number of typical clinical practice populations for sufficient statistical power and generalizable results, inclusion and exclusion criteria should be clear, simple but much less restrictive than in a RCT testing the efficacy of new drugs in similar patients. For example, the RELY trial was a phase III trial that the examined efficacy and safety of dabigatran for approval and the exclusion criteria of RELY trial consisted of more than 20 conditions,⁵ whereas the cohort study comparing warfarin and dabigatran by FDA sentinel project had only 3 disease-related conditions as exclusion criteria.⁶ Selected patient subgroups can also be predefined according to clinical questions but the feasibility of dividing patients into subgroups should be assessed. Outcomes are another part of clinical question. Outcomes in observational studies should be more (or equally) objective and severer than those in RCTs. When patients with atherosclerotic cardiovascular diseases are focused on in observational studies, outcomes may be death, nonfatal myocardial infarction and nonfatal stroke, whereas double-blind RCTs are allowed to assess less objective and milder endpoints such as worsening of heart failure or angina. Outcomes in observational studies should also be easier to diagnose than in RCTs. Most diagnostic criteria in clinical trials appear to be unfriendly and cannot be translated for use in large observational studies like as inclusion and exclusion criteria. Improvement of feasibility, usually at lower cost than in pharmaceutical trials, at the expense of the precision of diagnosis may be acceptable traded off. A debate still lives on in terms of accuracy of case-specific mortality even in clinical trials.^7,8 Because fewer researchers contribute to observational studies than clinical trials and reporting cases with selected information carries unavoidable and unadjustable biases, intensive laboring to determine the cause of death in many cases, which may be more complicated than those in RCTs, may be impractical.

Variables

Because the aim of observational studies may principally be seeking a possible association of target variables with outcomes and confounding is a key threat to the validity of results, logical selection and definition of target variables (independent variables to be tested) and variables confounding results (confounders or adjusters) in accordance with the purpose of the study is necessary.

Registration of Patients and Collection of Their Information

Once the patient group is well defined, patients must be consecutively registered. Either intentional or unintentional exclusion of eligible patients causes a selection bias. Missing patients are usually not missing at random. Logically excluded patients from the registry should also be recorded and reported precisely with the reason of exclusion for validity of the summarized data and sensitivity analysis. An advantage of observational studies is inclusion of a population representative of clinical practice, so exclusion of patients may reduce this advantage. Recently developed data storage systems may help consecutive registration of patients through automatic data extraction systems. Care is needed, however, because coded diagnosis is not necessarily correct and therefore adequate validation of extraction system is absolutely required.

Registration of 3,000 patients receiving newly approved drugs as typical post-marketing surveillance in Japan has little value in terms of assessment of safety and efficacy because of intentional selection of patients, lack of comparators, and sometimes forced switching from drugs competing new drugs without any sensible clinical reasons. Such studies should not be regarded as proper observational studies but just seeding trials only for promotion of new drugs.

Relevant variables and outcomes also should be collected with similar caution. Reliability of results from prospective cohort studies may depend on completeness of follow-up. As mentioned before, missing variables and outcomes, which are not usually missing at random, may cause biases. Researchers, hopefully with biostatisticians and research coordinators, are advised to discuss which and how many variables and outcomes should be collected. Standard operating procedures for data collection and data management at participating sites and central data centers should be established. Although intensive monitoring of data, such as source document verification, done in the same way as pharmaceutical trials is difficult in observational cohort studies, central quality control of collected data at data centers by a biostatistician may improve the accuracy of results.

Registries of Patients as Platforms for Any Clinical Research (Figure)

Appropriately constructed registries of patients can be platforms for any clinical research. Prospective or even retrospective collection of well-defined outcomes and variables may allow researchers to conduct sensible cohort studies, case-control studies and cross-sectional studies based on one registry. Proper data management and central statistical monitoring of registries by biostatisticians may improve the quality of data at lower cost. From this point of view, registries of patients can also be platforms for RCTs. In fact, the Thrombus Aspiration in ST-Elevation Myocardial Infarction in Scandinavia (TASTE) trial investigators recently successfully conducted a large, clinical question-based RCT at very low cost based on the platform of a well-constructed registry of patients.⁹ Unlike pharmaceutical trials, researchers in academic trials are haunted by concerns about cost, enrolment of patients and quality control. Registry-based RCTs as well as observational studies may help researchers overcome such obstacles.

Figure.

Well-constructed registrations of patients can be used as platforms for several types of clinical research.

Disclosures

None.

References

• 1.    Ely  JW,  Osheroff  JA,  Ebell  MH,  Bergus  GR,  Levy  BT,  Chambliss  ML, et al. Analysis of questions asked by family doctors regarding patient care. BMJ 1999; 319: 358–361.
• 2.    Minakata  K,  Bando  K,  Tanaka  S,  Takanashi  S,  Konishi  H,  Miyamoto  Y, et al. Preoperative chronic kidney dysfunction as a strong predictor for postoperative infection and mortality after coronary artery bypass grafting. Circ J 2014; 78: 2225–2231.
• 3.    Daida  H,  Miyauchi  K,  Ogawa  H,  Yokoi  H,  Matsumoto  M,  Kitakaze  M, et al; PACIFIC investigators. Management and two-year long-term clinical outcome of acute coronary syndrome in Japan: Prevention of atherothrombotic incidents following ischemic coronary attack (PACIFIC) registry. Circ J 2013; 77: 934–943.
• 4.   Japanese Circulation Society Resuscitation Science Study Group. Chest-compression-only bystander cardiopulmonary resuscitation in the 30:2 compression-to-ventilation ratio era: Nationwide observational study. Circ J 2013; 77: 2742–2750.
• 5.    Ezekowitz  MD,  Connolly  S,  Parekh  A,  Reilly  PA,  Varrone  J,  Wang  S, et al. Rationale and design of RE-LY: Randomized evaluation of long-term anticoagulant therapy, warfarin, compared with dabigatran. Am Heart J 2009; 157: 805–810.
• 6.   The U.S. Food and Drug Administration (FDA). A Protocol for Assessment of Dabigatran and Selected Safety Outcomes. http://mini-sentinel.org/assessments/medical_events/details.aspx?ID=219 (accessed July 25, 2014).
• 7.    Gottlieb  SS. Dead is dead: Artificial definitions are no substitute. Lancet 1997; 349: 662–663.
• 8.    Hauptman  PJ. Does it matter why and how patients with heart failure die?: A debate that lives on. Circ Heart Fail 2008; 1: 89–90.
• 9.    Fröbert  O,  Lagerqvist  B,  Olivecrona  GK,  Omerovic  E,  Gudnason  T,  Maeng  M, et al. Thrombus aspiration during ST-segment elevation myocardial infarction. N Engl J Med 2013; 369: 1587–1597.