SEMINAR
Introduction to Difference-in-differences Design
2021 年 3 巻 3 号 p. 74-77
詳細
2021 年 3 巻 3 号 p. 74-77
Because it is difficult to conduct randomized controlled trials, observational studies are often used when evaluating the effects of health care policies. However, observational studies are subject to bias, such as a failure to eliminate the effects of trends in the outcome over time and permanent differences between treatment and control groups. The difference-in-differences design removes these biases by observing outcomes for the two groups at two time points. This article introduces the methods and assumptions for the difference-in-differences design and provides some examples of studies that have used this design.
Because it is difficult to conduct randomized controlled trials, observational studies are often used when evaluating the effects of health care policies. In this context, two study designs are considered: the before-and-after design and the cohort study design. The before-and-after design compares a group before the implementation of an intervention with another group after the implementation of the intervention to estimate the effect of the intervention. However, in the before-and-after design, it is not possible to determine whether the estimated effect is an actual effect of the intervention or a trend in the outcome over time [1, 2]. For example, when using a before-and-after design to examine the effects of a policy aimed at preventing passive smoking, it is not possible to eliminate the effects of a decreasing trend in the number of people smoking in public places that began before the policy was implemented.
Conversely, cohort studies compare a group in a specific area where an intervention has been implemented with another group in a different area where the intervention has not been implemented. However, effects estimated in cohort studies are often biased toward permanent differences between the treatment and control groups. For example, lung cancer may be consistently more prevalent in areas that have implemented the intervention than in areas that have not, but the cohort study design is not equipped to eliminate the effects of this permanent difference.
The difference-in-differences design removes the biases mentioned above by observing outcomes for two groups at two time points. This article introduces the methods and assumptions for difference-in-differences design and provides some examples of studies that have used this design.
The difference-in-differences design sets up a control group that has the same trend over time as the intervention group but that is not affected by the intervention [3]. The intervention group is exposed to the intervention at the second time point but not at the first time point, whereas the control group is not exposed to the intervention at either time point. The intervention effects are estimated by subtracting the effects in the control group from the effects in the treatment group. In this way, the difference-in-differences design is able to eliminate the influences of trends and permanent differences between the two groups and correctly evaluate the effect of the intervention.
Fig. 1 shows a conceptual diagram of difference-in-differences analysis. In the control group, the outcome changes from C1 to C2; in the intervention group, the outcome changes from T1 to T2. The intervention group has the same trend as the control group, and, if there were no intervention in the intervention group, the outcome in the intervention group at the second time point would be T2'. Therefore, the effect of the intervention is T2 − T2'. Difference 1, the change in the outcome before and after the intervention in the control group, is expressed as “C2 − C1.” Difference 2, the change in outcome for the intervention group over the same period, is expressed as “T2 − T1.” Because the intervention and control groups have the same trend, “C2 − C1” and “T2' − T1” are equal. Therefore, the effect of the intervention is expressed as Difference 2 − Difference 1 = (T2 − T1) − (C2 − C1) = (T2 − T1) − (T2' −T1) = T2 − T2'. Because we are taking the difference between Difference 2 and Difference 1, this design is called “difference-in-differences.”
T1and T2 indicate the outcomes in the intervention group before and after the intervention. C1 and C2 indicate the outcomes in the control group before and after the intervention. If the trends in the intervention and control groups are parallel, the post-intervention outcome in the intervention group would be T2' if this group did not receive the intervention. Therefore, the effect of the intervention is estimated as T2 − T2'.
Regression models are commonly used to estimate the effect of an intervention [2, 3]. Three variables are included in the basic difference-in-differences model: (i) the group assignment variable (presence or absence of the intervention), (ii) time point (before or after the intervention), and (iii) the interaction term of these two factors (multiplication of the two factors). The interaction term is the difference-in-differences estimator.
The equation of the difference-in-differences regression model is
Y = α + β1 * (intervention) + β2 * (time point) + β3 * (intervention) * (time point),
where Y is the outcome of interest, intervention is a dummy variable for group assignment (intervention group = 1, control group = 0), and time point is a dummy variable for the time point (post-intervention = 1, pre-intervention = 0).
Table 1 shows how to estimate the intervention effect from the regression equation. The outcome before the intervention in the intervention group is α + β1, substituting 1 for intervention and 0 for time point in the above regression equation. The post-intervention outcome for the intervention group is α + β1 + β2 + β3, substituting 1 for intervention and for time point. Therefore, the change in the outcome of the intervention group is (α + β1 + β2 + β3) − (α + β1) = β2 + β3. Similarly, the change in the outcome of the control group before and after the intervention is calculated as (α + β2) − α = β2. The difference between these two outcomes, (β2 + β3) − β2 = β3, which is the coefficient of the interaction term, shows the intervention effect.
| Before introduction of the intervention (time point = 0) | After introduction of the intervention (time point = 1) | After–Before | |
|---|---|---|---|
| The intervention group (intervention = 1) | α + β1 | α + β1 + β2 + β3 | β2 + β3 (=difference 2) |
| The control group (intervention = 0) | α | α + β2 | β2 (=difference 1) |
| Intervention-control | β1 | β1 + β3 | β3 (=difference-in-differences) |
The parallel trends assumption is that the change in the outcome over time is similar in the intervention and control groups during the observation period [1–3]. If there are parallel trends in the changes in outcomes for the intervention and control groups, the difference-in-differences design correctly estimates the intervention effect. Although we cannot see the change in the outcome for the intervention group in the absence of the intervention, if the trends of the two groups are parallel before the intervention, it makes sense to assume that the trends will be parallel after the intervention.
To check whether this assumption holds, a comparison is often made by plotting the trends of the two groups before the intervention. Fig. 2 shows schematic diagrams of trends before the intervention. If the pre-intervention trends are parallel, we can assume that the post-intervention trends are also parallel. Conversely, if the pre-intervention trends are not parallel, the difference-in-differences design cannot correctly estimate the effect of the intervention. Another option to check the parallel trends assumption is to test whether the pre-intervention trends are statistically different between the intervention and control groups.
The left panel shows parallel pre-intervention trends. In this case, the effect of the intervention can be correctly estimated. The right panel shows a violation of the parallel trends assumption. If this assumption is violated, the effect estimated by difference-in-differences analysis is invalid.
A shock is an unexpected event that is unrelated to the intervention. According to the common shocks assumption, such an event occurring at the same time as or following the implementation of the intervention has the same impact on outcomes in both the intervention group and the control group [1, 4]. If the assumption does not hold, the effect of the intervention cannot be estimated correctly using the difference-in-differences design. Unfortunately, this assumption cannot be verified from the data. When interpreting research using a difference-in-differences design, attention should be paid to any events occurring during the observation period.
This section presents examples of studies using the difference-in-differences design.
1. “Impact of Centralizing Acute Stroke Services in English Metropolitan Areas on Mortality and Length of Hospital Stay: Difference-in-Differences Analysis” [5]In 2010, acute stroke services were centralized in London and Greater Manchester in England. This study investigated whether the centralization of acute stroke services reduced 90-day mortality and length of hospital stay, using a difference-in-differences design.
This study used the Hospital Episode Statistics database, which contains inpatient data from treatment centers funded by the National Health Service. Centralization of acute stroke care occurred in April 2010 in Greater Manchester and in July 2010 in London. The study participants were inpatients with stroke living in urban areas from January 2008 to March 2012. The intervention group comprised patients with stroke living in London or Greater Manchester, and the control group comprised those living in other areas.
The regression model in this study was
Y = α + β1 * (Area) + β2 * (time point) + β3 * (Area) * (time point),
where Y is the outcome of interest, Area is a dummy variable for residence in the two areas where stroke services were centralized (Area = 1 for London or Greater Manchester, Area = 0 for other areas), time point is a dummy variable for the time point (time point = 0 for before the centralization, time point = 1 for after the centralization), and β3 is a difference-in-differences estimator.
The comparison of pre-intervention mortality and length of stay between the two centralized areas and other areas showed no significant difference, which means the parallel trends assumption was not violated.
The results showed that 90-day mortality was significantly decreased in London (−1.1%, 95% confidence interval: −2.1 to −0.1) but not in Greater Manchester (0.1%, 95% confidence interval: −1.1 to 1.3). Length of hospital stay was significantly decreased in both areas (−1.4 days, 95% confidence interval: −2.3 to −0.5 in London; −2.0 days, 95% confidence interval: −2.8 to −1.2 in Greater Manchester).
2. “Timing of Antibiotic Prophylaxis in Cesarean Section: Retrospective, Difference-in-Differences Estimation of the Effect on Surgical Site Infection” [6]This study used a difference-in-differences design to determine whether changing the timing of prophylactic antimicrobial administration for cesarean section from immediately after cord ligation to preoperative administration reduced the incidence of surgical site infection.
This study was a single-center report from the facility with the highest number of deliveries in Israel, observed over four years from January 1, 2012, to December 31, 2015. Before January 2014, prophylactic antimicrobial agents were administered immediately after cord ligation for cesarean sections at this facility. In January 2014, the policy was changed to administering the agents immediately before the start of surgery. The control group comprised patients who underwent total hysterectomy at the same institution. Preoperative administration of prophylactic antimicrobial agents for total hysterectomy was recommended throughout the study period.
The regression model in this study was
SSI = α + β1 * (CS) + β2 * (time point) + β3 * (CS) * (time point),
where SSI is surgical site infection, CS is a dummy variable for cesarean section (CS = 1 for cesarean section, CS = 0 for total hysterectomy), time point is a dummy variable for the time point (time point = 0 for prior to January 2014, time point = 1 for January 2014 or later), and β3 is a difference-in-differences estimator.
The results showed no statistically significant difference in the risk of surgical site infection after the change in policy in January 2014: −0.6% (p = 0.663).
It is unclear whether the two assumptions of the difference-in-differences design were met in this study because there is no description of either parallel trends or common shocks.
Before-and-after studies and cohort studies are subject to bias, such as the inability to identify the effects of trends over time and permanent differences between groups. Difference-in-differences analysis eliminates the effect of trends over time and permanent differences between the groups. Two critical assumptions in the difference-in-differences design involve parallel trends and common shocks.
None
This work was supported by the Ministry of Education, Culture, Sports, Science and Technology (Grant Numbers 19K19394, 21H03159).
