SEMINAR
Propensity score analysis revisited
2025 Volume 7 Issue 3 Pages 99-104
Details
2025 Volume 7 Issue 3 Pages 99-104
Propensity score (PS) is the probability of the exposure being assigned, conditional on the observed baseline covariates. Many observational studies have used PS analyses to investigate the effects of exposure on outcomes. This report reviews the five steps of PS analyses: 1) calculating PS; 2) checking the overlap of PS; 3) implementing a matching or weighting method including PS matching, inverse-probability-of-treatment weighting, standardized mortality ratio weighting, matching weighting, and overlap weighting; 4) diagnosing the covariate balance; and 5) comparing the outcomes. Two groups are often compared in PS analyses; however, three-group comparisons can provide clinicians with more benefits in many situations in routine clinical practice. Thus, we describe not only two-group comparisons but also three-group comparisons by introducing a few studies that used generalized PS to compare three groups.
Propensity score (PS) is the probability of assignment to one exposure, conditional on the observed baseline covariates1). PS analysis is a common technique for removing confounding bias in observational studies. It is generally used when comparing two exposure groups, such as treatments A and B. However, certain situations call for the comparison of more than two groups, such as treatment A vs. treatment B vs. treatment C. We first review how to compare two groups using PS analyses and then how the methods can extend to three or more groups.
Three assumptions are needed to determine the causal effect of exposures on outcomes: conditional exchangeability, positivity, and consistency2). Conditional exchangeability implies that, within a level of confounders, all other covariates are equally distributed in the exposed and unexposed groups, which corresponds to the absence of unmeasured confounders3). Positivity implies that at least one patient is needed in both the exposed and unexposed groups in all the strata divided by confounders. Consistency implies that the exposure should be defined sufficiently based on expert knowledge so that different variants of the exposure do not have different effects on the outcome3). These three assumptions must be satisfied before beginning PS analyses.
Four methods are used for PS analyses: matching, weighting, stratification, and covariate adjustment. Matching and weighting are the focus of this report because they are the most commonly used methods. The steps of PS analysis are outlined as follows.
1. Calculate PS
2. Check the overlap of the PS distribution
3. Implement a matching or weighting method
4. Diagnose the balance of covariates between the two groups
5. Compare the outcomes
The first step is to calculate PS. PS is generally calculated using logistic regression models and sometimes other machine learning algorithms4). Covariates incorporated into the model should be determined using direct acyclic graphs based on expert knowledge3). Confounders, which are defined as common causes of the exposure and outcome, as well as the variables that affect the outcome without affecting the exposure should be included (Fig. 1)5). Mediators, which exist between the exposure and outcome, as well as the variables that affect only the exposure but not the outcome, should not be included in the model because they will increase the variance or bias3,5).
Confounders are common causes of the exposure and outcome. Mediators are affected by the exposure and affect the outcome. L1 is a cause of the outcome, but not a cause of the exposure. L2 is a cause of the exposure, but not a cause of the outcome. Confounders and L1 should be included in the model to calculate the propensity score, but mediators and L2 should not.
The second step is to check the overlap in the PS distribution. The calculated PS of the exposed and unexposed groups should overlap. High overlap indicates that the two groups are comparable (Fig. 2), whereas low overlap indicates the necessity of reconsidering the research question.
The third step involves implementing a matching or weighting method. Inverse-probability-of-treatment (IPT) weighting is used when the main outcome of interest is the average treatment effect, where we assume that the exposure of interest can be applied to all patients in the study6). The weights for the exposed are calculated as 1/PS and those for the unexposed as 1/(1-PS)6). Standardized mortality ratio weighting is used when the main interest is the average treatment effect on the treated, where we assume that the patients in the unexposed group would never be exposed because certain characteristics of the patients in the exposed group are different from those in the unexposed group6). The weights for the exposed are calculated as 1 and those for the unexposed as PS/(1-PS)6). The IPT and standardized mortality ratio weights will be trimmed or truncated to decrease the bias and variance if they become very large7). The 1st and 99th percentiles of the weights are often used as the cut-off value, but they are arbitrary7). Stabilized IPT weighting is another solution to mitigate the effect of extreme weights7). Stabilized IPT weights are calculated as p/PS for the exposed and (1-p)/(1-PS) for the unexposed (p: the proportion of the exposure in all patients)7).
PS matching (not weighting) is performed when the main interest is the average treatment effect in a subset with clinical equipoise, where patients have similar characteristics with equal possibility of being assigned to either group8). One control is matched to one exposed patient based on their PS values in many studies, but 1:n (e.g., n = 2) matching can also be used to increase precision9). Whether the controls should be replaced or not, the method for searching the controls (greedy nearest neighbor matching or optimal matching), and the caliper distance (e.g., 0.2 of the standard deviation of the logit of the PS obtained) have to be determined as well1).
The matching weighting10) and overlap weighting11) are other methods that can be used when the main interest is the average treatment effect in a subset with clinical equipoise10,11). Overlap weighting has been reported to be more efficient than matching weighting and guarantees exact balance between exposure groups for all covariates12). This might be one reason why overlap weighting has gained popularity in recent years (Fig. 3). The matching weights are calculated as the minimum of PS or 1-PS divided by PS for the exposed and the minimum of PS or 1-PS divided by 1-PS for the unexposed10). The overlap weights are calculated as 1-PS for the exposed and PS for the unexposed11). Matching weighting and overlap weighting are better than PS matching in terms of computational burden, although they are asymptotically equivalent to PS matching10,11,13). For instance, two-group PS matching with 4000 patients per group generate 16 million possible match configurations. Another problem with PS matching is that some patients are excluded, resulting in decreased precision owing to a decrease in sample size. Matching weighting and overlap weighting are better than IPT and standardized mortality ratio weighting because their weights range between 0 and 1 without generating extreme weights10,11). They do not need trimming or truncation, whereby some patients are excluded based on an arbitrarily chosen cut-off, leading to variance in the population analyzed from study to study and decreased precision due to a decrease in sample size11). In matching weighting and overlap weighting, patients with PS close to 0 or 1 contribute little to the weighted samples, while those with PS close to 0.5, who could be in either group, contribute largely10,11). Thus, the weighted samples can focus on the patients at clinical equipoise without discarding the outliers.
Studies that used matching weighting were searched using the query: (“matching weight” [Title/Abstract] OR “matching weighting” [Title/Abstract] OR “matching-weighted” [Title/Abstract] OR “matching weighted” [Title/Abstract]) AND propensity. Studies that used overlap weighting were searched using the query: (“overlap weight” [Title/Abstract] OR “overlap weighting” [Title/Abstract] OR “overlap-weighted” [Title/Abstract] OR “overlap weighted” [Title/Abstract]) AND propensity. The searches were performed on 7 January 2025.
The fourth step is to diagnose the balance of covariates between the two groups. When determining the causal inference, covariate balance between exposure groups should be prioritized over generalizability11). Standardized difference is often used to check the balance1). It can be calculated by dividing the difference in the average of a covariate between the exposed and unexposed groups by the pooled standard deviation1). The standardized difference d for continuous variables is calculated using the formula:
where
where
The fifth step involves the comparison of the outcomes of the exposed and unexposed groups. Linear models can be used to compare continuous variables such as blood pressure, and logistic models can be used to compare binary variables such as death. Robust variance should be used to calculate the confidence interval when weighting has been performed15). Cluster-robust variance with pair membership as the clustering variable should be used to calculate the confidence interval when matching without replacement has been performed16,17).
The need for comparing three exposure groups may arise in different situations during routine clinical practice (e.g., treatment A, B, and C). In these situations, researchers must decide whether to perform pairwise contrasts (treatment A vs. B, treatment A vs. C, and treatment B vs. C) or contrast the three groups simultaneously (treatment A vs. B vs. C). The latter method is clinically meaningful because it focuses on the subpopulation of patients who could be assigned to any of the three groups18). This estimate would be like a three-arm randomized controlled trial in which patients are in equipoise at the randomization stage18).
The overall flow of PS analyses with three groups simultaneously is almost the same as that with two groups. First, PS is calculated using multinomial logistic regression. This score is referred to as the generalized PS13,18,19). Second, the covariate balance is evaluated using the standardized difference. Third, a matching or weighting method is chosen. PS matching for three groups can be implemented using the method proposed by Rassen et al.18). The goal of this method is to perform optimized matching without replacement by finding trios of patients (one receiving treatment A, B, and C) with the smallest within-trio distance18). This method has less bias than the pairwise approach but similar point estimates18). Matching weighting and overlap weighting can be easily extended to three groups and are referred to as generalized matching weight and generalized overlap weight13,19). The generalised matching weight is calculated as the smallest PS divided by the PS of the assigned exposure13). Suppose a patient actually received treatment A with a generalised PS of 0.6 for treatment A, 0.3 for treatment B and 0.1 for treatment C (the sum of the generalised PS should be 1). The matching weight for this patient would be
The aforementioned methods have been used to compare three groups in real-world settings. In a previous study, generalized PS matching was applied to the Medicare beneficiary database to investigate the effectiveness and safety of oral anticoagulants in patients with atrial fibrillation and multiple chronic conditions20). Three exposure groups were included: dabigatran, rivaroxaban and warfarin. Standardized difference was used to check the covariate balance after matching. Cox proportional hazards regression models were used to compare the mortality rates. It was found that the dabigatran and rivaroxaban groups had significantly lower mortality rates than the warfarin group, with hazard ratios of 0.52–0.84 according to comorbidity levels.
The matching weighting was applied to the Diagnosis Procedure Combination database, a nationwide inpatient database in Japan, to investigate the prognosis of patients with spontaneous intracerebral haemorrhage21). Three exposure groups were included: craniotomy, endoscopic evacuation, and stereotactic aspiration. Standardized difference was used to check the covariate balance after matching weighting. Multinomial logistic regression models were used to compare poor prognosis represented by a modified Rankin Scale score of 5–6 at discharge. The craniotomy group displayed a significantly higher risk of poor prognosis than the endoscopic evacuation group with an odds ratio of 2.5 (95% confidence interval, 1.1 to 5.7), while the endoscopic evacuation and stereotactic aspiration groups showed no significant difference.
The overlap weighting was applied to the Diagnosis Procedure Combination database to investigate the effectiveness and safety of multimodal non-opioid analgesia in hip fractures22). The study included three exposure groups: non-steroidal anti-inflammatory drugs alone, acetaminophen alone, and combination of non-steroidal anti-inflammatory drugs and acetaminophen. Standardized difference was used to check the covariate balance after overlap weighting. Logistic and linear regression models were used to compare in-hospital mortality and complications. The mortality rates were similar among the three groups, while the combination group had a significantly lower risk of complications than the acetaminophen group with a risk difference of −1.9% (95% confidence interval, −3.2% to −0.6%).
We reviewed the PS analyses for two- and three-group comparisons. We recommend that researchers follow five steps: 1) calculate the PS, 2) check the overlap in PS distribution, 3) implement a matching or weighting method, 4) diagnose covariate balance between groups, and 5) compare outcomes. Considering the appearance of new therapeutic options, patients may have more than two treatment options in many situations. Both two- and three-way PS analyses will become increasingly essential for making causal inferences.
No conflicting relationship exists for any author.
None.
None.
Research Design: YH and HY.
Data Acquisition and/or Research Execution: YH and HY.
Data Analysis and/or Interpretation: YH.
Manuscript Preparation: YH and HY.
Hideo Yasunaga is one of the Editorial Board members of Annals of Clinical Epidemiology. This author was not involved in the peer-review or decision-making process for this paper.
