Predicting compliance with G20 commitments
Statistical models can be used to predict how G20 members will perform on the commitments they’ve made – and the results can be used to improve the ability of the G20 to achieve its goals
Since the first G20 summit in 2008, leaders have made 3,032 collective, precise, future-oriented and politically binding commitments to address pivotal issues such as global health, financial stability and economic development. Historically, however, of the commitments assessed for compliance by the G20 Research Group, only 52% of these commitments have been complied with in full.
Research has shown that the underlying factors shaping whether a member will comply with a commitment relate primarily to immutable properties such as its economic position or past compliance with similar commitments. These factors cannot contribute to improving the probability of compliance, since they cannot be changed in the short term. However, a model to predict compliance might help identify which members are the least likely to meet their G20 commitments – enabling resources to be leveraged to assist those members fulfil their obligations. This could improve the overall effectiveness of the G20.
Factors that affect G20 compliance
Data on commitment outcomes collected by the G20 Research Group make it possible to build a quantitative prediction model for compliance. Based on each G20 member’s outcomes for 371 individual commitments, the impacts of various summit and commitment characteristics were fitted to an ordinal logistic regression model.
This revealed that several key characteristics of summits and commitments, such as meetings of G20 ministers and references to other G20 summits, were associated with a significant increase in the probability of a given member complying with a specific G20 commitment. By contrast, commitments that mentioned specific dates or monetary values were associated with a significantly lower probability of being met (see Figure 1).
There was also variation in the probability of compliance among G20 members. Australia, the United Kingdom, Japan and Canada were more likely to meet commitments, and Italy, Indonesia and Saudi Arabia were less likely to do so (see Figure 2).
Likewise, international organisation membership and economic factors also appear to play a role in the probability of compliance with commitments. Countries that were classified by the International Monetary Fund as having advanced economies were almost three times more likely to meet G20 commitments (see Figure 3). The ability to comply seems to matter much.
Finally, the probability of compliance varied by the commitment’s central subject. Commitments related to microeconomic policy were more than three times as likely to be achieved, the highest of any commitment subject. Compliance was also more likely for commitments on energy, terrorism, macroeconomic policy, labour and employment, and information and communication technology. Compliance was less likely for commitments on international cooperation, the environment and reform international financial institutions (see Figure 4).
Prediction model
Despite many significant variables, the overall explanatory power of this model is very low. Even when all summit characteristics, member properties and commitment features are considered together, only 8.3% of the variance in G20 compliance could be explained (McFadden’s pseudo R2). This suggests that most of the compliance with G20 commitments may be determined by unknown factors or could even be highly random.
Nonetheless, there is hope for increasing G20 effectiveness. All the significant variables examined can be combined into a model to predict future compliance. The ordinal logistic regression model can predict compliance in a holdout set with 62% accuracy, while a random forest classifier model trained on the same data is able to predict compliance with 77%. Although not perfect, this second model performs much better than chance, enabling potential compliance issues to be detected as soon as commitments are made so that resources can be directed accordingly (see Table 1).
Conclusion
Increasing G20 effectiveness by leveraging patterns associated with higher probabilities of compliance is extremely difficult, as most performance is likely determined by factors outside the members’ control. However, by using available information to predict future compliance, it may be possible to direct resources towards assisting members at a higher risk of failing to meet their obligations – thus improving the overall ability of the G20 to achieve its goals.
Note: Full data and code are available at https://github.com/rapsoj/g20-compliance.
