How you set up your comparison and treatment groups is key to properly evaluating the impact of an event. In this blog post, we’ll explore how to set up the right groups for reliable analysis.
In economics, it is often necessary to evaluate the effects of events for evidence-based policy discussions. Since hypothetical outcomes are unobservable, the effect of an event is actually assessed by comparing the outcome of an intervention group, which is made up of a sample of people who experienced the event, with the outcome of a comparison group, which is made up of people who did not experience the event. The key to this task is to construct two groups whose outcomes have no reason to differ except for the event.
For example, when evaluating the effect of an event on wages, you want to construct two groups such that the average wages of the treatment and comparison groups would have been the same in the absence of the event. Ideally, an experimental method would design the event so that the two groups are randomly assigned. However, this is often not possible when dealing with human samples or social issues. In these situations, alternative methods are needed, one of which is the double difference method.
The effect of an event is measured as the change in the treatment group minus the change in the comparison group. It evaluates the effect of an event based on the parallel trend assumption that a change of the same magnitude as the change in the comparison group would have occurred in the treatment group even in the absence of the event. If this assumption is met, the two groups do not need to be formed so that their pre-event status is the same on average.
The double difference method is said to have been first used by John Snow in 1854. He focused on residents of the same neighborhood in London who received water from two water companies. Only one of the two companies switched water sources, and the residents did not know their water source. By comparing the changes in cholera mortality rates before and after the switch between those who switched and those who didn’t, Snow concluded that cholera is transmitted through water, not air. In economics, the method was first used in the 1910s to determine the effects of introducing a minimum wage.
Applying the double difference method to cases where the parallel trend assumption is not met can lead to an incorrect assessment of the effect of an event. For example, when evaluating the employment-growth effects of a worker training program, the parallel trends assumption would not be met if a larger proportion of workers in the treatment group than in the comparison group were employed in an industry that is rapidly losing jobs.
Therefore, caution should be exercised when using the double difference method. Many factors need to be considered to accurately assess the effect of an event. For example, you need to make sure that the comparison and treatment groups have similar characteristics and that factors other than the event do not influence the results. This will increase the reliability of your evaluation results.
Creating multiple comparison groups and confirming that the evaluation results are the same after applying a two-tailed ANOVA to each provides confidence that the parallel trend assumption is met. You can also reduce the likelihood that the parallel trends assumption is threatened by constructing comparison groups that are statistically similar to the treatment group in many characteristics. These methods can increase confidence in the evaluation of a double difference method.
In addition, there are many other ways to evaluate the effect of an event beyond the double difference method. For example, regression interrupted designs, instrumental variables, and matching techniques. Each of these methods has its own advantages and disadvantages, so it’s important to choose the right method for your research objectives and context. Regression interrupted designs evaluate the effect of an event based on a specific threshold, while instrumental variables use exogenous variables to estimate the effect of an event. Matching techniques seek to match the characteristics of the treatment and comparison groups as closely as possible. By utilizing these different methods appropriately, you can more accurately assess the effect of an event.
Finally, when evaluating the effect of an event, care must be taken in interpreting the results. Not only statistical significance, but also practical significance should be considered, and the limitations of generalizing results should be clearly recognized. This will provide a more reliable basis for policy decisions.
