Experiments - Part 1

What is an experiment? In simple terms, an experiment is “a scientific procedure undertaken to make a discovery, test a hypothesis, or demonstrate a known fact.”. Human beings have been using experiments for a very, very long time. Remember Dexter’s Laboratory where Dexter, the wiz kid does crazy experiments? They are somewhat exaggerated, but then, such labs do exist, and marketers (and social scientists by extension) do use them from time to time to take decisions.

At the heart of any experiment is the ability to make a ‘causal inference’. That is, you want to say the act of ‘X’ caused the occurrence of ‘Y’. If you take a step back and think about it logically, you will want a few pre-requisites to establish causation. They are:

1. Temporal sequencing — in other words, the Cause X must necessarily occur before the effect Y. Example: the rooster must crow before the sun rises? Or is it? (This relationship could not be causal, correct?)

2. Non-spurious relationship — The relationship between X and Y is logically explainable and not by chance. For example, Harvard Law School student Tyler Vigen shows that there is a positive correlation between the number of swimming pool drownings and Nicolas Cage movies. How many of you think this is just by chance? Is is really possible that people see Cage’s movies, get so upset, and decide to go for a swim and get a stroke while they are swimming and then die?

3. Eliminate alternate causes — All variables that could influence Y need to be accounted for. It’s very possible that X is correlated with some Z, and that Z impacts Y.

Now, in real life, we can prove causation in a few different ways, all of them involve experimentation. In some cases, these experiments are specifically designed, while in others, these occur in nature and prove instructive of some causal effect. At the heart of all these techniques is randomness. Let’s just dig a little deeper into this.

If you want to prove that the occurrence of X caused Y, all you need to do is create to completely equivalent groups. In one of the groups, you introduce X (also called the treatment group) and in the other you do not introduce X (also called the control group). Let some time pass, and then you check if there is a statistically significant difference in the means of Y (the variable interest) in the control and treatment group. If there is, that in essence means that X caused Y. The critical issue here is in creating ‘completely equivalent groups. This is done using randomness. For instance, we could just flip an unbiased coin and allocate members to one group or the other (Figure below for illustration)

Laboratories in social sciences do not look as jazzy as Dexter’s lab. They do not have laser guided equipment, ultrasound generators, RADAR, or heat tracking sensors. They are very simple. They have tables and chairs. You get lucky if you have some computers. It’s basically a space to separate human beings and get them to answer questions on paper (or on screen). Our work does not deal with atoms and molecules but with human minds. Our instruments are printed on paper, and they ask, “On a scale of 1-7, what do you think about A?”, “would you do Z again should B have occurred?”, “How much would you pay for this product?”. Much of the beauty of the work we do happens in designing the perfect experiments, ensuring that we ask all possible questions that are relevant. If the researcher was resourceful, even classrooms can transform to state-of-the-art research laboratories. Imagine this. I could basically make Section A and B the control group and Section C and D treatment group for some interesting experiment that I have developed. I could perhaps give you different inputs and test this using a question paper. All of that can and often do happen. However, recall that the control and treatment groups need to be separated and they should never know what the other person is subjected to. If that were to happen, people could change their answers accordingly. Now that’s why it’s critical that researchers do not communicate what they are testing to their respondents. It’s possible that they come to know about it after the measurements are done, and that’s ok.

Now that we understand what man-made experiments are, we can discuss a little bit about natural experiments (also known as quasi experiments).

but before we begin, watch this YouTube Short first:

After watching the video, ask yourself. If you observed this, would you be able to make causal inferences about the effects of Di Nitro phenol? How would you be able to make causal claims about its consumption using an experiment (check this link if you are curious)? Should you use an experiment? is there another way?

Honestly, I famished now.

Let’s agree to take them up in the next edition of the newsletter. Until then, stay curious.

Good night students!