Correlation

The full picture

Every time you notice that umbrella sales rise when it rains, or that petrol prices climb when crude oil gets expensive, you are already thinking about correlation. Correlation is a statistical measure that tells us how strongly two variables tend to move together and in which direction. If both variables increase together, the correlation is positive. If one rises while the other falls, the correlation is negative. And if there is no consistent pattern between them, the correlation is close to zero.

The most widely used measure is the Karl Pearson Correlation Coefficient, denoted by the letter r. It can only take values between −1 and +1. A value of +1 means a perfect positive linear relationship — every time one variable goes up, the other also goes up in a consistent linear fashion (not necessarily proportionally, just along a straight line with a positive slope). A value of −1 means a perfect negative linear relationship — one always rises as the other falls, following a straight line with a negative slope. A value of 0 means no linear relationship exists between the variables. In practice, r rarely hits exactly +1, −1, or 0; you will typically see values like +0.82 (strong positive) or −0.45 (moderate negative).

You can first get a visual feel for correlation by drawing a scatter diagram. Plot one variable on the x-axis and the other on the y-axis, then mark a dot for each data pair. If the dots form a tight band sloping upward, r is strongly positive. If they slope downward in a tight band, r is strongly negative. If the dots scatter randomly with no clear slope, r is near zero. After the visual check, you calculate r precisely using the Pearson formula — it compares how much the two variables vary together (their covariance) against how much each varies on its own (their standard deviations). This ratio always stays between −1 and +1, which is what makes it so convenient for comparing relationships across different datasets.

Spearman's Rank Correlation is a second method you need to know, particularly when your data is given as ranks rather than raw numbers. Suppose a panel of judges ranks ten students — 1st to 10th — by their public speaking ability, and separately ranks them by their essay scores. Here the data is ordinal (ranked order), not measured numerically. Pearson's formula is not designed for this type of data, so you use Spearman's formula instead, which is based on the difference between each student's two ranks. If the two rankings agree closely, Spearman's r is close to +1; if they are very different, it falls toward −1.

The single most important idea to take away from this chapter is that correlation does not imply causation. Two variables can move together for a reason completely unrelated to either of them. For example, ice cream sales and cases of sunstroke both rise in the summer months — they are positively correlated — but eating ice cream does not cause sunstroke. A third variable, hot weather, causes both. Examiners at the Kerala HSE board test this idea directly, so always check: is there a plausible third factor driving both variables? That question is the hallmark of careful statistical thinking.

An Indian example

Priya runs a small stationery and notebook shop near a government school in Thrissur. She notices that every time a new academic year starts, both her notebook sales and her pen sales shoot up together. Curious, she records monthly sales for twelve months: notebooks (in packs) and pens (in dozens). When she plots a scatter diagram, all the dots cluster tightly along an upward slope. She calculates the Karl Pearson Correlation Coefficient and gets r = 0.91 — a very strong positive relationship. This tells her that planning her stock for both items together makes sense: if she expects notebook sales to be unusually high in a month, pen sales will very likely be high too. However, Priya is careful not to say that selling notebooks causes pen sales to rise — both are driven by the same underlying factor, the school calendar. She uses the correlation purely to plan her inventory and avoid running out of stock during rush periods.

Common misconceptions to watch for

• Wrong belief: 'A correlation of +1 proves that one variable causes the other.' This is false. r = +1 only means the two variables move together in a perfect positive linear fashion. A third variable could be causing both — correlation shows a pattern, never a cause.
• Wrong belief: 'If correlation is zero, the two variables have nothing to do with each other.' This is false. Zero correlation means no linear relationship, but a strong non-linear (curved) relationship can still exist. For example, human physical strength rises from childhood to about age 25–30, then declines — an inverted-U-shaped curve (like a hill) — but a linear correlation of those two variables could be close to zero.
• Wrong belief: 'Pearson and Spearman correlation always give the same answer, so it does not matter which one you use.' This is false. Pearson works on raw numerical data and is affected by outliers and skewed distributions. Spearman works on ranks and is more reliable when data is ordinal or contains extreme values. For rank-based data you must use Spearman; using Pearson on ranks gives a misleading result.