Fiche de révision : Fundamentals of Regression and Hypothesis Testing

📋 Course Outline

1. Regression assumptions and model fit
2. Estimating slope and intercept
3. Slope significance and prediction intervals
4. Correlation tests and rank measures
5. Chi-square and variance tests
6. Choosing the right hypothesis test
7. Sampling error and standard errors

📖 1. Regression assumptions and model fit

🔑 Key Concepts & Definitions

• Normality of residuals : Normality of residuals means the regression errors should follow a normal distribution for inference validity.
• Homoskedasticity assumption : Homoskedasticity means the variance of the residuals stays constant across values of the independent variable.
• Random residual pattern : A random residual pattern means residuals show no systematic structure when plotted against the independent variable.

📝 Essential Points

• Normality in simple linear regression applies to the residuals, not to the dependent and independent variables themselves.
• For large samples, the normality requirement for residuals can be relaxed by the central limit theorem.
• A non-random residual pattern versus the independent variable indicates a violation like heteroskedasticity or non-independence rather than satisfied homoskedasticity.
• Model fit in simple regression uses $R^2=\text{SSR}/\text{SST}$ and $F=\text{MSR}/\text{MSE}$ with $\text{MSR}=\text{SSR}/1$ and $\text{MSE}=\text{SSE}/(n-2)$.
• With $\text{SSR}=90$, $\text{SSE}=110$, $\text{SST}=200$, and $n=22$, $R^2=0.45$ and $F=16.36$.

📖 2. Estimating slope and intercept

🔑 Key Concepts & Definitions

• Estimated slope coefficient : The estimated slope is the regression line’s $b_1$ computed from centered cross-products and the centered sum of squares of $X$.
• Estimated intercept : The estimated intercept is the regression line’s $b_0$ found from Y\bar and the estimated slope times X\bar.

📝 Essential Points

• For simple linear regression, $\hat b_1=\frac{\sum (X_i-\bar X)(Y_i-\bar Y)}{\sum (X_i-\bar X)^2}$.
• For simple linear regression, $\hat b_0=\bar Y-\hat b_1\bar X$.
• With $\sum (X_i-\bar X)(Y_i-\bar Y)=240$, $\sum (X_i-\bar X)^2=160$, $\bar X=8.0$, and $\bar Y=25.0$, the estimates are $\hat b_1=1.50$ and $\hat b_0=13.0$.
• Swapping the ratio to $\frac{160}{240}$ inverts the slope and then propagates an incorrect intercept via $\hat b_0=\bar Y-\hat b_1\bar X$.
• Adding instead of subtracting in $\hat b_0=\bar Y-\hat b_1\bar X$ produces an intercept that is too large (e.g., $25.0+1.50\times 8.0=37.0$).

💡 Memory Hook

Slope = cross-product over $X$-spread; intercept = mean of $Y$ minus (slope × mean of $X$).

📖 3. Slope significance and prediction intervals

🔑 Key Concepts & Definitions

• Slope t-test : A slope t-test checks whether a regression slope differs from 0 using a t statistic and df = n − 2.
• Standard error of forecast : The standard error of a prediction for Y includes $s_e$ times a factor $\sqrt{1+\tfrac{1}{n}+\tfrac{(X_f-\bar X)^2}{\sum (X_i-\bar X)^2}}$.
• Prediction interval : A prediction interval for a single future Y value equals the forecasted mean $\hat Y_f$ plus or minus a critical t value times the standard error of forecast.

📝 Essential Points

• For H₀: b₁=0, the slope t statistic is $t=(\hat b_1-b_1)/s(\hat b_1)$ with df=n−2, where $s(\hat b_1)=s_e/\sqrt{\sum (X_i-\bar X)^2}$.
• In the example with n=27, $\sum (X_i-\bar X)^2=100$, $s_e=2.0$, and $\hat b_1=0.80$, the t statistic is 4.00 and exceeds ±2.060 at the 5% two-sided level so H₀ is rejected.
• Prediction for Y at $X_f$ uses $\hat Y_f=\hat b_0+\hat b_1X_f$ and $PI=\hat Y_f\pm t\,s_f$ with $s_f=s_e\sqrt{1+\tfrac{1}{n}+\tfrac{(X_f-\bar X)^2}{\sum (X_i-\bar X)^2}}$.
• In the example with Ŷ = 4.0 + 1.2X, n=12, $s_e=3.0$, $\bar X=5.0$, $\sum (X_i-\bar X)^2=150$, and $X_f=7.0$, the standard error of forecast is $s_f=3.0\times\sqrt{1.11}=3.16$ and the 95% PI is $12.40\pm 7.04=(5.36,19.44)$.
• For prediction intervals, omitting the leading “1 +” term inside $s_f$ (and using $\pm t\,s_e$ instead) produces a too-narrow interval.
• The prediction interval width is minimized when $X_f=\bar X$ because $s_f$ includes the term $\frac{(X_f-\bar X)^2}{\sum (X_i-\bar X)^2}$, so it widens as $|X_f-\bar X|$ grows.

💡 Memory Hook

Slope-significance uses slope SE: divide by $\sqrt{\sum (X_i-\bar X)^2}$; prediction intervals add the “1 + …” term for extra forecast uncertainty.

📖 4. Correlation tests and rank measures

🔑 Key Concepts & Definitions

• Pearson correlation t-test : A parametric test that evaluates whether a population correlation between two approximately normal continuous variables is zero using the sample correlation.
• Spearman rank-correlation test : A nonparametric test that assesses correlation using rank data when continuous variables show marked departures from normality such as skewness and outliers.
• Spearman rank correlation coefficient : A rank-based correlation measure computed from paired rank differences to quantify monotonic association.
• Chi-square test of independence : A contingency-table test that checks whether two categorical classifications are independent rather than correlated.

📝 Essential Points

• For two approximately normal continuous variables, use the parametric (Pearson) correlation t-test to test whether the population correlation is zero.
• For continuous variables with pronounced non-normality like marked skewness and outliers, use the Spearman rank-correlation test computed on ranks.
• For testing population correlation with sample correlation $r$ and $n$ paired observations, use $t=\dfrac{r\sqrt{n-2}}{\sqrt{1-r^2}}$ with $n-2$ degrees of freedom.
• Spearman rank correlation uses $r_s=1-\dfrac{6\sum d^2}{n(n^2-1)}$, so with $n=6$ and $\sum d^2=14$ you get $r_s=0.60$.
• Each chi-square cell contributes $\dfrac{(O-E)^2}{E}$, so with $O=60$ and $E=48$ the contribution is $3.00$ and for $3$ rows by $4$ columns the degrees of freedom are $(3-1)(4-1)=6$.

📖 5. Chi-square and variance tests

🔑 Key Concepts & Definitions

• Chi-square variance test : A variance test uses a chi-square statistic formed from $(n − 1)s^2/\sigma_0^2 with$n − 1$ degrees of freedom.
• F-test for equal variances : An F-test compares two population variances by using the ratio of the two sample variances.

📝 Essential Points

• For a contingency table with r rows and c columns, chi-square test degrees of freedom are (r − 1)(c − 1), not r × c.
• For testing a single normal population variance, $\chi^2=(n−1)s^2/\sigma_0^2$ with $n−1$ degrees of freedom.
• In the variance example with n=25, s=5% so s²=25 and σ0²=36, the chi-square statistic is 16.67 using n−1=24.
• Comparing two variances of independent normal samples requires an F-test, not a chi-square test.
• Reducing the significance level (e.g., 5% to 1%) lowers Type I error and increases Type II error, so power falls.

💡 Memory Hook

Degrees of freedom for independence multiply shrinkage: (rows−1)(cols−1); variance chi-square scales s² by (n−1)/σ0².

📖 6. Choosing the right hypothesis test

🔑 Key Concepts & Definitions

• Level of significance : The level of significance is the probability of making a Type I error under the null hypothesis.
• Paired comparisons test : A paired comparisons test uses within-subject differences to test a mean difference when the two samples are dependent (matched observations).
• Two-tailed alternative : A two-tailed alternative tests whether a parameter is different from a value in either direction, corresponding to a “not equal to” (≠) alternative.

📝 Essential Points

• Lowering the significance level from 5% to 1% reduces the Type I error probability directly.
• When the rejection region shrinks after lowering α, the null is rejected less often, so the Type II error probability increases.
• For comparing mean returns measured on the same days for two indexes, the data are paired so the appropriate test is a paired comparisons test.
• If the question asks whether means are “different,” the correct hypothesis is a two-tailed (≠) alternative, not a one-tailed test.

💡 Memory Hook

Lower α → smaller rejection region → less Type I → more Type II (power falls).

📖 7. Sampling error and standard errors

🔑 Key Concepts & Definitions

• Standard error of the sample mean : The standard error measures the typical sampling variation in an estimate of the population mean.
• Sampling distribution of the sample mean : The sampling distribution is the distribution of the sample mean over repeated samples from the same population.
• Bootstrap resampling : Bootstrap resampling creates many resamples by drawing with replacement from the original data to approximate a statistic’s sampling variability.
• Jackknife : The jackknife estimates sampling variability by recomputing a statistic after systematically leaving out one observation at a time.

📝 Essential Points

• If the population standard deviation σ is known, the standard error is $\sigma_{\bar{x}}=\sigma/\sqrt{n}$.
• With variance $0.0036$, $\sigma=\sqrt{0.0036}=0.06=6\%$ and for $n=64$ the standard error is $6\%/8=0.75\%$.
• If the standard error must be at most 1% and $\sigma=8\%$, then $1\%=8\%/\sqrt{n}$ gives $\sqrt{n}=8$ so $n=64$.
• By the central limit theorem, for finite-variance populations the sampling distribution of $\bar{x}$ is approximately normal for large $n$ (generally $n\ge 30$), even if the population is non-normal.
• For estimating the standard error of a median, bootstrap works by repeatedly resampling with replacement from the original data and using the resampled medians’ variability.
• In contrast, the jackknife leaves out one observation at a time deterministically, so it produces essentially the same result each run on the same data set.

💡 Memory Hook

SE shrinks like 1/√n: bigger samples tighten $\bar{x}$ around \mu{}, and bootstrap handles stats where $s/\sqrt{n}$ won’t.

📊 Synthesis Tables

Correlation test choice

Prediction vs confidence for regression

⚠️ Common Pitfalls & Confusions

1. Mixing up normality: the normality assumption applies to regression residuals, not to the dependent and independent variables themselves.
2. Using a residuals-vs-X plot: a non-random pattern indicates a problem (e.g., heteroskedasticity or non-independence), not that homoskedasticity is satisfied.
3. In slope/intercept computation, swapping the ratio for b1 (using 160/240 instead of 240/160) flips the slope and then forces an incorrect intercept.
4. For the slope t-test, dividing by the wrong standard error (e.g., using se instead of s(b̂1)=s_e/√∑(X_i−X̄)^2) understates t.
5. For prediction intervals, omitting the leading “1 +” inside s_f (and using ±t·s_e instead) produces an interval that is too narrow.
6. In 0/1 dummy regression, interpreting the intercept as the group mean when the indicator=1 (it is for indicator=0), and interpreting the slope as the level rather than the difference in means.
7. Using chi-square for variance comparisons: chi-square is for a single population variance; comparing two independent variances requires an F-test.

✅ Exam Checklist

1. State the correct normality/homoskedasticity/residual-randomness interpretation for assumptions in simple linear regression.
2. Identify which quantities are assumed normal in simple regression (residuals) and when CLT can relax the requirement.
3. Compute R^2 and the model-fit F statistic from SSR, SSE, SST, and n using MSR=SSR/1 and MSE=SSE/(n−2).
4. Calculate slope b̂1 using the centered cross-product divided by the centered sum of squares of X, and calculate intercept b̂0=ȳ−b̂1 x̄.
5. Recognize the consequences of common algebra mistakes in b̂1/b̂0 (inverting the ratio, adding instead of subtracting in the intercept).
6. For testing H0: b1=0, compute the slope t statistic using df=n−2 and s(b̂1)=s_e/√∑(X_i−X̄)^2, then decide reject/fail to reject with the correct two-sided critical values.
7. Construct a 95% prediction interval for a single future Y: compute Ŷ_f=b̂0+b̂1X_f, then s_f=s_e√(1+1/n+(X_f−X̄)^2/∑(X_i−X̄)^2), and use PI=Ŷ_f±t·s_f.
8. Decide which correlation test to use (Pearson vs Spearman vs chi-square) based on variable type (continuous vs categorical) and distributional features (approximately normal vs skew/outliers).
9. Compute the Pearson correlation t-test statistic t= r√(n−2)/√(1−r^2) with df=n−2.
10. Compute Spearman’s rank correlation using r_s=1−6∑d^2/[n(n^2−1)], and compute the specified chi-square cell contribution (O−E)^2/E for a given observed/expected pair.
11. Compute chi-square degrees of freedom for independence as (r−1)(c−1), compute chi-square for a single variance as χ^2=(n−1)s^2/σ0^2, and distinguish it from equal-variances F-testing.
12. Choose the correct test under dependence and alternative direction: paired vs independent means, and two-tailed (≠) vs one-tailed alternatives; also explain how lowering α changes Type I/Type II and power.
13. Use standard error facts for the sample mean (σ known: σ/√n), CLT justification for approximate normality of x̄, and when bootstrap vs jackknife is appropriate for estimating sampling variability (median).