Statistics and Econometrics

Overview

Read econometrics as a chain of questions. What is the population and what is the sample? What parameter is being estimated and what estimator is used? Is the estimator unbiased, consistent and efficient? Is the standard error correct for the data at hand? Does the estimated relationship answer a causal question or merely describe association? Each later topic adds a link that can break: heteroskedasticity breaks efficiency and the usual standard errors, autocorrelation breaks independence over time, endogeneity breaks unbiasedness itself, and non-stationarity can break the entire regression. Learning the diagnostics in parallel with the estimators, rather than as an afterthought, is the fastest route to strong answers, because most examination questions are structured as a model, a violation, a consequence and a remedy.

Probability theory and distributions

A random experiment has a sample space of possible outcomes, events are subsets of that space, and probability assigns numbers between zero and one to events satisfying the axioms of non-negativity, unit total probability and additivity over disjoint events. Conditional probability is the probability of an event given that another has occurred, and Bayes theorem reverses conditioning: it updates a prior probability with new evidence through the likelihood to give a posterior probability. Independence means the joint probability factorises into the product of marginal probabilities; uncorrelatedness is weaker than independence except under joint normality. A random variable maps outcomes to numbers. Discrete variables have probability mass functions; continuous variables have probability density functions, and probabilities are areas under the density. The cumulative distribution function gives the probability that the variable does not exceed a value, for either type. Expectation is the probability-weighted average of a variable, and variance measures spread around that average as the expected squared deviation. Standard deviation is the square root of variance and shares the units of the variable. Covariance measures joint movement of two variables and correlation standardises it to lie between minus one and plus one. Moment generating functions, when they exist, generate moments by differentiation and identify distributions uniquely. Key distributions recur throughout economics. The binomial counts successes in a fixed number of independent trials with constant success probability. The Poisson approximates rare events. The normal distribution is symmetric and bell-shaped, fully described by mean and variance, with the standard normal as its standardised form. The chi-square, t and F distributions arise in inference: chi-square from sums of squared standard normals, t for a standardised mean with estimated variance, and F for ratios of variances used in joint tests.

Descriptive statistics

Descriptive statistics summarise a dataset before any modelling begins, and they protect against modelling nonsense. Measures of central tendency locate the centre: the mean is the arithmetic average and uses all values but is sensitive to outliers; the median is the middle value and is robust to extreme observations; the mode is the most frequent value and is the natural centre for categorical data. For a symmetric distribution mean, median and mode coincide; for a right-skewed distribution the mean exceeds the median, and for a left-skewed distribution the reverse holds. Measures of dispersion describe spread: range, interquartile range, variance, standard deviation and coefficient of variation, which expresses standard deviation relative to the mean and allows comparison across variables with different scales. Quartiles, deciles and percentiles locate positions in the distribution, and a box plot displays median, quartiles and outliers compactly. Skewness measures asymmetry and kurtosis measures tail heaviness relative to the normal distribution; excess kurtosis is zero for a normal distribution, positive for heavy tails and negative for light tails. A frequency distribution groups data into classes, and a histogram displays it, with class width chosen to reveal shape without hiding detail or inventing noise. Correlation measures linear association between two variables, but it is not causation: correlation can arise from a common cause, reverse causation, coincidence or a nonlinear relationship that linear correlation misses. A scatter plot should always precede a correlation coefficient, because the same coefficient can describe very different clouds of points, including a strong nonlinear pattern with zero linear correlation. Index numbers summarise price or quantity change over time: Laspeyres uses base-period weights, Paasche uses current-period weights, and Fisher is their geometric mean. Each has biases: Laspeyres overstates a cost-of-living rise by ignoring substitution toward cheaper goods, while Paasche understates it for the symmetric reason.

Sampling and statistical inference

Inference generalises from a sample to a population. The population is the complete set of interest, a sample is the subset observed, a parameter is a fixed population quantity, and a statistic is a function of sample data used to estimate it. Sampling methods matter: simple random sampling gives every unit an equal chance, stratified sampling divides the population into homogeneous strata and samples within each, cluster sampling samples whole groups, and systematic sampling takes every kth unit. Non-random samples, including convenience and voluntary samples, cannot support the same probability statements, because selection may correlate with the outcome. An estimator is judged by bias, the difference between its expected value and the true parameter; variance, its spread across repeated samples; mean squared error, which combines bias and variance; consistency, convergence to the true value as sample size grows; and efficiency, having smaller variance than competitors. The standard error of an estimator measures its sampling variability and shrinks roughly with the square root of sample size, so quadrupling the sample halves the standard error. A confidence interval gives a range of plausible parameter values with a stated confidence level, and its correct interpretation is about the procedure over repeated samples, not the probability that this particular interval contains the parameter. Hypothesis testing sets a null hypothesis and an alternative, chooses a significance level, computes a test statistic, and rejects the null when the statistic lies in the critical region or, equivalently, when the p-value falls below the significance level. Type I error is rejecting a true null; Type II error is failing to reject a false null; power is one minus the probability of Type II error and rises with sample size, effect size and significance level. Tests for means use t statistics when variance is estimated, tests for variances use chi-square, and tests for proportions use normal approximations in large samples.

Linear regression models and BLUE

The classical linear regression model writes a dependent variable as a linear function of explanatory variables plus an error term. In matrix form, the vector of outcomes equals the regressor matrix multiplied by the coefficient vector plus the error vector. Ordinary least squares chooses coefficients to minimise the sum of squared residuals, the vertical distances between observed and fitted values. The resulting estimator has a closed form involving the inverse of the cross-product matrix, provided regressors are not perfectly collinear. Fitted values are projections of outcomes onto the space spanned by regressors, and residuals are orthogonal to regressors by construction, which is why least squares always passes through the means and why the sum of residuals is zero when an intercept is included. The coefficient of determination, R-squared, is the fraction of variation in the dependent variable explained by the model, and adjusted R-squared penalises additional regressors to avoid rewarding mere addition of variables. The Gauss-Markov assumptions are linearity in parameters, random sampling, no perfect collinearity, zero conditional mean of errors given regressors, and homoskedasticity with no autocorrelation. Under the first four, least squares is unbiased; adding homoskedasticity and no autocorrelation makes it the best linear unbiased estimator, the BLUE result: among all linear unbiased estimators it has smallest variance. Adding normality of errors permits exact finite-sample t and F inference; without normality, large-sample asymptotic inference still applies via the central limit theorem. Violations have distinct consequences. Multicollinearity, high correlation among regressors, does not bias least squares but inflates variances and makes individual coefficients imprecise, diagnosed by variance inflation factors and remedied by dropping variables, collecting more data or using penalised methods, never by pretending the problem away.

Identification and simultaneous equations

Identification asks whether a structural parameter can be recovered from observable data at all, even with an infinite sample. The classic problem is demand and supply: observed price and quantity pairs trace out neither curve alone, because both curves shift and the equilibrium points are intersections. A regression of quantity on price therefore estimates neither demand elasticity nor supply elasticity, but a mixture determined by the relative variability of demand and supply shocks. Endogeneity is the general name for this failure: a regressor correlates with the error term, so least squares is biased and inconsistent, because it attributes to the regressor part of the error variation. Sources of endogeneity include simultaneity, omitted variables, and measurement error in regressors, which attenuates coefficients toward zero in the classical errors-in-variables case. Instrumental variables solve endogeneity when a valid instrument exists: a variable correlated with the endogenous regressor, relevance, but uncorrelated with the error term, exogeneity. Two-stage least squares first regresses the endogenous regressor on instruments and exogenous variables, then regresses the outcome on the predicted values, purging the endogenous component. Weak instruments, only slightly correlated with the regressor, produce imprecise and badly biased estimates in finite samples, so first-stage strength must be reported. The order condition counts restrictions: an equation is identified if the number of excluded exogenous variables is at least the number of included endogenous regressors, and the rank condition, which is necessary and sufficient, requires that the excluded instruments genuinely shift the equation. In a system of simultaneous equations, estimation can be equation by equation with two-stage least squares or jointly with three-stage least squares, which exploits cross-equation error correlation for efficiency, or by full information maximum likelihood. Reduced form equations express endogenous variables as functions of exogenous variables only and are always estimable by least squares, but they do not by themselves reveal structural parameters unless identification restrictions hold.

Discrete choice models and time series

Many economic outcomes are discrete: a person works or does not, buys or does not, chooses one transport mode among several. Linear probability models apply least squares to a binary outcome, but predicted probabilities can fall outside zero and one and errors are necessarily heteroskedastic, so they are at best an approximation. The latent variable approach is standard: an unobserved index crosses a threshold to produce the observed choice. If the latent error is normal, the probit model results; if it is logistic, the logit model results. Coefficients in these models are not marginal effects: the effect of a regressor on probability depends on where on the curve the observation sits, largest near probability one half and smallest near the extremes, so marginal effects must be computed and reported at meaningful values or averaged over the sample. Multinomial logit extends the logic to unordered choices among several alternatives but imposes independence of irrelevant alternatives, the property that the relative odds of two choices do not depend on other alternatives present; nested logit relaxes this by grouping similar alternatives. Ordered probit and logit handle ranked outcomes such as satisfaction scales, and count models handle non-negative integers, with Poisson as the baseline and negative binomial allowing overdispersion. Time series data are ordered in time, so observations are not independent draws. A stationary series has constant mean, variance and autocovariances that depend only on lag, not on calendar time; a non-stationary series such as a random walk has a variance that grows with time and shocks that persist forever. Regressing one non-stationary series on another can produce spurious regression: high R-squared and significant t statistics for unrelated series that merely trend together. Unit root tests, notably the Dickey-Fuller test, test for non-stationarity, and differencing often restores stationarity, at the cost of losing long-run information.

Error types in hypothesis testing

Regression violations: consequence and remedy

Discrete choice models compared