Multivariate Regression

\(y=\beta_1+\beta_1x_1+\beta_2x_2+\epsilon\)

How do we interpret \(\beta_1\), \(\beta_2\)?

• \(y=10+3x_1+4x_2\), \(x_1=5\), \(x_2=3\)
  
  
• \(y=10+18+20=48\)
  
  
• 1 unit increase in \(x_1\) led to a \(\beta_1\) increase in \(y\) (just like bivariate regression)
• But what about \(x_2\)? It did not change. So this change is only true holding \(x_2\) constant
• We can hold \(x_2\) constant to see how \(y\) changes as \(x_1\) changes at that level of \(x_2\)

Evaluating the Model: Adjusted \(\text{R}^2\)

• Recall we can use \(\text{R}^2=1-\text{SSR}/\text{TSS}\)
• When we add a new independent variable, \(\text{TSS}\) does not change. \(\text{TSS}=(u-\text{mean}(y))^2\)
• However, the new variable will always cause \(\text{SSR}\), \((y-\hat{y})^2\) to decrease. Therefore, \(\text{R}^2\) will always decrease, which makes adding more variables ostensibly better
• Adjusted \(\text{R}^2\) adds a disincentive (penalty) for adding new variables: $$\text{Adj R}^2=1-\frac{(n-1)}{n-k-1}\frac{\text{SSR}}{TSS}$$

Calculating the bias effect

1. Population model (true relationship): \(y=\beta_0+\beta_1x_1+\beta_2x_2+\nu\)

2. Our model: \(y=\hat{\beta}_0+\hat{\beta}_1x_1+\upsilon\)

3. Auxiliary model: \(x_2=\delta_0+\delta_1x_1+\epsilon\)

Example: Bostom Housing Data

Multicollinearity

• Multivariate linear models cannot handle perfect multicollinearity
• Example: we have two variables: \(x_1\) and \(x_2=3 \times x_1\)
• Fit model to predict \(y\) with \(x_1\) and \(x_2\):
  • \(y=\beta_0+\beta_1x_1+\text{NA}\), where \(\text{NA}\) stands for not a value
    
    
• We can think of this as \(\beta_1\) containing the entire effect for both \(x_1\) and \(x_2\). After all, these variables are the same.
• Including highly correlated variables in our model will not produce biased estimates, but it will harm our precision.

Baseball example

• Use home runs, batting average, and RBI to predict salary
• Variables are defined as follows:
  • \(\text{salary}=\text{homeruns}\times 10,000+\epsilon\)
  • \(\text{BA}=\text{homeruns}+270+\epsilon\)
  • \(\text{RBI}=\text{homeruns}\times 3+\epsilon\)
  • Example: \(\text{homeruns}=30\), \(\text{BA}=300\), \(\text{RBI}=90\), \(\text{salary}=300,000\)
• Fit a model for each variable individually:
  • \(\text{salary}=9,934.27\times\text{HR}\)
  • \(\text{salary}=1,002.95\times\text{BA}\)
  • \(\text{salary}=3,291.02\times\text{RBI}\)
• Fit a model with all three: \(\text{salary}=9,226.169\times \text{HR}+225.884 \times \text{RBI}+2.982 \times\text{BA}\)
• What is this model saying? Why not: \(\text{salary}=9,934.27\times \text{HR}+3,291.02 \times \text{RBI}+1,002.95 \times\text{BA}\)

Helpful resource

• Omitted variable bias and multicollinearity discussion: https://are.berkeley.edu/courses/EEP118/current/handouts/OVB%20versus%20Multicollinearity_eep118_sp15.pdf

Hypothesis testing

Back to baseball

• To perform an F test, we compare a model with restrictions to a model without restrictions and see if there is a significant difference. Think of restrictions as features not included in the model
• \(\text{salary}=\text{years}+\text{gmsYear}+\text{HR}+\text{RBI}+\text{BA}\)
• If \(\text{HR}\), \(\text{RBI}\), \(\text{BA}\) all have no effect on \(\text{salary}\), then the model \(\text{salary}=\text{years}+\text{gmsYears}\) should perform just as well
• How do we measure performance? Sum of squared residuals (SSR)!
• Test statistics: \(\frac{\text{SSR}_\text{r}-\text{SSR}_\text{ur}/q}{\text{SSR}_\text{ur}/(n-k-1)}\)
• The above fraction is the ratio of two chi squared variables divided by their degrees of freedom, which makes this F-distributed
• Remember adding variables can only improve the model, so the F statistic will always be positive

Affine

• Affine transformations are transformations that do not affect the fit of the model. The most common example is scaling transformations

• Example:

  • \(\text{weight(lbs)}=5+2.4\times \text{height(inches)}\)
  • \(\text{weight(lbs)}=5+0.094\times \text{height(mm)}\)

• This is why scaling variables is not necessary for linear regression, but knowing the scale of your variables is important for interpretation

Polynomial

• Linear regression can still be used to fit data with a non-linear distribution
• The model is linear in parameters, not necessarily variables
• i.e. we must have \(\beta_1\), \(\beta_2\), \(\beta_3\), but we can utilize \(x_1^2\) or \(x_2/x_3\)
• We might leverage the above to generate a curved regression line, providing a better fit in some cases
• How do we now interpret the coefficients?

$$\hat{\text{wage}}=3.12+.447\text{exp}-0.007\text{exp}^2$$

• The big difference is the effect of an increase in experience on wage now depends on the level of experience

Logarithmic

Recall that the natural logarithm is the inverse of the exponential function, so \(\text{ln}(e^x)=x\), and:

Interpreting log variables

• \(\beta_0=5\), \(\beta_1=0.2\)

• Level-log: \(y=5+0.2\text{ln}(x)\)

  • 1% change in \(x=\beta_1/100\) change in \(y\)
    
    

• Log-level: \(\text{ln}(y)=5+0.2(x)\)

  • 1 unit change in \(x=\beta_1 \times 100\text{%}\) change in \(y\)
    
    

• Log-log: \(\text{ln}(y)=5+0.2\text{ln}(x)\)

  • 1% change in \(x=\beta_1\text{%}\) change in \(y\)
    
    

Dummy variables

• Dummy variables is how categorical variables can be mathematically represented
• They represent groups or place continuous variables into bins
• What is this regression telling us?
  • \(\text{nbaSalary}=5\times \text{PPG}+10.5\times \text{guard}+9.6\times \text{forward}+10.8\times \text{center}\)
    
    
• Do we need dummy variables for \(guard\), \(forward\), \(center\)?
• How would the regression change if we only used 2 out of 3?
• \(\text{nbaSalary}=10.5 + 5\times \text{PPG}-0.9\times \text{forward}+0.3\times \text{center}\)