Sample Solution

Statement:

Explanation:

1. Indicator Variable 1 ($D1$$D1$D1D1$D1$):
   • $D1=1$$D1=1$D1=1D1 = 1$D1=1$ if the observation belongs to category A
   • $D1=0$$D1=0$D1=0D1 = 0$D1=0$ otherwise
2. Indicator Variable 2 ($D2$$D2$D2D2$D2$):
   • $D2=1$$D2=1$D2=1D2 = 1$D2=1$ if the observation belongs to category B
   • $D2=0$$D2=0$D2=0D2 = 0$D2=0$ otherwise

Justification:

• $D1$$D1$D1D1$D1$ for category A
• $D2$$D2$D2D2$D2$ for category B
• $D3$$D3$D3D3$D3$ for category C

Correct Approach:

Conclusion:

Example:

• $X$$X$XX$X$ = A, B, C (categorical variable)

• $D1$$D1$D1D1$D1$ = 1 if $X$$X$XX$X$ = A, 0 otherwise
• $D2$$D2$D2D2$D2$ = 1 if $X$$X$XX$X$ = B, 0 otherwise

Definitions and Formulas:

1. Coefficient of Determination ($R^2$$R^2$R^(2)R^2$R^2$):
   $$R^2=0.833$$$$R^2=0.833$$R^(2)=0.833R^2 = 0.833$$R^2=0.833$$
2. Number of Observations (n):
   $$n=12$$$$n=12$$n=12n = 12$$n=12$$
3. Number of Explanatory Variables (k):
   $$k=3$$$$k=3$$k=3k = 3$$k=3$$
4. Adjusted $R^2$$R^2$R^(2)R^2$R^2$ Formula:
   $$\text{Adjusted}{R}^2=1-\left(\frac{(1-R^2)(n-1)}{n-k-1}\right)$$$$\text{Adjusted}{R}^2=1-\left(\frac{(1-R^2)(n-1)}{n-k-1}\right)$$“Adjusted “R^(2)=1-(((1-R^(2))(n-1))/(n-k-1))\text{Adjusted } R^2 = 1 – \left( \frac{(1 – R^2)(n – 1)}{n – k – 1} \right)$$\text{Adjusted}{R}^2=1-\left(\frac{(1-R^2)(n-1)}{n-k-1}\right)$$

Calculation:

1. Calculate the numerator:
   $$1-R^2=1-0.833=0.167$$$$1-R^2=1-0.833=0.167$$1-R^(2)=1-0.833=0.1671 – R^2 = 1 – 0.833 = 0.167$$1-R^2=1-0.833=0.167$$
2. Calculate the degrees of freedom adjustment:
   $$n-1=12-1=11$$$$n-1=12-1=11$$n-1=12-1=11n – 1 = 12 – 1 = 11$$n-1=12-1=11$$
   $$n-k-1=12-3-1=8$$$$n-k-1=12-3-1=8$$n-k-1=12-3-1=8n – k – 1 = 12 – 3 – 1 = 8$$n-k-1=12-3-1=8$$
3. Calculate the fraction:
   $$\frac{(1-R^2)(n-1)}{n-k-1}=\frac{0.167\times 11}{8}=\frac{1.837}{8}=0.229625$$$$\frac{(1-R^2)(n-1)}{n-k-1}=\frac{0.167\times 11}{8}=\frac{1.837}{8}=0.229625$$((1-R^(2))(n-1))/(n-k-1)=(0.167 xx11)/(8)=(1.837)/(8)=0.229625\frac{(1 – R^2)(n – 1)}{n – k – 1} = \frac{0.167 \times 11}{8} = \frac{1.837}{8} = 0.229625$$\frac{(1-R^2)(n-1)}{n-k-1}=\frac{0.167\times 11}{8}=\frac{1.837}{8}=0.229625$$
4. Calculate the Adjusted $R^2$$R^2$R^(2)R^2$R^2$:
   $$\text{Adjusted}{R}^2=1-0.229625=0.770375$$$$\text{Adjusted}{R}^2=1-0.229625=0.770375$$“Adjusted “R^(2)=1-0.229625=0.770375\text{Adjusted } R^2 = 1 – 0.229625 = 0.770375$$\text{Adjusted}{R}^2=1-0.229625=0.770375$$

Conclusion:

Definitions and Concepts

1. Leverage: In a regression model, the leverage value $h_{ii}$$h_{ii}$h_(ii)h_{ii}$h_{ii}$ measures the influence of the $i$$i$ii$i$-th observation on the fitted values. It is a diagonal element of the hat matrix $\mathbf{H}$$\mathbf{H}$H\mathbf{H}$\mathbf{H}$, which projects the observed values onto the fitted values.
2. Hat Matrix: The hat matrix $\mathbf{H}$$\mathbf{H}$H\mathbf{H}$\mathbf{H}$ is defined as:
   $$\mathbf{H}=\mathbf{X}({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T$$$$\mathbf{H}=\mathbf{X}({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T$$H=X(X^(T)X)^(-1)X^(T)\mathbf{H} = \mathbf{X} (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T$$\mathbf{H}=\mathbf{X}({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T$$
   where $\mathbf{X}$$\mathbf{X}$X\mathbf{X}$\mathbf{X}$ is the design matrix.
3. Leverage Value: The leverage values $h_{ii}$$h_{ii}$h_(ii)h_{ii}$h_{ii}$ range from $1/n$$1/n$1//n1/n$1/n$ to 1, where $n$$n$nn$n$ is the number of observations. In general, a leverage value significantly higher than the average leverage value $\overline{h}=\frac{2}{n}$$\overline{h}=\frac{2}{n}$bar(h)=(2)/(n)\bar{h} = \frac{2}{n}$\overline{h}=\frac{2}{n}$ (for simple linear regression) is considered an indication of a potential leverage point.
4. Identifying High Leverage Points: For a simple linear regression model (with one predictor), the average leverage value is:
   $$\overline{h}=\frac{2}{n}$$$$\overline{h}=\frac{2}{n}$$bar(h)=(2)/(n)\bar{h} = \frac{2}{n}$$\overline{h}=\frac{2}{n}$$
   Observations with leverage values substantially higher than $\overline{h}$$\overline{h}$bar(h)\bar{h}$\overline{h}$ are considered high leverage points.

Calculation

• $n=15$$n=15$n=15n = 15$n=15$
• $h_{ii}=0.37$$h_{ii}=0.37$h_(ii)=0.37h_{ii} = 0.37$h_{ii}=0.37$

Analysis

• The leverage value $h_{ii}=0.37$$h_{ii}=0.37$h_(ii)=0.37h_{ii} = 0.37$h_{ii}=0.37$ is significantly higher than the average leverage value $\overline{h}=0.133$$\overline{h}=0.133$bar(h)=0.133\bar{h} = 0.133$\overline{h}=0.133$.
• A leverage value of 0.37 is indeed high compared to the average leverage value, suggesting that the observation corresponding to $h_{ii}=0.37$$h_{ii}=0.37$h_(ii)=0.37h_{ii} = 0.37$h_{ii}=0.37$ has a high influence on the fitted regression model.

Conclusion

Hypothesis Testing for ${\beta }_1$${\beta }_1$beta_(1)\beta_1${\beta }_1$

• If ${\mathrm{H}}_0$${\mathrm{H}}_0$H_(0)\mathrm{H}_0${\mathrm{H}}_0$ is not rejected (i.e., there is not enough evidence to conclude that ${\beta }_1$${\beta }_1$beta_(1)\beta_1${\beta }_1$ is significantly different from zero), it suggests that $X_1$$X_1$X_(1)X_1$X_1$ does not contribute significantly to the prediction of $Y$$Y$YY$Y$ given the presence of $X_2$$X_2$X_(2)X_2$X_2$.

Implications of Not Rejecting ${\mathrm{H}}_0$${\mathrm{H}}_0$H_(0)\mathrm{H}_0${\mathrm{H}}_0$

• Statistical Significance: Not rejecting ${\mathrm{H}}_0$${\mathrm{H}}_0$H_(0)\mathrm{H}_0${\mathrm{H}}_0$ means that ${\beta }_1$${\beta }_1$beta_(1)\beta_1${\beta }_1$ is not statistically significant at the chosen significance level (e.g., 0.05). This suggests that $X_1$$X_1$X_(1)X_1$X_1$ may not have a meaningful impact on $Y$$Y$YY$Y$.
• Model Simplification: In practice, if a variable’s coefficient is not statistically significant, analysts often consider removing that variable from the model to simplify it. This helps to avoid overfitting and makes the model more interpretable.

Decision to Retain or Remove $X_1$$X_1$X_(1)X_1$X_1$

• Retaining $X_1$$X_1$X_(1)X_1$X_1$: If $X_1$$X_1$X_(1)X_1$X_1$ is retained in the model despite ${\beta }_1$${\beta }_1$beta_(1)\beta_1${\beta }_1$ not being significant, it could be due to several reasons such as theoretical considerations, potential multicollinearity, or because $X_1$$X_1$X_(1)X_1$X_1$ might become significant in a different model or with more data.
• Removing $X_1$$X_1$X_(1)X_1$X_1$: Often, when ${\mathrm{H}}_0$${\mathrm{H}}_0$H_(0)\mathrm{H}_0${\mathrm{H}}_0$ is not rejected, $X_1$$X_1$X_(1)X_1$X_1$ is removed from the model to simplify it unless there is a strong justification for keeping it.

Conclusion

Example

• ${\mathrm{H}}_0:{\beta }_1=0$${\mathrm{H}}_0:{\beta }_1=0$H_(0):beta_(1)=0\mathrm{H}_0: \beta_1 = 0${\mathrm{H}}_0:{\beta }_1=0$ has a p-value of 0.25 (not significant)
• ${\mathrm{H}}_0:{\beta }_2=0$${\mathrm{H}}_0:{\beta }_2=0$H_(0):beta_(2)=0\mathrm{H}_0: \beta_2 = 0${\mathrm{H}}_0:{\beta }_2=0$ has a p-value of 0.01 (significant)

Logit Link Function

Comparison with the Given Function

• The logit function transforms the probability $\pi$$\pi$pi\pi$\pi$ into the log-odds scale.
• The given function $\log [-\log (1-\pi )]$$\log [-\log (1-\pi )]$log[-log(1-pi)]\log [-\log (1-\pi)]$\log [-\log (1-\pi )]$ is not the same as the logit function.

Proof of False Statement

1. Logit Function:
   $$\text{logit}(\pi )=\log \left(\frac{\pi }{1-\pi }\right)$$$$\text{logit}(\pi )=\log \left(\frac{\pi }{1-\pi }\right)$$“logit”(pi)=log((pi)/(1-pi))\text{logit}(\pi) = \log \left( \frac{\pi}{1-\pi} \right)$$\text{logit}(\pi )=\log \left(\frac{\pi }{1-\pi }\right)$$
2. Given Function:
   $$\log [-\log (1-\pi )]$$$$\log [-\log (1-\pi )]$$log[-log(1-pi)]\log [-\log (1-\pi)]$$\log [-\log (1-\pi )]$$

Simplification

True Logit Function Example

• Logit function:$$\text{logit}(0.5)=\log \left(\frac{0.5}{1-0.5}\right)=\log (1)=0$$$$\text{logit}(0.5)=\log \left(\frac{0.5}{1-0.5}\right)=\log (1)=0$$“logit”(0.5)=log((0.5)/(1-0.5))=log(1)=0\text{logit}(0.5) = \log \left( \frac{0.5}{1-0.5} \right) = \log (1) = 0$$\text{logit}(0.5)=\log \left(\frac{0.5}{1-0.5}\right)=\log (1)=0$$

• Given function:$$\log [-\log (1-0.5)]=\log [-\log (0.5)]\approx \log (0.693)\approx -0.366$$$$\log [-\log (1-0.5)]=\log [-\log (0.5)]\approx \log (0.693)\approx -0.366$$log[-log(1-0.5)]=log[-log(0.5)]~~log(0.693)~~-0.366\log [-\log (1-0.5)] = \log [-\log (0.5)] \approx \log (0.693) \approx -0.366$$\log [-\log (1-0.5)]=\log [-\log (0.5)]\approx \log (0.693)\approx -0.366$$

Conclusion

Multicollinearity

Definition

Causes

• Inclusion of similar variables: Including variables that measure the same phenomenon.
• Dummy variables trap: Including all dummy variables for a categorical predictor (should use $k-1$$k-1$k-1k-1$k-1$ dummies for $k$$k$kk$k$ categories).
• Data collection methods: Sampling methods that inherently create correlations among variables.

Consequences

• Unstable estimates: Regression coefficients can become highly sensitive to changes in the model.
• Inflated standard errors: The standard errors of the regression coefficients are inflated, leading to wider confidence intervals and less reliable statistical tests.
• Difficulty in assessing individual predictor contributions: It becomes challenging to determine the individual effect of each predictor variable on the dependent variable.
• Multicollinearity does not affect the goodness of fit of the model: The overall predictive power remains unchanged, but the interpretation of individual predictors becomes problematic.

Detection

• Variance Inflation Factor (VIF): A measure of how much the variance of a regression coefficient is inflated due to multicollinearity.
  $$\text{VIF}=\frac{1}{1-R_j^2}$$$$\text{VIF}=\frac{1}{1-R_j^2}$$“VIF”=(1)/(1-R_(j)^(2))\text{VIF} = \frac{1}{1 – R^2_j}$$\text{VIF}=\frac{1}{1-R_j^2}$$
  where $R_j^2$$R_j^2$R_(j)^(2)R^2_j$R_j^2$ is the coefficient of determination for the regression of the $j$$j$jj$j$-th predictor on all other predictors. A VIF value greater than 10 is often considered indicative of serious multicollinearity.
• Tolerance: The reciprocal of VIF. A tolerance value below 0.1 indicates high multicollinearity.
  $$\text{Tolerance}=\frac{1}{\text{VIF}}$$$$\text{Tolerance}=\frac{1}{\text{VIF}}$$“Tolerance”=(1)/(“VIF”)\text{Tolerance} = \frac{1}{\text{VIF}}$$\text{Tolerance}=\frac{1}{\text{VIF}}$$
• Correlation matrix: A high correlation coefficient (e.g., above 0.8) between pairs of predictors suggests multicollinearity.

Remedies

• Remove highly correlated predictors: Simplify the model by removing one of the correlated variables.
• Combine predictors: Create a single predictor through techniques such as Principal Component Analysis (PCA).
• Ridge regression: Adds a penalty to the regression to shrink coefficients and reduce multicollinearity effects.
• Increase sample size: Collect more data if possible, as multicollinearity problems can be mitigated with larger samples.

Autocorrelation

Definition

Causes

• Omitted variables: Excluding important variables that capture the time series pattern or spatial structure.
• Incorrect functional form: The model may not adequately capture the relationship between variables.
• Lagged dependent variable: Including a lagged dependent variable can induce autocorrelation if not properly modeled.

Consequences

• Inefficient estimates: Ordinary Least Squares (OLS) estimates remain unbiased, but they are no longer efficient, meaning that they do not have the minimum variance among all linear unbiased estimators.
• Underestimated standard errors: This leads to overestimation of t-statistics and can result in misleading conclusions about the significance of predictors.
• Invalid hypothesis tests: The presence of autocorrelation violates the assumption of independent errors, which can invalidate statistical tests of significance.

Detection

• Durbin-Watson test: Tests for the presence of autocorrelation in the residuals from a regression analysis. Values close to 2 suggest no autocorrelation, values approaching 0 indicate positive autocorrelation, and values approaching 4 indicate negative autocorrelation.
  $$d\approx 2(1-\hat{\rho })$$$$d\approx 2(1-\widehat{\rho })$$d~~2(1- hat(rho))d \approx 2(1 – \hat{\rho})$$d\approx 2(1-\hat{\rho })$$
  where $\hat{\rho }$$\widehat{\rho }$hat(rho)\hat{\rho}$\hat{\rho }$ is the estimated autocorrelation.
• Ljung-Box test: A more general test that can detect autocorrelation at multiple lags.
• Residual plots: Plotting residuals against time can visually reveal patterns indicative of autocorrelation.

Remedies

• Add lagged dependent variables: Including lagged values of the dependent variable can help model the autocorrelation structure.
• Use differencing: For time series data, differencing the data can remove autocorrelation.
• Generalized Least Squares (GLS): This method modifies the regression to account for the autocorrelation structure.
• Newey-West standard errors: Adjusts standard errors to account for heteroscedasticity and autocorrelation.

Summary

• Multicollinearity involves high correlations among predictor variables, leading to unstable and unreliable coefficient estimates. It is detected using VIF and remedied by removing or combining predictors, using regularization techniques, or increasing sample size.
• Autocorrelation refers to the correlation of error terms across observations, often due to temporal or spatial structure in the data. It is detected using the Durbin-Watson test or residual plots and remedied by adding lagged variables, differencing, or using GLS.