Posted: April 1, 2017
Due: April 21, 2017
Last Update: March 23, 2017

In this part of this project you will practice and experiment with linear regression using data from gapminder.org. I recommend spending a little time looking at material there, it is quite an informative site.

We will use a subset of data provided by gapminder provided by Jennifer Bryan described in it’s github page.

Get the data from: https://github.com/jennybc/gapminder/blob/master/data-raw/08_gap-every-five-years.tsv

import pandas as pd
data = pd.read_csv("gap.tsv", sep='\t')
data.head()

## # A tibble: 6 × 6
##       country continent  year lifeExp      pop gdpPercap
##        <fctr>    <fctr> <int>   <dbl>    <int>     <dbl>
## 1 Afghanistan      Asia  1952  28.801  8425333  779.4453
## 2 Afghanistan      Asia  1957  30.332  9240934  820.8530
## 3 Afghanistan      Asia  1962  31.997 10267083  853.1007
## 4 Afghanistan      Asia  1967  34.020 11537966  836.1971
## 5 Afghanistan      Asia  1972  36.088 13079460  739.9811
## 6 Afghanistan      Asia  1977  38.438 14880372  786.1134

For this exercise you will explore how life expectancy has changed over 50 years across the world, and how economic measures like gross domestic product (GDP) are related to it.

Exercise 1: Make a scatter plot of life expectancy across time.

Question 1: Is there a general trend (e.g., increasing or decreasing) for life expectancy across time? Is this trend linear? (answering this qualitatively from the plot, you will do a statistical analysis of this question shortly)

A slightly different way of making the same plot is looking at the distribution of life expectancy across countries as it changes over time:

ggplot(aes(x='year', y='lifeExp'), data=data) +\
    geom_violin() +\
    labs(title="Life expectancy over time",
         x = "year",
         y = "life expectancy")

This type of plot is called a violin plot, and it displays the distribution of the variable in the y-axis for each value of the variable in the x-axis.

Question 2: How would you describe the distribution of life expectancy across countries for individual years? Is it skewed, or not? Unimodal or not? Symmetric around it’s center?

Based on this plot, consider the following questions.

Question 3: Suppose I fit a linear regression model of life expectancy vs. year (treating it as a continuous variable), and test for a relationship between year and life expectancy, will you reject the null hypothesis of no relationship? (do this without fitting the model yet. I am testing your intuition.)

Question 4: What would a violin plot of residuals from the linear model in Question 3 vs. year look like? (Again, don’t do the analysis yet, answer this intuitively)

Question 5: According to the assumptions of the linear regression model, what should that violin plot look like?

Exercise 2: Fit a linear regression model using, e.g., the LinearRegression function from Scikit-Learn or the closed-form solution we derived in class, for life expectancy vs. year (as a continuous variable).

Question 6: On average, by how much does life expectancy increase every year around the world?

Question 7: Do you reject the null hypothesis of no relationship between year and life expectancy? Why?

Exercise 3: Make a violin plot of residuals vs. year for the linear model from Exercise 2.

Question 8: Does the plot of Exercise 3 match your expectations (as you answered Question 4)?

Exercise 4: Make a boxplot (or violin plot) of model residuals vs. continent.

Question 9: Is there a dependence between model residual and continent? If so, what would that suggest when performing a regression analysis of life expectancy across time?

Exercise 5: As in the Moneyball project, make a scatter plot of life expectancy vs. year, grouped by continent, and add a regression line.

Question 10: Based on this plot, should your regression model include an interaction term for continent and year? Why?

Exercise 6: Fit a linear regression model for life expectancy including a term for an interaction between continent and year.

Question 11: Are all parameters in the model significantly different from zero? If not, which are not significantly different from zero?

Question 12: On average, by how much does life expectancy increase each year for each continent? (Provide code to answer this question by extracting relevant estimates from model fit)

Exercise 7: Perform an F-test that compares how well two models fit your data: (a) the linear regression models from Exercise 2 (only including year as a covariate) and (b) Exercise 6 (including interaction between year and continent).

Question 13: Is the interaction model significantly better than the year-only model? Why?

Exercise 8: Make a residuals vs. year violin plot for the interaction model. Comment on how well it matches assumptions of the linear regression model. Do the same for a residuals vs. fitted values model. .

Submission

Prepare an iPython Notebook file with code and answers. Submit to ELMS, along with your Notebook for Part II, with the link on Part II's webpage.

Web Accessibility