Segment 31. A Tale of Model Selection

From Computational Statistics Course Wiki
Jump to: navigation, search

Watch this segment

(Don't worry, what you see statically below is not the beginning of the segment. Press the play button to start at the beginning.)

The direct YouTube link is http://youtu.be/_G1gzqQzbuM

Links to the slides: PDF file or PowerPoint file

Problems

Modelselection.png

To Calculate

(These problems will be the class activity on Monday, but you can get a head start on them if you want.)

I measured the temperature of my framitron manifold every minute for 1000 minutes, with the same accuracy for each measurement. The data is plotted on the right (with data points connected by straight lines), and is in the file Modelselection.txt.

1. From the data, estimate the measurement error Failed to parse (unknown error): \sigma . (You can make any reasonable assumptions that follow from looking at the data.)

2. Write down a few guesses for functional forms, with different (or adjustable) numbers of parameters that might be good models for the data. Order these by their model complexity (number of parameters) from least to most.

3. Fit each of your models to the data, obtaining the parameters and Failed to parse (unknown error): \chi^2_{min} for each. (Hint: write your code generally enough that you can change from model to model by changing only one or two lines.)

4. Which of your models "wins" the model selection contest if you use AIC? Which for BIC?

To Think About

1. Both AIC and BIC decide whether to allow a new parameter based on a Failed to parse (unknown error): \Delta\chi^2 . So it is possible to think about each as a p-value test for whether a null hypothesis ("no new parameter") is ruled out at some significance level. Viewed in this way, what are the critical p-values being used by each test?

2. Can you give a reasonable rationale, that might be used by a proponent of BIC, for why its Failed to parse (unknown error): \Delta\chi^2 should be larger in magnitude as Failed to parse (unknown error): N (the number of data points) increases?