The population considered in the book is contained in the nycflights13 dataset. In the nycflights13 dataset contains all of the departing flights from airports in the NYC Area.
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(skimr)
Attaching package: 'skimr'
The following object is masked from 'package:mdsr':
skim
Recall the flights dataframe.
flights
Take a sample of flights from NYC to SF. Get all of the flights to SF and take a sample of 25 of them.
Note that I did not set a seed for the sample, so your answers will differ from what is in the book.
SF <- flights |>filter(dest =="SFO", !is.na(arr_delay))SF
sf_25 <- SF |>slice_sample(n =25)sf_25
Get the summary statistics for the sample taken.
sf_25 |>skim(arr_delay)
Data summary
Name
sf_25
Number of rows
25
Number of columns
19
_______________________
Column type frequency:
numeric
1
________________________
Group variables
None
Variable type: numeric
skim_variable
n_missing
complete_rate
mean
sd
p0
p25
p50
p75
p100
hist
arr_delay
0
1
-10.24
23.81
-48
-27
-11
5
35
▃▇▅▃▃
Since the SF dataset contains all flights from NYC to SF in 2013, the statistics computed from the SF dataset are the calculated population paramaters.
SF |>skim(arr_delay)
Data summary
Name
SF
Number of rows
13173
Number of columns
19
_______________________
Column type frequency:
numeric
1
________________________
Group variables
None
Variable type: numeric
skim_variable
n_missing
complete_rate
mean
sd
p0
p25
p50
p75
p100
hist
arr_delay
0
1
2.67
47.67
-86
-23
-8
12
1007
▇▁▁▁▁
Get the 98th percentile of the sample.
sf_25 |>summarize(q98 =quantile(arr_delay, p =0.98))
Get the estimated proportion of flights to SF with a delay less than 90 minutes.
SF |>group_by(arr_delay <90) |>count() |>mutate(pct = n /nrow(SF))
Compare with the 98th percentile. 90 minutes is the 95th percentile.
SF |>summarize(q98 =quantile(arr_delay, p =0.98))
Sample Statistics
The sampling distribution.
Usually sampling is done without replacement.
n <-25SF |>slice_sample(n = n) |>summarize(mean_arr_delay =mean(arr_delay))
Note that different random sample produce different values for the sample statistic.
SF |>slice_sample(n = n) |>summarize(mean_arr_delay =mean(arr_delay))
Using simulation we can approximate the sampling distribution of the sample statistics. Here we are computing the mean, but we could do this for any statistic we are interested in, the median, sample variance, sample standard deviation, etc.
sf_25_means |>summarize(x_bar =mean(mean_arr_delay),se =sd(mean_arr_delay) ) |>mutate(ci_lower = x_bar -2* se, # approximately 95% of observations ci_upper = x_bar +2* se # are within two standard errors )
Using a larger sample size give a smaller Standard Error.
n <-100sf_100_means <-1:500|>map_dfr(~ SF |>slice_sample(n = n) |>summarize(mean_arr_delay =mean(arr_delay)) ) |>mutate(n = n)
Plots to compare the sampling distributions with different sample sizes.
With the Bootstrap we use sampling with replacement
three_flights |>slice_sample(n =3, replace =TRUE)
n <-200orig_sample <- SF |>slice_sample(n = n, replace =FALSE)orig_sample
The reason for sampling with replacement is that we only have the Original Sample of size \(n\). We do not have the population. So if we sampled with out replacement we would only have one sample to look at. Sampling the Original Sample with replace allow us to general lots of samples and to investiage the variability of these samples.
orig_sample |>slice_sample(n = n, replace =TRUE) |>summarize(mean_arr_delay =mean(arr_delay))
`stat_bin()` using `bins = 30`. Pick better value `binwidth`.
Question: Do the histograms look somewhat similar?
orig_sample |>summarize(q98 =quantile(arr_delay, p =0.98))
n <-nrow(orig_sample)sf_200_bs <-1:num_trials |>map_dfr(~orig_sample |>slice_sample(n = n, replace =TRUE) |>summarize(q98 =quantile(arr_delay, p =0.98)) )sf_200_bs |>skim(q98)
