Topics
- Interval estimates
- Simulation
Earlier, we calculated point estimates from samples drawn from a population. This activity introduces the calculation of interval estimates, i.e. confidence intervals. While a point estimate consists of a single value, a confidence interval gives a plausible range of values for a parameter.
Install packages
You will need openintro and tidyverse packages. We will also use the infer package for resampling.
if (!require("openintro")) {install.packages("openintro"); require("openintro")}
if (!require("tidyverse")) {install.packages("tidyverse"); require("tidyverse")}
library(openintro)
library(tidyverse)
load the data
Load the dataset yrbss available in the openintro package
# load the data
data("yrbss")
Inspect the data
Exercise 1: Calculating confidence intervals
Run the following code to take a random sample of 30 individuals from yrbss and store the subset as yrbss.sample. The code includes a section that removes any rows from yrbss where there are no data recorded for weight. The version of yrbss without missing weight values is stored as yrbss.complete.
#remove rows with missing values
yrbss.complete = yrbss[complete.cases(yrbss$weight), ]
#set parameters
sample.size = 30
#set seed for pseudo-random sampling
set.seed(5011)
#obtain random sample of row numbers
sample.rows = sample(1:nrow(yrbss.complete), sample.size)
#create yrbss.sample
yrbss.sample = yrbss.complete[sample.rows, ]
- A confidence interval is calculated from four quantities: the sample mean ¯x, the sample standard deviation s, the sample size n, and the critical z-value z⋆.
- Calculate ¯xweight and sweight, the mean and standard deviation of weight in the sample.
The sample mean ¯xweight is 69.63 kg and the sample standard deviation sweight is 18.26 kg.
Do you need to include na.rm = TRUE. Why or why not?
- For a 95% confidence interval, z⋆ is the point on a standard normal distribution that has area 0.975 to the left (and area 0.025 to the right). Calculate the value of z⋆ for a 95% confidence interval.
## [1] 1.959964
- Calculate a 95% confidence interval based on the sampled weights. The quantity (z⋆×s√n) is known as the margin of error, m. ¯x±z⋆×s√n→(¯x−z⋆s√n, ¯x+z⋆s√n)
#use r as a calculator
ci.lower.bound =
ci.upper.bound =
#confidence interval
ci.lower.bound; ci.upper.bound
x.bar
s
- The standard deviation of weight in the sample is 18.26 kg. Suppose that the standard deviation in the sample were 20 kg or 25 kg, but that ¯x and n remain constant. Re-run the calculation from part c) and describe the effect of larger (sample) standard deviation on the confidence interval.
#use r as a calculator to see effect of increasing variability in the sample on confidence interval.
ci.lower.bound =
ci.upper.bound =
#confidence interval
ci.lower.bound; ci.upper.bound
#confidence interval
ci.lower.bound; ci.upper.bound
Less variability in a sample gives a more precise estimate of the population parameter. Looking at the formula, the margin of error increases as s increases, so the confidence interval should get wider as variability in a sample increases.
Exercise 2: Changing confidence level
In general, for a confidence interval of (1−α)(100)%, z⋆ is the point on a standard normal distribution that has area 1−(α/2) to the left (and area α/2 to the right). For a 95 % confidence interval, α=0.05; z⋆ is the point on a standard normal distribution with area 1−(0.05/2)=0.975 to the left.
- Calculate a 90% confidence interval for mean weight based on the sample data.
#use r as a calculator
ci.lower.bound =
ci.upper.bound =
#confidence interval
ci.lower.bound; ci.upper.bound
- Calculate a 99% confidence interval for mean weight based on the sample data.
#use r as a calculator
ci.lower.bound =
ci.upper.bound =
#confidence interval
ci.lower.bound; ci.upper.bound
- Compare the 95% confidence interval calculated in the previous question to the 90% and 99% confidence intervals. Describe the relationship between confidence level and width of the interval.
As confidence level increases, the width of the interval __________.
- Which of the intervals calculated (90%, 95%, 99%) do you find to be the most informative as an estimate of the mean weight of high school age students in the US? Explain your answer.
Consider an extreme case—a theoretical “100%” confidence interval that captures the population mean with absolute certainty would be (0,+∞) kg. Such a wide interval is completely uninformative for providing an plausible range of values for μ.
Exercise 3: Using t.test() in R to calculate confidence intervals.
The t.test() command can be used to calculate confidence intervals. For example, the command to calculate a 95% confidence interval for height in yrbss.complete is
t.test(yrbss.complete$height, conf.level = 0.95)$conf.int
- Calculate a 95% confidence interval for mean weight using
t.test().
The answer will differ slightly from the one in Question 1 because `R calculates confidence intervals using a critical value from the t distribution rather than from the standard normal distribution. The t distribution will be looked into more depth later.
- Examine the effect of larger sample sizes on the confidence interval by re-running the code for sample sizes of 50, 100, and 300. Describe your observations.
As the sample size increases, the confidence intervals become ______.
Intuitively, one would expect that having more data from the population allows for a more precise estimate of the population parameter. This effect can also be predicted since as sample size increases, margin of error ________.
Exercise 4. The interpretation of “confidence”
The method discussed for computing an x confidence interval will produce an interval that x times out of 100 (on average) contains the population mean.
Consider the individuals in yrbss.complete as the target population. Calculate the population mean weight, μweight.
Does the 95% interval calculated in part b) of Question 3 for n=100 contain μweight?
Exercise 5. Simulation
Run the following code to take 1,000 random samples of size 100 from yrbss.complete. For each sample, R calculates mean weight for participants in the sample and stores the value in the vector sample.means. The margin of error m is calculated according to the defined confidence level and stored in the vector m. The logical variable contains.mu records TRUE if a confidence interval contains μweight and FALSE otherwise.
#set parameters
sample.size = 100
conf.level = 0.95
replicates = 1000
#set seed for pseudo random sampling
set.seed(2017)
#create empty vectors to store results
sample.means = vector("numeric", replicates)
m = vector("numeric", replicates)
#calculate sample means and margins of error
for(k in 1:replicates){
sample.rows = sample(nrow(yrbss.complete), sample.size)
z.star = qnorm(1 - (1 - conf.level)/2)
sample.means[k] = mean(yrbss.complete$weight[sample.rows])
m[k] = z.star * (sd(yrbss.complete$weight[sample.rows]) / sqrt(sample.size))
}
#define upper and lower bounds of confidence interval
ci.lb = sample.means - m
ci.ub = sample.means + m
#does the confidence interval contain mu?
mu = mean(yrbss.complete$weight)
contains.mu = (ci.lb < mu) & (ci.ub > mu)
table(contains.mu)
## contains.mu
## FALSE TRUE
## 55 945
How many intervals contain the population mean μweight?
Re-run the simulation with confidence levels of 0.90 and 0.99. What happens to the number of intervals that contain μweight?
What is a disadvantage associated with using an interval that gives a more precise estimate of the parameter; e.g., a 90% interval rather than a 95% interval?
A more precise interval is less likely to contain the population parameter.
- From what you have observed from the exercises above about the relationship between an interval estimate (¯x±m) and the population mean (μ), evaluate the following statement:
“The 95% confidence interval as calculated from the 13,572 sampled high school students in the 2013 YRBSS survey is (67.61, 68.20) kg. It is possible to definitively conclude that this interval contains the mean weight of the 21.2 million high school students in the US in 2013.”
This activity is modified from the Open Intro Biostatistics lab at openintro.org
LS0tCnRpdGxlOiAiYFJgIEFjdGl2aXR5IDM6IENvbmZpZGVuY2UgSW50ZXJ2YWwgZm9yIG9uZSBtZWFuIgphdXRob3I6ICJEci4gVW1hIFJhdmF0IgpkYXRlOiAiSnVseSAxNCAyMDIxIgpvdXRwdXQ6IG9wZW5pbnRybzo6bGFiX3JlcG9ydAotLS0KCiMjIyBUb3BpY3MgCgoxLiBJbnRlcnZhbCBlc3RpbWF0ZXMKMi4gU2ltdWxhdGlvbgoKRWFybGllciwgd2UgY2FsY3VsYXRlZCBwb2ludCBlc3RpbWF0ZXMgZnJvbSBzYW1wbGVzIGRyYXduIGZyb20gYSBwb3B1bGF0aW9uLiBUaGlzIGFjdGl2aXR5IGludHJvZHVjZXMgdGhlIGNhbGN1bGF0aW9uIG9mIGludGVydmFsIGVzdGltYXRlcywgaS5lLiAqKmNvbmZpZGVuY2UgaW50ZXJ2YWxzKiouIFdoaWxlIGEgcG9pbnQgZXN0aW1hdGUgY29uc2lzdHMgb2YgYSBzaW5nbGUgdmFsdWUsIGEgY29uZmlkZW5jZSBpbnRlcnZhbCBnaXZlcyBhIHBsYXVzaWJsZSByYW5nZSBvZiB2YWx1ZXMgZm9yIGEgcGFyYW1ldGVyLgoKCiMjIyBCYWNrZ3JvdW5kIGluZm9ybWF0aW9uCgpUaGlzIGFjdGl2aXR5IHVzZXMgZGF0YSBmcm9tIHRoZSBZb3V0aCBSaXNrIEJlaGF2aW9yYWwgU3VydmVpbGxhbmNlIFN5c3RlbSAoWVJCU1MpLCBhIHllYXJseSBzdXJ2ZXkgY29uZHVjdGVkIGJ5IHRoZSBVUyBDZW50ZXJzIGZvciBEaXNlYXNlIENvbnRyb2wgdG8gbWVhc3VyZSBoZWFsdGgtcmVsYXRlZCBhY3Rpdml0eSBpbiBoaWdoLXNjaG9vbCBhZ2VkIHlvdXRoLiBUaGUgZGF0YXNldCBgeXJic3NgIGNvbnRhaW5zIHJlc3BvbnNlcyBmcm9tIHRoZSAxMyw1NzIgcGFydGljaXBhbnRzIGluIDIwMTMgZm9yIGEgc3Vic2V0IG9mIHRoZSB2YXJpYWJsZXMgaW5jbHVkZWQgaW4gdGhlIGNvbXBsZXRlIHN1cnZleSBkYXRhLgoKVmFyaWFibGVzIGluIGB5cmJzc2AgaW5jbHVkZToKCi0gYGFnZWA6IGFnZSBpbiB5ZWFycwotIGBnZW5kZXJgOiBnZW5kZXIgb2YgcGFydGljaXBhbnQsIHJlY29yZGVkIGFzIGVpdGhlciBgZmVtYWxlYCBvciBgbWFsZWAKLSBgZ3JhZGVgOiBncmFkZSBpbiBoaWdoIHNjaG9vbCAoOS0xMikKLSBgaGVpZ2h0YDogaGVpZ2h0LCBpbiBtZXRlcnMgKDEgbSA9IDMuMjggZnQpCi0gYHdlaWdodGA6IHdlaWdodCwgaW4ga2lsb2dyYW1zICgxIGtnID0gMi4yIGxicykKClRoZSBDREMgdXNlZCB0aGUgcmVzcG9uc2UgZnJvbSB0aGUgMTMsNTcyIHN0dWRlbnRzIHRvIGVzdGltYXRlIHRoZSBoZWFsdGggYmVoYXZpb3JzIG9mIHRoZSB0YXJnZXQgcG9wdWxhdGlvbjogdGhlIDIxLjIgbWlsbGlvbiBoaWdoIHNjaG9vbCBhZ2VkIHN0dWRlbnRzIGluIHRoZSBVbml0ZWQgU3RhdGVzIGluIDIwMTMuCgpUaGUgZ29hbCBpbiB0aGlzIGFjdGl2aXR5IGlzIHRvIG9ic2VydmUgdGhlIGVmZmVjdCBvZiBzYW1wbGluZyBieSB0cmVhdGluZyB0aGUgMTMsNTcyIGluZGl2aWR1YWxzIGluIGB5cmJzc2AgYXMgYSB0YXJnZXQgcG9wdWxhdGlvbiBhbmQgZHJhd2luZyByYW5kb20gc2FtcGxlcy4gSG93IGRvIGludGVydmFsIGVzdGltYXRlcyBvZiBtZWFuIHdlaWdodCwgJChcb3ZlcmxpbmV7eH1fe3dlaWdodH0gLSBtLCBcb3ZlcmxpbmV7eH1fe3dlaWdodH0gKyBtKSQsIGNhbGN1bGF0ZWQgZnJvbSByYW5kb20gc2FtcGxlcyBjb21wYXJlIHRvIHRoZSBwb3B1bGF0aW9uIHBhcmFtZXRlciwgJFxtdV97d2VpZ2h0fSQ/CgoKIyMjIEluc3RhbGwgcGFja2FnZXMgCgpZb3Ugd2lsbCBuZWVkIGBvcGVuaW50cm9gIGFuZCBgdGlkeXZlcnNlYCBwYWNrYWdlcy4gV2Ugd2lsbCBhbHNvIHVzZSB0aGUgKippbmZlcioqIHBhY2thZ2UgZm9yIHJlc2FtcGxpbmcuCgpgYGB7ciBsb2FkLXBhY2thZ2VzLCBtZXNzYWdlPUZBTFNFfQppZiAoIXJlcXVpcmUoIm9wZW5pbnRybyIpKSB7aW5zdGFsbC5wYWNrYWdlcygib3BlbmludHJvIik7IHJlcXVpcmUoIm9wZW5pbnRybyIpfQppZiAoIXJlcXVpcmUoInRpZHl2ZXJzZSIpKSB7aW5zdGFsbC5wYWNrYWdlcygidGlkeXZlcnNlIik7IHJlcXVpcmUoInRpZHl2ZXJzZSIpfQpsaWJyYXJ5KG9wZW5pbnRybykKbGlicmFyeSh0aWR5dmVyc2UpCgpgYGAKCiMjIyBsb2FkIHRoZSBkYXRhIAoKTG9hZCB0aGUgZGF0YXNldCBgeXJic3NgIGF2YWlsYWJsZSBpbiB0aGUgYG9wZW5pbnRyb2AgcGFja2FnZQoKYGBge3IgbG9hZC1kYXRhfQojIGxvYWQgdGhlIGRhdGEKZGF0YSgieXJic3MiKQpgYGAKSW5zcGVjdCB0aGUgZGF0YSAKCmBgYHtyLCBldmFsID0gRkFMU0V9ClZpZXcoeXJic3MpCmBgYAoKCiMjIyBFeGVyY2lzZSAxOiBDYWxjdWxhdGluZyBjb25maWRlbmNlIGludGVydmFscwoKUnVuIHRoZSBmb2xsb3dpbmcgY29kZSB0byB0YWtlIGEgcmFuZG9tIHNhbXBsZSBvZiAzMCBpbmRpdmlkdWFscyBmcm9tIGB5cmJzc2AgYW5kIHN0b3JlIHRoZSBzdWJzZXQgYXMgYHlyYnNzLnNhbXBsZWAuIFRoZSBjb2RlIGluY2x1ZGVzIGEgc2VjdGlvbiB0aGF0IHJlbW92ZXMgYW55IHJvd3MgZnJvbSBgeXJic3NgIHdoZXJlIHRoZXJlIGFyZSBubyBkYXRhIHJlY29yZGVkIGZvciBgd2VpZ2h0YC4gVGhlIHZlcnNpb24gb2YgYHlyYnNzYCB3aXRob3V0IG1pc3Npbmcgd2VpZ2h0IHZhbHVlcyBpcyBzdG9yZWQgYXMgYHlyYnNzLmNvbXBsZXRlYC4KCmBgYHtyfQojcmVtb3ZlIHJvd3Mgd2l0aCBtaXNzaW5nIHZhbHVlcwp5cmJzcy5jb21wbGV0ZSA9IHlyYnNzW2NvbXBsZXRlLmNhc2VzKHlyYnNzJHdlaWdodCksIF0KCiNzZXQgcGFyYW1ldGVycwpzYW1wbGUuc2l6ZSA9IDMwCgojc2V0IHNlZWQgZm9yIHBzZXVkby1yYW5kb20gc2FtcGxpbmcKc2V0LnNlZWQoNTAxMSkKCiNvYnRhaW4gcmFuZG9tIHNhbXBsZSBvZiByb3cgbnVtYmVycwpzYW1wbGUucm93cyA9IHNhbXBsZSgxOm5yb3coeXJic3MuY29tcGxldGUpLCBzYW1wbGUuc2l6ZSkKCiNjcmVhdGUgeXJic3Muc2FtcGxlCnlyYnNzLnNhbXBsZSA9IHlyYnNzLmNvbXBsZXRlW3NhbXBsZS5yb3dzLCBdCmBgYAoKMS4gQSBjb25maWRlbmNlIGludGVydmFsIGlzIGNhbGN1bGF0ZWQgZnJvbSBmb3VyIHF1YW50aXRpZXM6IHRoZSBzYW1wbGUgbWVhbiAkXG92ZXJsaW5le3h9JCwgdGhlIHNhbXBsZSBzdGFuZGFyZCBkZXZpYXRpb24gJHMkLCB0aGUgc2FtcGxlIHNpemUgJG4kLCBhbmQgdGhlIGNyaXRpY2FsICR6JC12YWx1ZSAkel5cc3RhciQuCgphKSBDYWxjdWxhdGUgJFxvdmVybGluZXt4fV97d2VpZ2h0fSQgYW5kICRzX3t3ZWlnaHR9JCwgdGhlIG1lYW4gYW5kIHN0YW5kYXJkIGRldmlhdGlvbiBvZiB3ZWlnaHQgaW4gdGhlIHNhbXBsZS4KICAKVGhlIHNhbXBsZSBtZWFuICRcb3ZlcmxpbmV7eH1fe3dlaWdodH0kIGlzIDY5LjYzIGtnIGFuZCB0aGUgc2FtcGxlIHN0YW5kYXJkIGRldmlhdGlvbiAkc197d2VpZ2h0fSQgaXMgMTguMjYga2cuCgoKYGBge3J9CgpgYGAKCkRvIHlvdSBuZWVkIHRvIGluY2x1ZGUgYG5hLnJtID0gVFJVRWAuIFdoeSBvciB3aHkgbm90PwoKYikgRm9yIGEgOTUlIGNvbmZpZGVuY2UgaW50ZXJ2YWwsICR6XlxzdGFyJCBpcyB0aGUgcG9pbnQgb24gYSBzdGFuZGFyZCBub3JtYWwgZGlzdHJpYnV0aW9uIHRoYXQgaGFzIGFyZWEgMC45NzUgdG8gdGhlIGxlZnQgKGFuZCBhcmVhIDAuMDI1IHRvIHRoZSByaWdodCkuIENhbGN1bGF0ZSB0aGUgdmFsdWUgb2YgJHpeXHN0YXIkIGZvciBhIDk1JSBjb25maWRlbmNlIGludGVydmFsLgoKICAgIApgYGB7cn0KcW5vcm0oMC45NzUpCmBgYAogICAgCiAgICAKICAgIApjKSBDYWxjdWxhdGUgYSA5NSUgY29uZmlkZW5jZSBpbnRlcnZhbCBiYXNlZCBvbiB0aGUgc2FtcGxlZCB3ZWlnaHRzLiBUaGUgcXVhbnRpdHkgJCh6XlxzdGFyIFx0aW1lcyBcZnJhY3tzfXtcc3FydHtufX0pJCBpcyBrbm93biBhcyB0aGUgbWFyZ2luIG9mIGVycm9yLCAkbSQuClxbXG92ZXJsaW5le3h9IFxwbSB6XlxzdGFyIFx0aW1lcyBcZnJhY3tzfXtcc3FydHtufX0gIFxyaWdodGFycm93ICBcbGVmdCggXG92ZXJsaW5le3h9IC0gel5cc3RhciBcZnJhY3tzfXtcc3FydHtufX0sIFwgXG92ZXJsaW5le3h9ICsgel5cc3RhciBcZnJhY3tzfXtcc3FydHtufX0gXHJpZ2h0KSBcXQoKCgpgYGB7ciBldmFsID0gRkFMU0V9CiN1c2UgciBhcyBhIGNhbGN1bGF0b3IKCmNpLmxvd2VyLmJvdW5kID0gCmNpLnVwcGVyLmJvdW5kID0gCgojY29uZmlkZW5jZSBpbnRlcnZhbApjaS5sb3dlci5ib3VuZDsgY2kudXBwZXIuYm91bmQKeC5iYXIKcwpgYGAKCgpkKSBUaGUgc3RhbmRhcmQgZGV2aWF0aW9uIG9mIHdlaWdodCBpbiB0aGUgc2FtcGxlIGlzIDE4LjI2IGtnLiBTdXBwb3NlIHRoYXQgdGhlIHN0YW5kYXJkIGRldmlhdGlvbiBpbiB0aGUgc2FtcGxlIHdlcmUgMjAga2cgb3IgMjUga2csIGJ1dCB0aGF0ICRcb3ZlcmxpbmV7eH0kIGFuZCAkbiQgcmVtYWluIGNvbnN0YW50LiBSZS1ydW4gdGhlIGNhbGN1bGF0aW9uIGZyb20gcGFydCBjKSBhbmQgZGVzY3JpYmUgdGhlIGVmZmVjdCBvZiBsYXJnZXIgKHNhbXBsZSkgc3RhbmRhcmQgZGV2aWF0aW9uIG9uIHRoZSBjb25maWRlbmNlIGludGVydmFsLgogICAgCgoKCgoKYGBge3IgZXZhbCA9IEZBTFNFfQojdXNlIHIgYXMgYSBjYWxjdWxhdG9yIHRvIHNlZSBlZmZlY3Qgb2YgaW5jcmVhc2luZyB2YXJpYWJpbGl0eSBpbiB0aGUgc2FtcGxlIG9uIGNvbmZpZGVuY2UgaW50ZXJ2YWwuIAoKCmNpLmxvd2VyLmJvdW5kID0gCmNpLnVwcGVyLmJvdW5kID0gCgojY29uZmlkZW5jZSBpbnRlcnZhbApjaS5sb3dlci5ib3VuZDsgY2kudXBwZXIuYm91bmQKCiNjb25maWRlbmNlIGludGVydmFsCmNpLmxvd2VyLmJvdW5kOyBjaS51cHBlci5ib3VuZApgYGAKCkxlc3MgdmFyaWFiaWxpdHkgaW4gYSBzYW1wbGUgZ2l2ZXMgYSBtb3JlIHByZWNpc2UgZXN0aW1hdGUgb2YgdGhlIHBvcHVsYXRpb24gcGFyYW1ldGVyLiBMb29raW5nIGF0IHRoZSBmb3JtdWxhLCB0aGUgbWFyZ2luIG9mIGVycm9yIGluY3JlYXNlcyBhcyAkcyQgaW5jcmVhc2VzLCBzbyB0aGUgY29uZmlkZW5jZSBpbnRlcnZhbCBzaG91bGQgZ2V0IHdpZGVyIGFzIHZhcmlhYmlsaXR5IGluIGEgc2FtcGxlIGluY3JlYXNlcy4gCiAgICAKIyMjIEV4ZXJjaXNlIDI6IENoYW5naW5nIGNvbmZpZGVuY2UgbGV2ZWwgCgogSW4gZ2VuZXJhbCwgZm9yIGEgY29uZmlkZW5jZSBpbnRlcnZhbCBvZiAkKDEgLSBcYWxwaGEpKDEwMClcJSQsICR6XlxzdGFyJCBpcyB0aGUgcG9pbnQgb24gYSBzdGFuZGFyZCBub3JtYWwgZGlzdHJpYnV0aW9uIHRoYXQgaGFzIGFyZWEgJDEgLSAoXGFscGhhLzIpJCB0byB0aGUgbGVmdCAoYW5kIGFyZWEgJFxhbHBoYS8yJCB0byB0aGUgcmlnaHQpLiBGb3IgYSA5NSAlIGNvbmZpZGVuY2UgaW50ZXJ2YWwsICRcYWxwaGEgPSAwLjA1JDsgJHpeXHN0YXIkIGlzIHRoZSBwb2ludCBvbiBhIHN0YW5kYXJkIG5vcm1hbCBkaXN0cmlidXRpb24gd2l0aCBhcmVhICQxIC0gKDAuMDUvMikgPSAwLjk3NSQgdG8gdGhlIGxlZnQuCgphKSBDYWxjdWxhdGUgYSA5MCUgY29uZmlkZW5jZSBpbnRlcnZhbCBmb3IgbWVhbiB3ZWlnaHQgYmFzZWQgb24gdGhlIHNhbXBsZSBkYXRhLgogICAgCiAgICAgICAgCiAgICAgIAogICAgCmBgYHtyIGV2YWwgPSBGQUxTRX0KI3VzZSByIGFzIGEgY2FsY3VsYXRvcgoKY2kubG93ZXIuYm91bmQgPSAKY2kudXBwZXIuYm91bmQgPSAKCiNjb25maWRlbmNlIGludGVydmFsCmNpLmxvd2VyLmJvdW5kOyBjaS51cHBlci5ib3VuZApgYGAgICAgCiAgIAogICAKICAgIApiKSBDYWxjdWxhdGUgYSA5OSUgY29uZmlkZW5jZSBpbnRlcnZhbCBmb3IgbWVhbiB3ZWlnaHQgYmFzZWQgb24gdGhlIHNhbXBsZSBkYXRhLgogICAKICAgICAgCiAgICAgICAgCgogICAKYGBge3IgZXZhbCA9IEZBTFNFfQojdXNlIHIgYXMgYSBjYWxjdWxhdG9yCgpjaS5sb3dlci5ib3VuZCA9IApjaS51cHBlci5ib3VuZCA9IAoKI2NvbmZpZGVuY2UgaW50ZXJ2YWwKY2kubG93ZXIuYm91bmQ7IGNpLnVwcGVyLmJvdW5kCmBgYCAgIAogICAKICAgCiAgICAKYykgQ29tcGFyZSB0aGUgOTUlIGNvbmZpZGVuY2UgaW50ZXJ2YWwgY2FsY3VsYXRlZCBpbiB0aGUgcHJldmlvdXMgcXVlc3Rpb24gdG8gdGhlIDkwJSBhbmQgOTklIGNvbmZpZGVuY2UgaW50ZXJ2YWxzLiBEZXNjcmliZSB0aGUgcmVsYXRpb25zaGlwIGJldHdlZW4gY29uZmlkZW5jZSBsZXZlbCBhbmQgd2lkdGggb2YgdGhlIGludGVydmFsLgoKIEFzIGNvbmZpZGVuY2UgbGV2ZWwgaW5jcmVhc2VzLCB0aGUgd2lkdGggb2YgdGhlIGludGVydmFsIF9fX19fX19fX18uCiAgICAKICAgIApkKSBXaGljaCBvZiB0aGUgaW50ZXJ2YWxzIGNhbGN1bGF0ZWQgKDkwJSwgOTUlLCA5OSUpIGRvIHlvdSBmaW5kIHRvIGJlIHRoZSBtb3N0IGluZm9ybWF0aXZlIGFzIGFuIGVzdGltYXRlIG9mIHRoZSBtZWFuIHdlaWdodCBvZiBoaWdoIHNjaG9vbCBhZ2Ugc3R1ZGVudHMgaW4gdGhlIFVTPyBFeHBsYWluIHlvdXIgYW5zd2VyLiAKICAgIAoKICAgIApDb25zaWRlciBhbiBleHRyZW1lIGNhc2UtLS1hIHRoZW9yZXRpY2FsICIxMDAlIiBjb25maWRlbmNlIGludGVydmFsIHRoYXQgY2FwdHVyZXMgdGhlIHBvcHVsYXRpb24gbWVhbiB3aXRoIGFic29sdXRlIGNlcnRhaW50eSB3b3VsZCBiZSAkKDAsICtcaW5mdHkpJCBrZy4gU3VjaCBhIHdpZGUgaW50ZXJ2YWwgaXMgY29tcGxldGVseSB1bmluZm9ybWF0aXZlIGZvciBwcm92aWRpbmcgYW4gcGxhdXNpYmxlIHJhbmdlIG9mIHZhbHVlcyBmb3IgJFxtdSQuCgoKIyMjIEV4ZXJjaXNlIDM6IFVzaW5nIGB0LnRlc3QoKWAgaW4gYFJgIHRvIGNhbGN1bGF0ZSBjb25maWRlbmNlIGludGVydmFscy4gCgpUaGUgYHQudGVzdCgpYCBjb21tYW5kIGNhbiBiZSB1c2VkIHRvIGNhbGN1bGF0ZSBjb25maWRlbmNlIGludGVydmFscy4gRm9yIGV4YW1wbGUsIHRoZSBjb21tYW5kIHRvIGNhbGN1bGF0ZSBhIDk1JSBjb25maWRlbmNlIGludGVydmFsIGZvciBgaGVpZ2h0YCBpbiBgeXJic3MuY29tcGxldGVgIGlzCgpgYGB7ciwgZXZhbCA9IEZBTFNFfQp0LnRlc3QoeXJic3MuY29tcGxldGUkaGVpZ2h0LCBjb25mLmxldmVsID0gMC45NSkkY29uZi5pbnQKYGBgCgoKYSkgQ2FsY3VsYXRlIGEgOTVcJSBjb25maWRlbmNlIGludGVydmFsIGZvciBtZWFuIHdlaWdodCB1c2luZyBgdC50ZXN0KClgLiAKICAgIAogICAgCmBgYHtyIGV2YWwgPSBGQUxTRX0KdC50ZXN0CmBgYAoKVGhlIGFuc3dlciB3aWxsIGRpZmZlciBzbGlnaHRseSBmcm9tIHRoZSBvbmUgaW4gUXVlc3Rpb24gMSBiZWNhdXNlIGBgUmAgY2FsY3VsYXRlcyBjb25maWRlbmNlIGludGVydmFscyB1c2luZyBhIGNyaXRpY2FsIHZhbHVlIGZyb20gdGhlICR0JCBkaXN0cmlidXRpb24gcmF0aGVyIHRoYW4gZnJvbSB0aGUgc3RhbmRhcmQgbm9ybWFsIGRpc3RyaWJ1dGlvbi4gVGhlICR0JCBkaXN0cmlidXRpb24gd2lsbCBiZSBsb29rZWQgaW50byBtb3JlIGRlcHRoIGxhdGVyLiAKCiAgICAKICAgIApiKSBFeGFtaW5lIHRoZSBlZmZlY3Qgb2YgbGFyZ2VyIHNhbXBsZSBzaXplcyBvbiB0aGUgY29uZmlkZW5jZSBpbnRlcnZhbCBieSByZS1ydW5uaW5nIHRoZSBjb2RlIGZvciBzYW1wbGUgc2l6ZXMgb2YgNTAsIDEwMCwgYW5kIDMwMC4gRGVzY3JpYmUgeW91ciBvYnNlcnZhdGlvbnMuCgpgYGB7ciwgZWNobyA9IEZBTFNFfQojY29kZSBmb3Igc2FtcGxlIHNpemUgIG9mIDUwIApgYGAKCmBgYHtyLCBlY2hvID0gRkFMU0V9CiNjb2RlIGZvciBzYW1wbGUgc2l6ZSAgb2YgMTAwCmBgYAoKYGBge3IsIGVjaG8gPSBGQUxTRX0KI2NvZGUgZm9yIHNhbXBsZSBzaXplICBvZiAzMDAgCmBgYAogICAgCkFzIHRoZSBzYW1wbGUgc2l6ZSBpbmNyZWFzZXMsIHRoZSBjb25maWRlbmNlIGludGVydmFscyBiZWNvbWUgX19fX19fLiAKCkludHVpdGl2ZWx5LCBvbmUgd291bGQgZXhwZWN0IHRoYXQgaGF2aW5nIG1vcmUgZGF0YSBmcm9tIHRoZSBwb3B1bGF0aW9uIGFsbG93cyBmb3IgYSBtb3JlIHByZWNpc2UgZXN0aW1hdGUgb2YgdGhlIHBvcHVsYXRpb24gcGFyYW1ldGVyLiBUaGlzIGVmZmVjdCBjYW4gYWxzbyBiZSBwcmVkaWN0ZWQgc2luY2UgYXMgc2FtcGxlIHNpemUgaW5jcmVhc2VzLCBtYXJnaW4gb2YgZXJyb3IgX19fX19fX18uCgoKCiMjIyBFeGVyY2lzZSA0LiBUaGUgaW50ZXJwcmV0YXRpb24gb2YgImNvbmZpZGVuY2UiCgpUaGUgbWV0aG9kIGRpc2N1c3NlZCBmb3IgY29tcHV0aW5nIGFuICR4JSQgY29uZmlkZW5jZSBpbnRlcnZhbCB3aWxsIHByb2R1Y2UgYW4gaW50ZXJ2YWwgdGhhdCAkeCQgdGltZXMgb3V0IG9mIDEwMCAob24gYXZlcmFnZSkgY29udGFpbnMgdGhlIHBvcHVsYXRpb24gbWVhbi4KCmEpIENvbnNpZGVyIHRoZSBpbmRpdmlkdWFscyBpbiBgeXJic3MuY29tcGxldGVgIGFzIHRoZSB0YXJnZXQgcG9wdWxhdGlvbi4gQ2FsY3VsYXRlIHRoZSBwb3B1bGF0aW9uIG1lYW4gd2VpZ2h0LCAkXG11X3t3ZWlnaHR9JC4gCiAgICAKCgpgYGB7cn0KCmBgYAogICAgCmIpIERvZXMgdGhlIDk1JSBpbnRlcnZhbCBjYWxjdWxhdGVkIGluIHBhcnQgYikgb2YgUXVlc3Rpb24gMyBmb3IgJG4gPSAxMDAkIGNvbnRhaW4gJFxtdV97d2VpZ2h0fSQ/CiAgICAKCiAgICAKCiMjIyBFeGVyY2lzZSA1LiBTaW11bGF0aW9uCgpSdW4gdGhlIGZvbGxvd2luZyBjb2RlIHRvIHRha2UgMSwwMDAgcmFuZG9tIHNhbXBsZXMgb2Ygc2l6ZSAxMDAgZnJvbSBgeXJic3MuY29tcGxldGVgLiBGb3IgZWFjaCBzYW1wbGUsIGBSYCBjYWxjdWxhdGVzIG1lYW4gd2VpZ2h0IGZvciBwYXJ0aWNpcGFudHMgaW4gdGhlIHNhbXBsZSBhbmQgc3RvcmVzIHRoZSB2YWx1ZSBpbiB0aGUgdmVjdG9yIGBzYW1wbGUubWVhbnNgLiBUaGUgbWFyZ2luIG9mIGVycm9yICRtJCBpcyBjYWxjdWxhdGVkIGFjY29yZGluZyB0byB0aGUgZGVmaW5lZCBjb25maWRlbmNlIGxldmVsIGFuZCBzdG9yZWQgaW4gdGhlIHZlY3RvciBgbWAuIFRoZSBsb2dpY2FsIHZhcmlhYmxlIGBjb250YWlucy5tdWAgcmVjb3JkcyBgVFJVRWAgaWYgYSBjb25maWRlbmNlIGludGVydmFsIGNvbnRhaW5zICRcbXVfe3dlaWdodH0kIGFuZCBgRkFMU0VgIG90aGVyd2lzZS4KCmBgYHtyLCBldmFsID0gVFJVRX0KI3NldCBwYXJhbWV0ZXJzCnNhbXBsZS5zaXplID0gMTAwCmNvbmYubGV2ZWwgPSAwLjk1CnJlcGxpY2F0ZXMgPSAxMDAwCgojc2V0IHNlZWQgZm9yIHBzZXVkbyByYW5kb20gc2FtcGxpbmcKc2V0LnNlZWQoMjAxNykKCiNjcmVhdGUgZW1wdHkgdmVjdG9ycyB0byBzdG9yZSByZXN1bHRzCnNhbXBsZS5tZWFucyA9IHZlY3RvcigibnVtZXJpYyIsIHJlcGxpY2F0ZXMpCm0gPSB2ZWN0b3IoIm51bWVyaWMiLCByZXBsaWNhdGVzKQoKI2NhbGN1bGF0ZSBzYW1wbGUgbWVhbnMgYW5kIG1hcmdpbnMgb2YgZXJyb3IKZm9yKGsgaW4gMTpyZXBsaWNhdGVzKXsKICAKICBzYW1wbGUucm93cyA9IHNhbXBsZShucm93KHlyYnNzLmNvbXBsZXRlKSwgc2FtcGxlLnNpemUpCiAgCiAgei5zdGFyID0gcW5vcm0oMSAtICgxIC0gY29uZi5sZXZlbCkvMikKICAKICBzYW1wbGUubWVhbnNba10gPSBtZWFuKHlyYnNzLmNvbXBsZXRlJHdlaWdodFtzYW1wbGUucm93c10pCiAgbVtrXSA9IHouc3RhciAqIChzZCh5cmJzcy5jb21wbGV0ZSR3ZWlnaHRbc2FtcGxlLnJvd3NdKSAvIHNxcnQoc2FtcGxlLnNpemUpKQogIAp9CgojZGVmaW5lIHVwcGVyIGFuZCBsb3dlciBib3VuZHMgb2YgY29uZmlkZW5jZSBpbnRlcnZhbApjaS5sYiA9IHNhbXBsZS5tZWFucyAtIG0KY2kudWIgPSBzYW1wbGUubWVhbnMgKyBtCgojZG9lcyB0aGUgY29uZmlkZW5jZSBpbnRlcnZhbCBjb250YWluIG11PwptdSA9IG1lYW4oeXJic3MuY29tcGxldGUkd2VpZ2h0KQpjb250YWlucy5tdSA9IChjaS5sYiA8IG11KSAmIChjaS51YiA+IG11KQp0YWJsZShjb250YWlucy5tdSkKYGBgCgphKSBIb3cgbWFueSBpbnRlcnZhbHMgY29udGFpbiB0aGUgcG9wdWxhdGlvbiBtZWFuICRcbXVfe3dlaWdodH0kPwoKCgogICAgCmIpIFJlLXJ1biB0aGUgc2ltdWxhdGlvbiB3aXRoIGNvbmZpZGVuY2UgbGV2ZWxzIG9mIDAuOTAgYW5kIDAuOTkuIFdoYXQgaGFwcGVucyB0byB0aGUgbnVtYmVyIG9mIGludGVydmFscyB0aGF0IGNvbnRhaW4gJFxtdV97d2VpZ2h0fSQ/CgoKCgoKYykgV2hhdCBpcyBhIGRpc2FkdmFudGFnZSBhc3NvY2lhdGVkIHdpdGggdXNpbmcgYW4gaW50ZXJ2YWwgdGhhdCBnaXZlcyBhIG1vcmUgcHJlY2lzZSBlc3RpbWF0ZSBvZiB0aGUgcGFyYW1ldGVyOyBlLmcuLCBhIDkwJSBpbnRlcnZhbCByYXRoZXIgdGhhbiBhIDk1JSBpbnRlcnZhbD8KCgpBIG1vcmUgcHJlY2lzZSBpbnRlcnZhbCBpcyBsZXNzIGxpa2VseSB0byBjb250YWluIHRoZSBwb3B1bGF0aW9uIHBhcmFtZXRlci4KCmQpIEZyb20gd2hhdCB5b3UgaGF2ZSBvYnNlcnZlZCBmcm9tIHRoZSBleGVyY2lzZXMgYWJvdmUgYWJvdXQgdGhlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIGFuIGludGVydmFsIGVzdGltYXRlICQoXG92ZXJsaW5le3h9IFxwbSBtKSQgYW5kIHRoZSBwb3B1bGF0aW9uIG1lYW4gKCRcbXUkKSwgZXZhbHVhdGUgdGhlIGZvbGxvd2luZyBzdGF0ZW1lbnQ6CiAgICAKIlRoZSA5NSUgY29uZmlkZW5jZSBpbnRlcnZhbCBhcyBjYWxjdWxhdGVkIGZyb20gdGhlIDEzLDU3MiBzYW1wbGVkIGhpZ2ggc2Nob29sIHN0dWRlbnRzIGluIHRoZSAyMDEzIFlSQlNTIHN1cnZleSBpcyAoNjcuNjEsIDY4LjIwKSBrZy4gSXQgaXMgcG9zc2libGUgdG8gZGVmaW5pdGl2ZWx5IGNvbmNsdWRlIHRoYXQgdGhpcyBpbnRlcnZhbCBjb250YWlucyB0aGUgbWVhbiB3ZWlnaHQgb2YgdGhlIDIxLjIgbWlsbGlvbiBoaWdoIHNjaG9vbCBzdHVkZW50cyBpbiB0aGUgVVMgaW4gMjAxMy4iCgoKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCgpUaGlzIGFjdGl2aXR5IGlzIG1vZGlmaWVkIGZyb20gdGhlIE9wZW4gSW50cm8gQmlvc3RhdGlzdGljcyBsYWIgYXQgb3BlbmludHJvLm9yZwoKICAKCiAgICAgICAgCiAgICAgICAg