Jame said:
Very nice Yakk. I like it a lot.
Where does the 1.96 standard deviation come form?
1.96 standard deviations is the radius of the mean-centered 95% confidence interval.
It comes from integrating e^(x^2) (I think that is the equation). Practically, people just use tables, because there isn't any simple closed-form formula for it.
Ie, the area under the curve:
e^(x^2)
between
x = -1.96
and
x = 1.96
is about .95, which represents 95% of the distribution. (There may be a multiplcation by Pi I'm missing somewhere, but you get the drift).
What's the sample size needed to achieve ± 3% confidence? ±1% confidence? Since Wiz confirmed it's a straight line all I need is two data points with a ton of samples to really pinpoint it.
I don't know "student's T"[1] distrubution well enough -- that is the one that will actually give you this result. And I'm too lazy to learn it. =)
But, "student's T" approaches the normal curve, so I'll pretend it is the normal curve, and give you a possibly incorrect answer.
It is somewhat a function of p (the probability in question).
Var = p*(1-p)*n
SD = sqrt(Var)
95% confidence = SD * 1.96
accuracy = 1.96 * SD / n
crunch crunch crunch...
The highest value p(1-p) can be, for p from 0 to 1, is at p=0.5. At this point, p(1-p) is 1/4. (thie higher p(1-p) is, the more samples you need. I'll assume it is maximal to give you a lower bound on the number of samples you need.)
Let's use this to find a sufficient number of samples:
accuracy = 1.96 * sqrt(0.25 * n)/n = 1.96 * sqrt(0.25/n)
if we want a 0.01 accuracy:
0.01 = 1.96 sqrt(0.25/n)
0.0001 = 3.84 * 0.25/n
n = 3.84 * 0.25 * 10000
n = 9604 for +/- 1%
for +/- 3%, n = 1067 or so.
To simplify:
1.96 * 1.96 is about 4.
so we end up with:
accuracy = 1/n^2
or
n = 1/accuracy^2
Rough chart of Accuracy vs Samples:
0.05 is 400 samples (5% accuracy requires about 400 samples)
0.04 is 625 samples
0.03 is 1111 samples (3% accuracy requires about 1000 samples)
0.02 is 2500 samples
0.01 is 10000 samples (1% accuracy requires about 10,000 samples)
0.005 is 40000 samples
0.004 is 62500 samples (0.4% accuracy requires about 60,000 samples)
0.002 is 250000 samples
0.001 is 1000000 samples (0.1% accuracy requires about 1 million samples)
You may be able to do better in specific situations. And I don't know how tightly the student's T compares to the normal curve -- however, odds are I've included enough fudge factors to make up for any difference, however.
Given two test a and b points with p_a and p_b, how accurate will the slope be?
slope:
(p_b-p_a)/(b-a)
but we only know p_b and p_a to within an error amount "err".
so the slope error ends up being:
slperr = 2*err/(b-a)
Lets start out with the assumption that p changing by 2% everty 10 cha is about correct.
That is a slope of 0.2% for every cha.
slperr/slp = 1000 * err /(b-a)
set our test points to a = 80 b = 280 gives a b-a of 200
slperr/slp = 5 * err
as a fraction of the calculated slope.
If our original points are accurate to within +/-3% raw chance (about 1000 samples), then we could be confident our slope is accurate to within +/- 15% of the calculated slope.
If you do 10000 samples at both 80 and 280 cha, and you end up with a value of 2% per 10 cha, you can be confident that the result is between 1.9% and 2.1% for every 10 cha (5% of 2% is 0.1%, so +/-0.1%).
10,000 samples is a 8 hour parse run using a bard autosinging every 3 seconds (or do they only autosing every 6? if so, 16 hours).
Using a = 100 and b = 280 would only increase the 20000 sample error to 1.89% to 2.11%, so don't worry about it if you can't hit 80 cha.
[1] cute story: the distribution is called "student's T" because the inventor of the distribution was working for a (beer? I think) company who wouldn't let him publish his statistics work under his own name. So he published his paper under the name "student".
If I remember right, student's T is the difference between a random sampling of a normal population and the theoretical normal distribution. It is how close your
sample's mean and variance get to the "real" mean and standard deviation as you increase the number of samples.
