Newsgroup sci.stat.math 11674

Directory

ahmed shabbir  in
 asks:
> How can I estimate (by sampling) upper and lower bounds
> of the variance of a random variable with unknown distribution.
The variance is a parametric statistic. If you know nothing about the
distribution then you cannot set exact confidence intervals. Therefore you
have two choices:
(1) Adopt a non-parametric measure of spread, say the interquartile range,
and set exact confidence intervals for any distribution; or
(2) Make some assumptions about your distribution, say that the tails
beyond 1% probability are exponential or thinner, then set worst-case
confidence intervals. Obviously, the stronger the assumptions you are
willing to make the tighter your confidence bands can be.
Aaron C. Brown
New York, NY

markwa@halcyon.com (Mark Walsen) in <568im4$4702@nwnews.wa.com> writes:
> I'm developing music software that "listens" to music (MIDI)
> and transcribes it into music notation.  One of the problems
> is determining what the meter of the song is, eg, 2:4, 3:4,
> 4:4, 6:8, etc. . . . I examine about 10 scalar variables that
> summarize the character of the song. . . . There tends to be
> clustering of points in the 10-dimensional space for songs
> with the same meter. . . . If you give me a new song, I'll
> measure its 10 variables related to meter, and locate the
> point for this song in the 10-dimension space. If the point
> falls clearly within one of the "shapes", then I'm confident
> in predicting its meter as being the same as the other songs
> already in that shape.  If the point falls between shapes, or
> if the point falls in an area where shapes overlap, then I have
> to do some guess work.  I need an appropriate statistical
> tool (math) for the guess work.  Some kind of cluster analysis.
"Cluster Analysis" usually refers to the problem of assigning data points
to shapes, "Discriminant Analysis" is the term for deciding which shape a
new point belongs to. You do not have a cluster analysis problem because
you know the meters of your original data.
Discriminant analysis is application-specific. There are some
general-purpose algorithms but I doubt they wil work for a subtle problem
like this.
I would be inclined to simply find the closest song and assign its meter
to the unknown song. After all, you don't really care if this song is like
the average of all 4:4 songs; you just want to know if it's similar to any
4:4 songs.
You could run a quick experiment and see what percentage of your data
songs are nearest to another song of the same meter. If it's a reasonably
high number, this method might work. You can rescale your dimensions to
improve the percentage as well.
If it's not a reasonably high number then you have a harder problem. I
still suspect the solution will be essentially local, for example you
might want to have a "vote" of all songs within a certain distance, or a
certain multiple of the minimum distance, or the ten closest songs.
Aaron C. Brown
New York, NY

Andrew,
I think a median test would work for you.  Judging from your data, all 5 
in one group are greater than the median of the combined groups.  In the 
other group, 5 are below the median.  Conover or most nonparametric texts 
will discuss the "asymptotic" distribution, but you might find a 
reference for the "exact" distribution since you have equal numbers in 
each group.  Conover discusses the exact distribution, but I am not sure 
whether it is implemented in any stat package.

jksnyder@aol.com in <19961108025700.VAA15372@ladder01.news.aol.com> wrote:
> Is there a difference between chaotic systems and random systems?
> If so, could the difference be measured/quantified by plotting the
> series on normal probability paper?
The previous responses have treated this as a question in mathematical
philosophy, in which case I think the answer is "yes, but it's very
subtle."
However as a practical matter chaotic systems appear random. Unless you
can exploit the chaotic nature of the system, you might as well regard it
as random.
For example, suppose you have a data series generated from the logistic
equation with a chaotic parameter. This would appear to be random and
could be analyzed using time series methods. However if you do a
two-dimensional plot of successive points you will see a clear pattern.
Now you can make exact predictions. So this series is clearly chaotic
rather than random.
Now suppose you have the same data measured with some random noise and
only sampled at every third point. In some cases you will do better
treating this series as random, in other cases as deterministic with
noise. It depends on the amount of noise, the sampling frequency and your
application.
The probability plot does not shed any light on the matter. Chaotic data
sets will have a distribution, there is no way to tell if it is the result
of randomness or determistic effects.
Aaron C. Brown
New York, NY

En 1997, les traditionnelles Journ'ees de Statistique auront lieu
a Carcassonne entre le 26 et le 30 mai 1997. R'eunissant chaque 
ann'ee chercheurs, enseignants et praticiens de la Statistique, 
ces journ'ees constituent la plus importante manifestation 
scientifique du monde francophone.
Elles sont organis'ees par l'Association pour la Statistique et 
ses Utilisations et parrain'ees par la Soci'et'e Statistique de 
France, la Soci'et'e de Statistique de Paris, le groupe Mod'elisation 
Al'eatoire et Statistique de la Soci'et'e de Math'ematiques 
 Appliqu'ees et Industrielles, le Groupe des Membres Francais 
de l'Institut International de Statistique, la Soci'et'e Francaise 
de Biom'etrie ainsi que la Soci'et'e Francophone de Classification.
Toutes les informations concernant, les th`emes, les conf'erences 
invit'ees, les tables rondes, la soumission de communications,
les r'eservations... sont disponibles sur le serveur web :
-->     http://www-sv.cict.fr/asu97
ou peuvent ^etre demand'ees par courrier 'electronique `a 
l'adresse :
-->     asu97@cict.fr
Philippe Besse
Michel Meste

Warren  wrote:
>Andrew,
>I think a median test would work for you.  Judging from your data, all 5 
>in one group are greater than the median of the combined groups.  In the 
>other group, 5 are below the median.  Conover or most nonparametric texts 
>will discuss the "asymptotic" distribution, but you might find a 
>reference for the "exact" distribution since you have equal numbers in 
>each group.  Conover discusses the exact distribution, but I am not sure 
>whether it is implemented in any stat package.
>
Andrew,
Looking at it again just now, I am pretty sure the exact median test is 
just Fisher's exact test.  For your data, you set up a two by two table:
          Sample
       1         2     Total
GT Md  5         0       5
LT Md  0         5       5
Total  5         5       10
and compute Fisher's exact for a one-tailed hypothesis...most stat packs 
will do this for 2x2's.  For your data, group 1 has a higher median than 
group 2.  Hope this helps.
Warren.      

I am embarked on a course of self study using Maddala's book and am
seeking the instructor's manual for answers to the exercises.
I have been told by Prentice Hall that this volume is no longer in
print. Can anyone help?
Thanks,
Ed

I am making available on the web a draft version of my latest
work.  The abstract follows:
        Selection of Distance Metrics and Feature Subsets 
                 for k-Nearest Neighbor Classifiers
                               ABSTRACT
   The k-nearest neighbor (kNN) classifier is a popular and
   effective method for associating a feature vector with
   a unique element in a known, finite set of classes.  A common
   choice for the distance metric used in kNN classification
   is the quadratic distance Q(x,A,y) = (x-y)' A (x-y), where x 
   and y are n-vectors of features, A is a symmetric n by n matrix, 
   and prime denotes transpose.  For finite sets of training samples 
   the choice of matrix A is important in optimizing classifier 
   performance.  We show that A can be approximately optimized via 
   gradient descent on a sigmoidally smoothed estimate of the classifier's
   probability of error.  We describe an algorithm for performing the 
   metric selection, and compare the performance of our method with that 
   of other methods.  We demonstrate that adding noise during the 
   descent process can reduce the effects of overfitting.  We further 
   suggest how feature subset selection can be treated as a special 
   case of this metric selection.  
A postscript version can be obtained from 
    http://www.cs.virginia.edu/~alb/techrep.html
or
    http://www.cs.virginia.edu/~alb/techrep/techrep.html
[I am making the draft version available partly because I am currently
on a hunger strike to protest the severe harassment I have been subjected
to.  Details of this matter are available in another part of my 
home pages.]
--
Allen L. Barker
http://www.cs.virginia.edu/~alb

In article <3288E748.6D2A@ramat-negev.org.il>, howarda@ramat-negev.org.il 
says...
>
>I am very interested in the subject of CPK.
>We are requested by our customers - we are a custom plastics injection
>company- to provid eproof of capability.
>We are requested to show cpk greater than 1.66 and sometimes 
>
>Sometimes we have to show this for performance tests that have just a
>minimum or maximum value.
>We have tried to explain that this is not a reasonable demand because of
>the one sidedness of the tolerances.
>Can any one help me to conviencingly persuade our customers that this is
>an unreasonable demand.
As I understand it, even if a two sided limit exists, the calculation of Cpk 
is made against only one of those limits.  (The one toward which the mean is 
biased.)
Cpk = Minimum(Upper Spec. Limit - xbar, xbar - Lower Spec. Limit) / 3s  where 
xbar is the sample mean and s is the sample standard deviation.
Therefore, the fact that you have a one sided limit does not preclude 
calculating a useful Cpk value.
>-- 
>Atkins Family, Kibbutz Revivim, D.N. Halutza, 85515, ISRAEL
>howarda@ramat-negev.org.il.
>http://www.ramat-negev.org.il/~howarda/
-- 
Mike Aupperle            m-aupperle@ti.com 
        71541.2164@compuserve.com

Lou Pecora (pecora@zoltar.nrl.navy.mil) wrote:
: In article <3287777C.73D8@colorado.edu>, dodier@colorado.edu wrote:
: > 
: > So I've pointed out that random processes can be low-dimensional, but 
: > I could make the argument a little more convincing by coming up with 
: > some examples of deterministic system which has an everyday distribution
: > as its invariant measure. So far I can't think of a low-dimensional
: > system
: > which has an invariant measure which is approximately normal, say. 
: > Can anyone name such a system?
: Hello, Robert,
: I'm not sure you and Troy are using the word dimension the same, but I was
: under the impression that any random process could not be embedded in a
: finite dimensional phase space (a la time delay embeddings).  I think
: that's what Troy meant (Troy, is that right?).  Is there a random process
: than can be embedded in a finite-dimensional phase space?  
Depends if you allow "stochastic" evolution rules or not.  If you require
'deterministic' evolution for your stochastic system, then no embedding can
turn water in to wine. 
More interesting would be the reconstruction whose "stochastic equations
of motion" are equivalent to the original.
Take for instance a 2-d stochastic map, e.g. the probability density of
the future state \rho(x(n)) = F(x(n),x(n-1)) depends on function of the past
observed value.  If you didn't consider past values, you might be tempted
to model the system as a 'zero-embedding dimension' IID process, i.e.
\rho(x(n)) = G(\x(n))
just a histogram.   But this is in some sense "wrong", because there is
more information that you haven't correctly discerned, and the observed data
stream is not IID.
I believe something relating to this concept is called "model order" by
the statisticians. 
But I don't know anything analogous to the arbitrary diffeomorphisms which
preserve 'equivalence' of deterministic dynamical systems. 
: Opinions/theorems, anyone?
: Lou Pecora
: code 6343
: Naval Research Lab
: Washington  DC  20375
: USA
:  == My views are not those of the U.S. Navy. ==
: ------------------------------------------------------------
:   Check out the 4th Experimental Chaos Conference Home Page:
:   http://natasha.umsl.edu/Exp_Chaos4/
: ------------------------------------------------------------
--
Matthew B. Kennel/mbk@caffeine.engr.utk.edu/I do not speak for ORNL, DOE or UT
Oak Ridge National Laboratory/University of Tennessee, Knoxville, TN USA/ 
  I would not, could not SAVE ON PHONE,    |==================================
  I would not, could not BUY YOUR LOAN,    |The US Government does not like
  I would not, could not MAKE MONEY FAST,  |spam either.  It is ILLEGAL!
  I would not, could not SEND NO CA$H,     |USC Title 47, section 227
  I would not, could not SEE YOUR SITE,    |p (b)(1)(C) www.law.cornell.edu/
  I would not, could not EAT VEG-I-MITE,   | /uscode/47/227.html
  I do *not* *like* GREEN CARDS AND SPAM!  |==================================
               M A D - I - A M!

Andrew (morgan@biosci.uq.edu.au) wrote:
: Here is Some More info on the Analysis:
: The enzyme measurement was quantitative, measuring the specific amount of
: enzyme produced
: Here is a sample of the data (each seedling gave one value):
: Treated enzyme activity: 107.0, 83.9, 106.9, 680.7 & 38.25
: Untreated enzyme activity: 26.9, 3.8, 20.8, 37.5 & 15.1 
: The values would appear to show an induction of the gene. but the
: heteroscedastic t-test (i assume this test is valid) shows that the
When the data need to be transformed, then that test is not particularly
good.  That warning seems to fit here.
Since you are looking at a biochemical measure, the log transformation
is a natural possibility.  There are no zeros, and the range of scores
is 100+fold, so the log transform is further supported as a good idea,
without no problem in trying it (i.e., no zero).
When you try a log transform,  then 
  there is no remaining heterogeneity of variance, and 
  there is a difference in the t-test between groups.
Rich Ulrich, biostatistician              wpilib+@pitt.edu
Western Psychiatric Inst. and Clinic   Univ. of Pittsburgh

I am conducting an experiment where I have a nested variable.  There are 4
factors A,B,C,D ...  D is nested within C.  3 levels of each factor.  This
would mean 81 experimental runs.  
I want to run less trials, but still estimate 2 factor interactions.  Does
anyone know a text that I can refer to.  The Montgomery book does not
address nested designs thouroughly enough. 

> 
> CpK for industrial SPC. open limits?
> 
> Howard Atkins 
> Tue, 12 Nov 1996 13:08:24 -0800
> howarda
> 
> Newsgroups:
> sci.stat.math
> References:
> <5583f7$1v7@news.vcd.hp.com> 
> <19961106204100.PAA23871@ladder01.news.aol.com> 
> <32820747.143D@kodak.com> <01bbcfc9$78872700$f1953ccf@qua9190> 
> <32873780.2D1C@kodak.com>
> 
> I am very interested in the subject of CPK.
> We are requested by our customers - we are a custom plastics injection
> company- to provid eproof of capability.
> We are requested to show cpk greater than 1.66 and sometimes 
> Sometimes we have to show this for performance tests that have just a
> minimum or maximum value.
> We have tried to explain that this is not a reasonable demand because 
of
> the one sidedness of the tolerances.
> Can any one help me to conviencingly persuade our customers that this 
is
> an unreasonable demand.
> -- 
> Atkins Family, Kibbutz Revivim, D.N. Halutza, 85515, ISRAEL
> howarda@ramat-negev.org.il.
> http://www.ramat-negev.org.il/~howarda/
A one-sided tolerance does not necessarily preclude the use of Cpk. 
Cpk is normally calculated for both the upper and lower sides of the 
tolerances, taking the minimum to be Cpk.
For a one sided tolerance, compute only the Cpk-upper for a maximum 
tolerance or Cpk-lower for a minimum tolerance. Since you have only one 
tolerance, don't worry about the other side or taking the minimum of the 
two.  Just use the one you do have as your Cpk value.
Beth Clarkson

ahmed shabbir wrote:
> 
> Hi,
> 
> How can I estimate (by sampling) upper and lower bounds of the variance
> of a random variable with unknown distribution.
> 
> Thanks in advance,
> 
> SHABBIR
> ______________________________________________________________________________
>   SHABBIR AHMED                       | 140 MEB, MC-244, 1206 W.Green st.,
>   Operations Research Lab.            | Urbana, IL61801.
>   Dept. of Mech.&Ind.; Eng.            | Ph:(217) 367-5073(home),333-0699(off)
>   Univ.of Illinois @ Urbana-Champaign | 
>                                       | 
> ______________________________________________________________________________
If you have a big enough sample, around 30, so that you can use
normal approximation, you can say the sum over i of (Xi-Xbar)^2/(n-1)
is approximately  sigma^2 times a variable that is chi square on
 n-1 degrees of freedom. Call it s^2.
 Since a chi square on k df has mean k and variance 2*k,
P(n-1-1.96*sqrt(2*(n-1)) <= s^2/sigma^2 <=n-1+1.96*sqrt(2*(n-1)) =~ .95.
Then an approximate 95% confidence interval for sigma^2 is
(s^2/(n-1+1.96*sqrt(2*(n-1)), s^2/(n-1-1.96*sqrt(2*(n-1)).

In article <19961112154800.KAA27005@ladder01.news.aol.com>,
   aacbrown@aol.com wrote:
>jksnyder@aol.com in <19961108025700.VAA15372@ladder01.news.aol.com> 
wrote:
> 
>> Is there a difference between chaotic systems and random systems?
>> If so, could the difference be measured/quantified by plotting the
>> series on normal probability paper?
>
>The previous responses have treated this as a question in 
mathematical
>philosophy, in which case I think the answer is "yes, but it's very
>subtle."
>
>However as a practical matter chaotic systems appear random. Unless 
you
>can exploit the chaotic nature of the system, you might as well 
regard it
>as random.
I have a naive question:
If chaotic systems can have rapidly diverging orbits just by changing 
the initial condidions, say out in the 50th decimal place, then how 
do we obtain information about chaotic behavior by computation? 
(i.e., things like regions of stability, etc.) Don't we have to be 
able to compute orbits to infinite accuracy in order to study chaotic 
behavior numerically?
Thanks,
Ray
>
>
>Aaron C. Brown
>New York, NY
------------------------------------------------------------------------------
R.G. Vickson
Department of Management Sciences
University of Waterloo
(519) 888-4729

> ahmed shabbir wrote:
>
> Hi,
>
> How can I estimate (by sampling) upper and lower bounds of the variance
> of a random variable with unknown distribution.
>
> Thanks in advance,
>
> SHABBIR
> ______________________________________________________________________________
>   SHABBIR AHMED                       | 140 MEB, MC-244, 1206 W.Green st.,
>   Operations Research Lab.            | Urbana, IL61801.
>   Dept. of Mech.&Ind.; Eng.            | Ph:(217) 367-5073(home),333-0699(off)
>   Univ.of Illinois @ Urbana-Champaign | 
>                                       | 
> ______________________________________________________________________________
 If you have a big enough sample, around 30, so that you can use
 normal approximation, you can say the sum over i of (Xi-Xbar)^2
 is approximately  sigma^2 times a variable that is chi square on
 n-1 degrees of freedom. Call it Y.
 Since a chi square on k df has mean k and variance 2*k,
 P(n-1-1.96*sqrt(2*(n-1)) <= Y/sigma^2 <=n-1+1.96*sqrt(2*(n-1)) =~ .95.
 Then an approximate 95% confidence interval for sigma^2 is
 (Y/(n-1+1.96*sqrt(2*(n-1)), Y/(n-1-1.96*sqrt(2*(n-1)).
Please ignore the previous version. Y, *not* Y/(n-1) is approximately
sigma^2 times a chi square on n-1 df.

In Japan, many researchers pronounce LaTeX as "latef." Is it correct? How do you 
pronounce TeX and LaTeX actually, especially in the united states?

One takes the minimum of 
Cpk = Minimum(Upper Spec. Limit - xbar, xbar - Lower
Spec. Limit) / 3s  where 
xbar is the sample mean and s is the sample standard
deviation.
If the requirement is 0-5 Dan the best is 0 Dan . If the mean is 0.2
then one should take the mean -0. This gives a "bad" cpk which is the
opposite of the requirements in fact.
To take 5 is not according to the rules but gives the truer result.
-- 
Atkins Family, Kibbutz Revivim, D.N. Halutza, 85515, ISRAEL
howarda@ramat-negev.org.il.
http://www.ramat-negev.org.il/~howarda/

Bonferroni's Method constructs the individual confidence 
interval for k simultaneous stataments, using an individual statement 
confidence coefficient result of dividing the indivdual statemenet 
confidence coefficient by the k number of simultaneous statements.
	This procedure ensures that this confidence coefficient is 'at 
least' the true joint confidence coefficient for the k simultaneous 
statements.
	My question is: 'at least' means that the true joint confidence 
coefficient for the k simultaneous stataments, should be lower than the 
result of dividing the indivdual statemnet confidence coefficient by the 
k number of simultaneous statements?. If so: How much lower should it 
be?
Thank you anyway!!

I've got this problem with Ornstein Uhlenbeck processes:
compute the distribution of the first passage time to a given level b.
The OU process is the solution of the SDE dX=K(a-X)dt+sdW
Is it well known?
Easy with standard machinery?
Hard?
Any idea will be wellcome.
Thank you very much.