colt: binomial distribution broken?

[sala]

Hi all

I have a problem with the Binomial Class in Colt. It usually works
fine. However, with small probabilities, it does not work anymore (it
always returns 0).

e.g.

int s = Binomial.staticNextInt(11,0.00009);

According to the distribution function, the probability of having
excactly one succes is about 0.001 (so it should return a 1 in about
a thousand cases).

However, it never returns anything other than 0, even if I run it
millions of times.

When I increase the p value 10 fold (to 0.0009), it works fine.

Might there be a precision bug?

Also, the official COLT email list does not seem to work. Hoschke
himself (the author of COLT) does not seem to reply to emails.

Can anybody help?

Thanks a lot!

 

[iandjmsmith]

Hi all

I have a problem with the Binomial Class in Colt. It usually works
fine. However, with small probabilities, it does not work anymore (it
always returns 0).

e.g.

int s = Binomial.staticNextInt(11,0.00009);

According to the distribution function, the probability of having
excactly one succes is about 0.001 (so it should return a 1 in about
a thousand cases).

However, it never returns anything other than 0, even if I run it
millions of times.

When I increase the p value 10 fold (to 0.0009), it works fine.

Might there be a precision bug?

Also, the official COLT email list does not seem to work. Hoschke
himself (the author of COLT) does not seem to reply to emails.

Can anybody help?

Thanks a lot!

I have had a quick look at the source and you are correct. It will
always return 0 in these circumstances.

I am not very sure what you want anyone to do. Give you a fix for the
code, suggest a workaround, suggest another library...?

A sensible workaround in this case would be to generate a poisson
variable with mean 11*0.00009.


I also had a quick look in probability.java and spotted

static public double beta(double a, double b, double x ) {
return Gamma.incompleteBeta( a, b, x );
}

and

static public double betaComplemented(double a, double b, double x ) {
return Gamma.incompleteBeta( b, a, x );
}

I think the second function should be

static public double betaComplemented(double a, double b, double x ) {
return Gamma.incompleteBeta( b, a, 1.0 - x );
}

and to be honest passing in the value 1.0 - x is asking for trouble
when x is small.

It would seem very surprising that nobody has reported this problem so
maybe I am having a bad day and I have made some simple mistake(s). If
not, I would be a bit nervous about using some of this software without
checking it out first.

Ian Smith

 

[Chris Uppal]

I have a problem with the Binomial Class in Colt. It usually works
fine. However, with small probabilities, it does not work anymore (it
always returns 0).

What do you mean by "it does not work anymore" ? Do you mean that it worked as
you expect on one version of the JDK/JRE but has stopped working in a later one
? Or that it works on one vendor's JVM but not on another's ? Or just that it
worked fine for your old code, but now you are doing something different with
it, it shows a bug that you hadn't seen before ?

If it's one of the first two then maybe someone can suggest something. If it's
the last possibility then you may need an expert in numerical methods rather
than a bunch of Java hackers ;-)

-- chris

 

[sala]

Ian,

I am not very sure what you want anyone to do. Give you a fix for the
code, suggest a workaround, suggest another library...?

Anything will do. As long as it solves the problem

A sensible workaround in this case would be to generate a poisson
variable with mean 11*0.00009.

Agreed, but since I don't know why this problem occurs, I would always
have to use poisson, which is not a solution that would make me very
happy (for obvious mathematical reasons).

It would seem very surprising that nobody has reported this problem so
maybe I am having a bad day and I have made some simple mistake(s). If
not, I would be a bit nervous about using some of this software without
checking it out first.

Totally agree. What makes me even more nervous is that there seems to
be no working mailing list, and the author of the library does not
answer to my emails.

 

[sala]

chris,

sorry for being so unspecific. I mean that it works for ceratin sets of
parameters, but then suddenly, for another set of parameters, it does
not work anymore. So, it's the last possibility that you mentioned.

And yes, if anybody could provide a working code for generating
binomial random numbers, that would fit all my needs ;-)

(On a more serious sidenote, I am slightly nervous that the most used
scientific library in Java has fundemental bugs like that...)

 

[Googmeister]

chris,

sorry for being so unspecific. I mean that it works for ceratin sets of
parameters, but then suddenly, for another set of parameters, it does
not work anymore. So, it's the last possibility that you mentioned.

And yes, if anybody could provide a working code for generating
binomial random numbers, that would fit all my needs ;-)

(On a more serious sidenote, I am slightly nervous that the most used
scientific library in Java has fundemental bugs like that...)

Yes, that's pretty embarrassing. I suspect the code uses
some kind of approximation to the binomial distribution which
breaks down for certain settings of the parameters (e.g., when
np is small).

Here's another prob-stat library.

http://opsresearch.com/OR-Objects/api/drasys/or/prob/package-summary.html

I offer no guarantees on its correctness, but it does return 1 every
once
in a while for the parameters you specified.

 

[Thomas Weidenfeller]

And yes, if anybody could provide a working code for generating
binomial random numbers, that would fit all my needs ;-)

If you don't find code, you could implement this:

http://statistik.wu-wien.ac.at/papers/92-04-07.wh.ps.gz

(On a more serious sidenote, I am slightly nervous that the most used
scientific library in Java has fundemental bugs like that...)

It hasn't been updated in four years. That looks as if it has been
abandoned.

/Thomas