simple regex benchmark inaccuracies

[it_says_BALLS_on_your forehead]

these are the results:

smmk94<WSP1>$ try_regex2.pl
Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
greedy: 0 wallclock secs ( 1.02 usr + 0.00 sys = 1.02 CPU) @
1235595.10/s (n=1260307)
stingy: 1 wallclock secs ( 1.09 usr + 0.00 sys = 1.09 CPU) @
1172432.11/s (n=1277951)
Rate stingy greedy
stingy 1172432/s -- -5%
greedy 1235595/s 5% --
smmk94<WSP1>$ try_regex2.pl
Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
greedy: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @
1180321.70/s (n=1251141)
stingy: 1 wallclock secs ( 1.07 usr + 0.01 sys = 1.08 CPU) @
1261175.00/s (n=1362069)
Rate greedy stingy
greedy 1180322/s -- -6%
stingy 1261175/s 7% --
smmk94<WSP1>$ try_regex2.pl
Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
greedy: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @
1180321.70/s (n=1251141)
stingy: 2 wallclock secs ( 1.05 usr + 0.00 sys = 1.05 CPU) @
1260306.67/s (n=1323322)
Rate greedy stingy
greedy 1180322/s -- -6%
stingy 1260307/s 7% --
smmk94<WSP1>$ try_regex2.pl
Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
greedy: 1 wallclock secs ( 1.16 usr + 0.00 sys = 1.16 CPU) @
1248870.69/s (n=1448690)
stingy: 2 wallclock secs ( 1.10 usr + 0.00 sys = 1.10 CPU) @
1137400.91/s (n=1251141)
Rate stingy greedy
stingy 1137401/s -- -9%
greedy 1248871/s 10% --



and here is the code:
=====
#!/usr/local/bin/perl

use strict; use warnings;
use Benchmark qw(cmpthese);

sub alt {
my $input =
'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';
return 1 if $input =~ /campaign\w*_fulltop\.gif/;
}

sub alt2 {
my $input =
'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';
return 1 if $input =~ /campaign\w*?_fulltop\.gif/;
}

cmpthese(-1, {
greedy => \&alt,
stingy => \&alt2,
});



....any comments on why there's a flipflop?

 

[xhoster]

these are the results:

smmk94<WSP1>$ try_regex2.pl
Benchmark: running greedy, stingy, each for at least 1 CPU seconds...
greedy: 0 wallclock secs ( 1.02 usr + 0.00 sys = 1.02 CPU) @
1235595.10/s (n=1260307)
stingy: 1 wallclock secs ( 1.09 usr + 0.00 sys = 1.09 CPU) @
1172432.11/s (n=1277951)
Rate stingy greedy
stingy 1172432/s -- -5%
greedy 1235595/s 5% --

What version of Benchmark are you running?

....

...any comments on why there's a flipflop?

What is there to say? The difference between them is negligible, the
results of Benchmark are subject to random error, systematic error, and
correlation between errors--just like every other measurement is--and you
are not running them long enough to average out even the error components
that can be averaged out.

Xho

 

[it_says_BALLS_on_your forehead]

What version of Benchmark are you running?

$VERSION = 1.00;

with Perl 5.6.1

...


What is there to say? The difference between them is negligible, the
results of Benchmark are subject to random error, systematic error, and
correlation between errors--just like every other measurement is--and you
are not running them long enough to average out even the error components
that can be averaged out.

perhaps, although the disparity seems too large to attribute to those
errors. i could understand if the difference was within 5%, but in one
case, greedy was 10% faster than stingy, and in the other, stingy was
7% faster than greedy. i understand that there are many factors
involved, but still i had hoped that the results from Benchmark would
prove more reliable and consistent. FYI, i ran again with -5 as the
first param to cmpthese, and got these results:

smmk94<WSP1>$ try_regex2.pl
Benchmark: running greedy, stingy, each for at least 5 CPU seconds...
greedy: 28 wallclock secs ( 5.16 usr + -0.02 sys = 5.14 CPU) @
1219724.32/s (n=6269383)
stingy: 19 wallclock secs ( 5.09 usr + 0.01 sys = 5.10 CPU) @
1334927.45/s (n=6808130)
Rate greedy stingy
greedy 1219724/s -- -9%
stingy 1334927/s 9% --

 

[ednotover]

these are the results:

stingy 1172432/s -- -5%
greedy 1235595/s 5% --

greedy 1180322/s -- -6%
stingy 1261175/s 7% --

...any comments on why there's a flipflop?

Probably because a large portion of what you're comparing is my $input
= 'blah'.

Ed

 

[it_says_BALLS_on_your forehead]

Probably because a large portion of what you're comparing is my $input
= 'blah'.

Ed

the input is a real data record, and since that is constant, would not
affect the results of the benchmark. Xho probably has the right of it,
although i still have my doubts. i don't think that the Benchmark
module can be completely absolved of responsibility for the
discrepancies in the results...

 

[ednotover]

the input is a real data record, and since that is constant, would not
affect the results of the benchmark. Xho probably has the right of it,
although i still have my doubts. i don't think that the Benchmark
module can be completely absolved of responsibility for the
discrepancies in the results...

If you try it, you'll see. If you run a comparison of your tested
function calls without the my declaration against those with the my
declaration, the functions run 60% faster. You can see below that the
RE resolutions are basically the same, whereas the exact same function
(with a my declaration) yields the type of flip flop you're seeing.
I'd assume the flip flop simply reflects the variances of memory
management (for handling the my declarations) on the machine at the
time of execution.

Ed


$ cat checkit.pl
#!/usr/local/bin/perl

use strict; use warnings;
use Benchmark qw(cmpthese);

sub my_call {
my $input =
'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';
}

cmpthese(-5, {
once => \&my_call,
twice => \&my_call,
});

my $input =
'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';

sub greedy {
return 1 if $input =~ /campaign\w*_fulltop\.gif/;
}

sub stingy {
return 1 if $input =~ /campaign\w*?_fulltop\.gif/;
}

cmpthese(-5, {
greedy => \&greedy,
stingy => \&stingy,
});
$ ./checkit.pl
Benchmark: running once, twice, each for at least 5 CPU seconds...
once: 6 wallclock secs ( 5.55 usr + 0.00 sys = 5.55 CPU) @
2945391.71/s (n=16346924)
twice: 5 wallclock secs ( 5.24 usr + 0.00 sys = 5.24 CPU) @
3180811.45/s (n=16667452)
Rate once twice
once 2945392/s -- -7%
twice 3180811/s 8% --
Benchmark: running greedy, stingy, each for at least 5 CPU seconds...
greedy: 5 wallclock secs ( 5.17 usr + 0.00 sys = 5.17 CPU) @
3359500.00/s (n=17368615)
stingy: 5 wallclock secs ( 5.23 usr + 0.00 sys = 5.23 CPU) @
3320958.89/s (n=17368615)
Rate stingy greedy
stingy 3320959/s -- -1%
greedy 3359500/s 1% --
$

 

[it_says_BALLS_on_your_forehead]

If you try it, you'll see. If you run a comparison of your tested
function calls without the my declaration against those with the my
declaration, the functions run 60% faster. You can see below that the
RE resolutions are basically the same, whereas the exact same function
(with a my declaration) yields the type of flip flop you're seeing.
I'd assume the flip flop simply reflects the variances of memory
management (for handling the my declarations) on the machine at the
time of execution.

Ed


$ cat checkit.pl
#!/usr/local/bin/perl

use strict; use warnings;
use Benchmark qw(cmpthese);

sub my_call {
my $input =
'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';
}

cmpthese(-5, {
once => \&my_call,
twice => \&my_call,
});

my $input =
'http://activequote.fidelity.com/webxpress/quotes_frame.phtml?QUOTE_TYPE=D&SID_VALUE_ID=.N225';

sub greedy {
return 1 if $input =~ /campaign\w*_fulltop\.gif/;
}

sub stingy {
return 1 if $input =~ /campaign\w*?_fulltop\.gif/;
}

cmpthese(-5, {
greedy => \&greedy,
stingy => \&stingy,
});
$ ./checkit.pl
Benchmark: running once, twice, each for at least 5 CPU seconds...
once: 6 wallclock secs ( 5.55 usr + 0.00 sys = 5.55 CPU) @
2945391.71/s (n=16346924)
twice: 5 wallclock secs ( 5.24 usr + 0.00 sys = 5.24 CPU) @
3180811.45/s (n=16667452)
Rate once twice
once 2945392/s -- -7%
twice 3180811/s 8% --
Benchmark: running greedy, stingy, each for at least 5 CPU seconds...
greedy: 5 wallclock secs ( 5.17 usr + 0.00 sys = 5.17 CPU) @
3359500.00/s (n=17368615)
stingy: 5 wallclock secs ( 5.23 usr + 0.00 sys = 5.23 CPU) @
3320958.89/s (n=17368615)
Rate stingy greedy
stingy 3320959/s -- -1%
greedy 3359500/s 1% --
$

that is very interesting. i based my method of benchmarking on Randal
Schwartz's, where he assigns the variable a value inside the
subroutine, so i thought that would be good:

http://www.stonehenge.com/merlyn/UnixReview/col49.html

....however, based on your results, maybe that's a bad idea?

 

[ednotover]

[ snipped discussion of how benchmarking functions with my declarations
can cause varying benchmark results due to varying execution times for
my() ]

that is very interesting. i based my method of benchmarking on Randal
Schwartz's, where he assigns the variable a value inside the
subroutine, so i thought that would be good:

http://www.stonehenge.com/merlyn/UnixReview/col49.html

...however, based on your results, maybe that's a bad idea?

My rule of thumb is to benchmark exactly what I'm trying to compare -
nothing more, and nothing less. So if I wanted to compare regexes,
then that's all I'd put into the comparison (as discussed previously in
this thread). It's like if I want to compare driving times from
Chicago to LA versus Chicago to Seattle. One way would be comparing
driving times from Miami to Chicago to LA versus Miami to Chicago to
Seattle - but why include the duplicated Miami to Chicago leg in there?
Remove the common factors and let the desired comparison stand alone.

So, yes, (in this one instance) I disagree with what Randal did there.
Note, though, that his results are such that a five to ten percent
swing wouldn't make any real difference. In your case, that five to
ten percent swing was effectively the entire difference.

As an aside, part of the issue in your code is that what you are
comparing is so quick, the my declarations contribute a significant
portion of the run time. Were you comparing something that took a good
bit longer, then the my declarations would be a minimal fraction of the
run time, and the run time variances in my() would be trivial compared
to the rest of the compared code.

Ed