What's Wrong with My Use of Benchmark (was: What use is the Schwartzian Transform?)

[James E Keenan <jkeen@verizon.net>]

On Sun, 11 Jan 2004 00:32:16 -0500, mjd-perl-pm@plover.com wrote:
>
> Something is obviously wrong with your benchmarks.  To start with,
> they claim that on your computer, Perl can sort 9952 items in 3.85
> microseconds flat.  I find that impossible to believe.
>

I've struggled with this for about 6 hours today.  I've tried two 
different ways of using the Benchmark module.  In each I _think_ I'm 
following the coding described in the Benchmark documentation.  But in 
one instance I'm getting the results which Mark characterized as 
"absurd," while in the other I'm getting results which closely track 
what he got with his stripped-down benchmarking code.  I'm wondering if 
anyone can spot what's going wrong in the first approach.

Approach 1:  Code to be benchmarked is single-quoted string which serves 
as value to key-value pair in anonymous hash passed to 
&Benchmark::timethese

timethese(10**7, {
	'CODE A' => '@sorted = sort { -M $b <=> -M $a } @files;',
	'CODE B' => '@sorted = map { $_->[0] } 
	                       sort {$b->[1] <=> $a->[1]} 
	                       map {[$_, -M $_]} @files;',
	'CODE C' => '$date{$_} = -M $_ for @files; 
	             @sorted = sort {$date{$b} <=> $date{$a} } @files; 
	             undef %date;',
	'CODE D' => '@sorted = map  $_->[0], 
	                       sort {$b->[1] <=> $a->[1]} 
	                       map [$_, -M $_], @files;',
	}
);

Results:
No. of files:  466
Benchmark: timing 10000000 iterations of CODE A, CODE B, CODE C, CODE 
D...
    CODE A: 38 wallclock secs 
        (38.84 usr +  0.00 sys = 38.84 CPU) @ 257466.53/s (n=10000000)
    CODE B: 49 wallclock secs 
        (48.11 usr +  0.00 sys = 48.11 CPU) @ 207856.99/s (n=10000000)
    CODE C: 103 wallclock secs 
        (102.38 usr +  0.00 sys = 102.38 CPU) @ 97675.33/s (n=10000000)
    CODE D: 50 wallclock secs 
        (49.93 usr +  0.00 sys = 49.93 CPU) @ 200280.39/s (n=10000000)

Approach 2: Code to be benchmarked in subroutine referenced in value to 
key-value pair in anonymous hash passed to &Benchmark::timethese.  (Note 
difference in number of iterations tested.)

timethese(10**1, {
	'CODE A' => \&A, 'CODE B' => \&B, 'CODE C' => \&C, 	'CODE D' => \&D,
	}
);

sub A { my @sorted = sort { -M $b <=> -M $a } @files; return \@sorted;}

sub B {my @sorted = 
    map { $_->[0] } sort {$b->[1] <=> $a->[1]} map {[$_, -M $_]} @files;
	return \@sorted;
}

sub C {
	my (%date, @sorted); 	for (@files) { $date{$_} = -M $_; }
	@sorted = sort {$date{$b} <=> $date{$a} } @files;
	undef %date;	return \@sorted;
}

sub D {
	my @sorted = map $_->[0], sort {$b->[1] <=> $a->[1]} 
        map [$_, -M $_], @files; return \@sorted;
}

Results:
No. of files:  466
Benchmark: timing 10 iterations of CODE A, CODE B, CODE C, CODE D...
    CODE A: 140 wallclock secs 
        (139.73 usr +  0.00 sys = 139.73 CPU) @  0.07/s (n=10)
    CODE B: 16 wallclock secs 
        (16.04 usr +  0.00 sys = 16.04 CPU) @  0.62/s (n=10)
    CODE C: 17 wallclock secs 
        (16.36 usr +  0.00 sys = 16.36 CPU) @  0.61/s (n=10)
    CODE D: 16 wallclock secs 
        (16.54 usr +  0.00 sys = 16.54 CPU) @  0.60/s (n=10)

Jim Keenan


-
**Majordomo list services provided by PANIX <URL:http://www.panix.com>**
**To Unsubscribe, send "unsubscribe phl" to majordomo@lists.pm.org**