data structures/query question

[ccc31807]

I've solved this twice, but still am not happy with my solution. I can
post working code that meets the requirements, and I will do so. But
first, I'd like to listen to the wisdom of this august body.

I have reproduced a sample of the data file below. The top row
consists of position codes. The left column consists of function
codes. The cells consists of the number of people who have the
position identified in the top row that perform the function
identified in the left column.

The requirement is: return a list of all position codes where at least
one person with that position performs all the functions in the query.
A query can look like this:
----------------query (in.txt)----------------
53ProcStuR 15DataEntr 54ProspStu 02AcadAdvUg 48PrtcipOr 68Trnscrpt
69TrnfrCrd 27GrdProcA

Here is a sample run:
------------output-----------------
jobs1>perl crunch_data_2.plx
Enter the name of your data file (comma separated values): Consol.csv
Enter the name of the functions file (space separated values): in.txt

The functions you specified are: 53ProcStuR 15DataEntr 54ProspStu
02AcadAdvUg 48PrtcipOr 68Trnscrpt 69Trn frCrd 27GrdProcA

The positions that matched: fID 1BDD 1ADSS 1SC 1E 1SD 1ASD 1AA 1SSS
1ASC 1DSS
------------end output---------------------

As I said, I can and will post working code that meets the
requirements, but I don't want to poison the well before hand. I would
appreciate very much your best idea of how to do this.

Here is a sample of the data file -- but only the first few columns
and the first few rows because the file is much larger than this.

---------------data file (Consol.csv)-------------------------
fID,1AA,1AAD,1AD,1ADA,1ADB,1ADBFA,1ADBO,1ADFG,1ADFS,1ADMAS,1ADOF,1ADSS,
1AE,1AF,1AS,1AS2,1ASC,1ASD,1ASDSS,1ASST,1ATRD,1ATTD,1BD,1BDD,1BDR,1BDS,
1BM,1CA,1CLK1,1CLK3
15DataEntr,31,1,1,1,1,1,1,,,1,1,5,1,1,1,1,4,6,1,1,1,1,1,3,1,1,,1,1,1
54ProspStu,31,,1,1,,,,,,1,1,5,1,,,,4,5,1,1,,,1,6,1,,,,1,1
03ClericSup,23,,,1,1,1,1,,1,1,,3,1,1,1,1,2,4,,1,1,1,,1,1,1,,1,1,1
48PrtcipOr,27,,1,1,,1,,,1,1,1,4,1,1,,1,3,5,1,,1,,,4,,,,,,
53ProcStuR,36,,1,1,,,,,,1,1,3,1,1,,,4,3,,,,,1,2,1,,,,,
42MPlnRcrt,12,1,,,,,,1,,,,3,,,,,1,5,,1,1,,1,7,1,1,,,,1
52PrepStuF,22,,,1,1,,,,,1,1,2,,1,,1,3,4,,,,,,1,,,,,,
55RcvScanD,13,,,,,1,,,,1,,,,1,,1,1,2,,1,1,1,,,,,,1,1,1
24FileDevM,15,,,1,1,1,,,,1,1,4,,1,1,1,2,3,1,1,,,,3,,,,1,1,1
56RecepDut,12,,,,,,,,,1,,,,1,,,1,2,,1,1,1,,,,,,1,1,1
65TestProc,18,,,1,,,,1,,,1,3,1,,,,4,2,,,1,,,2,,,,,,1
04AdmssProc,18,,,1,,,,,,1,1,3,1,1,,,3,3,,,,,,2,,,,,1,
02AcadAdvUg,30,,,1,,,,1,,,1,4,1,1,,,4,3,,,,,,2,,,,,,
27GrdProcA,23,1,,1,1,,1,,1,1,,6,,1,1,,2,4,1,1,,,,2,1,,,,,
09CommInvPR,8,1,1,,,,,1,,1,,2,,,,,2,4,1,1,,,1,7,1,1,,,,1
17RetenStr,7,1,1,1,,,,,,1,,4,,,,,2,6,1,,,,1,6,1,1,,,,
16RecrtStr,7,1,,,,,,1,,1,,1,1,,,,1,5,,,,,1,7,1,1,,,,
18DocImagi,9,,,1,,,,,,,1,2,,1,,1,2,3,,1,,,,,,,,1,1,
68Trnscrpt,27,,,,,,,,,,1,4,1,,,,2,3,,,,,1,1,1,,,,,
57RecrdsMa,8,,,1,1,1,,,,1,1,4,,1,,1,2,3,1,,,,,1,,,,,,
01AcadAdvGr,23,,,1,,,,1,,1,,6,,,,,3,3,,,,,,1,,,,,,
07ClassCoor,5,,1,1,,,,,,,1,3,,,,,1,6,,1,,,,,,,,,,1
58ReprtPre,9,,,,1,1,1,,,1,,1,,1,,1,1,2,1,,,1,,2,,,,1,,
28GrdProce,17,,1,1,,1,,,,,1,6,,,,1,3,3,1,,,,,,,,,,,
45SupEqpPu,1,,,,1,1,1,,1,,1,,,1,1,1,2,2,,,,1,,,,1,,,,
49EndTrmPr,20,,1,1,,,,,,,1,4,,,,,3,3,,,,,,,,,,,,
20EnrLeadM,3,1,,,,,,1,,1,,,,,,,1,3,1,,,,1,6,1,1,,,,
33MailRoom,1,,,,,1,,,1,,,,,1,1,1,1,2,,1,,1,,,,1,,1,,1
51ClassSch,4,1,,1,,,,,,,,2,1,,,,2,2,,,1,,,,,,,,,
29ImplPoli,1,1,1,,,1,2,1,,1,1,5,,,,,1,3,1,,,,,2,,,,,,
35MktgBusR,4,1,,,,,,,,1,,1,,,,,1,2,,,,,,6,,1,,,,
69TrnfrCrd,26,,,,,,,,,,1,3,1,,,,2,1,,,,,,1,,,,,,
13StuSvcsT,7,,1,1,,1,,,,,1,8,1,,,,2,3,1,,,,,1,,,,,,
70TrblshtS,7,,,,1,1,,,1,1,1,1,,1,1,1,3,2,,,,,,1,,,,,,
06BldgMonSe,3,1,,,,1,,,,,1,,1,,,,1,4,,,,,,,,,2,,,1
10HRDocs,,1,,,1,1,2,,1,,1,3,,,,,1,1,1,,,1,,,,,,,,
47OvrseeSt,5,,1,1,,1,,1,,,1,4,1,,,,1,2,1,,,,,,,,,,,
11ComplRpt,2,1,,,1,1,1,,,1,,,,1,,1,,2,,,1,,,1,,,,1,,
62StaffSup,,1,,,,1,1,,1,,1,3,,,,,1,1,,,,,,2,,,,,,

 

[Martijn Lievaart]

I've solved this twice, but still am not happy with my solution. I can
post working code that meets the requirements, and I will do so. But
first, I'd like to listen to the wisdom of this august body.

I have reproduced a sample of the data file below. The top row consists
of position codes. The left column consists of function codes. The cells
consists of the number of people who have the position identified in the
top row that perform the function identified in the left column.

From counting columns, I suspect that fID is the column name for the
function codes and not a real position.

The requirement is: return a list of all position codes where at least
one person with that position performs all the functions in the query. A
query can look like this:
----------------query (in.txt)---------------- 53ProcStuR 15DataEntr
54ProspStu 02AcadAdvUg 48PrtcipOr 68Trnscrpt 69TrnfrCrd 27GrdProcA

Here is a sample run:
------------output-----------------
jobs1>perl crunch_data_2.plx
Enter the name of your data file (comma separated values): Consol.csv
Enter the name of the functions file (space separated values): in.txt

The functions you specified are: 53ProcStuR 15DataEntr 54ProspStu
02AcadAdvUg 48PrtcipOr 68Trnscrpt 69Trn frCrd 27GrdProcA

The positions that matched: fID 1BDD 1ADSS 1SC 1E 1SD 1ASD 1AA 1SSS 1ASC
1DSS
------------end output---------------------

Your output does not match your sample data. F.i. there is no position
code 1E.

As I said, I can and will post working code that meets the requirements,
but I don't want to poison the well before hand. I would appreciate very
much your best idea of how to do this.

OK, first note that we can skip any line that starts with a function code
we're not interested in. After that, we just count the number of non-zero
functions per position and print the ones that have the same number as
the query.

M4


#!/usr/bin/perl

use strict;
use warnings;

# I/O simplyfied
# Error handling omitted

my @in = qw/53ProcStuR 15DataEntr 54ProspStu 02AcadAdvUg 48PrtcipOr
68Trnscrpt 69TrnfrCrd 27GrdProcA/;

my $line = <DATA>;
chomp $line;
my @positioncodes = split /,/, $line;
shift @positioncodes; # skip fID column

my %positions;
while ($line = <DATA>) {
my ($function, @people) = split /,/, $line;
next unless grep { $_ eq $function } @in; # skip uninteresting
for my $pos (@positioncodes) {
my $t = shift @people;
$positions{$pos}++ if $t;
}
}

my @matches;
while (my ($position, $nr) = each %positions) {
# print "$position=$nr\n";
push @matches, $position if ($nr == @in);
}

print "The positions that matched: @matches\n";

exit

__DATA__
fID,1AA,1AAD,1AD,1ADA,1ADB,1ADBFA,1ADBO,1ADFG,1ADFS,1ADMAS,1ADOF,1ADSS,1AE,1AF,1AS,1AS2,1ASC,1ASD,1ASDSS,1ASST,1ATRD,1ATTD,1BD,1BDD,1BDR,1BDS,1BM,1CA,1CLK1,1CLK3
15DataEntr,31,1,1,1,1,1,1,,,1,1,5,1,1,1,1,4,6,1,1,1,1,1,3,1,1,,1,1,1
54ProspStu,31,,1,1,,,,,,1,1,5,1,,,,4,5,1,1,,,1,6,1,,,,1,1
03ClericSup,23,,,1,1,1,1,,1,1,,3,1,1,1,1,2,4,,1,1,1,,1,1,1,,1,1,1
48PrtcipOr,27,,1,1,,1,,,1,1,1,4,1,1,,1,3,5,1,,1,,,4,,,,,,
53ProcStuR,36,,1,1,,,,,,1,1,3,1,1,,,4,3,,,,,1,2,1,,,,,
42MPlnRcrt,12,1,,,,,,1,,,,3,,,,,1,5,,1,1,,1,7,1,1,,,,1
52PrepStuF,22,,,1,1,,,,,1,1,2,,1,,1,3,4,,,,,,1,,,,,,
55RcvScanD,13,,,,,1,,,,1,,,,1,,1,1,2,,1,1,1,,,,,,1,1,1
24FileDevM,15,,,1,1,1,,,,1,1,4,,1,1,1,2,3,1,1,,,,3,,,,1,1,1
56RecepDut,12,,,,,,,,,1,,,,1,,,1,2,,1,1,1,,,,,,1,1,1
65TestProc,18,,,1,,,,1,,,1,3,1,,,,4,2,,,1,,,2,,,,,,1
04AdmssProc,18,,,1,,,,,,1,1,3,1,1,,,3,3,,,,,,2,,,,,1,
02AcadAdvUg,30,,,1,,,,1,,,1,4,1,1,,,4,3,,,,,,2,,,,,,
27GrdProcA,23,1,,1,1,,1,,1,1,,6,,1,1,,2,4,1,1,,,,2,1,,,,,
09CommInvPR,8,1,1,,,,,1,,1,,2,,,,,2,4,1,1,,,1,7,1,1,,,,1
17RetenStr,7,1,1,1,,,,,,1,,4,,,,,2,6,1,,,,1,6,1,1,,,,
16RecrtStr,7,1,,,,,,1,,1,,1,1,,,,1,5,,,,,1,7,1,1,,,,
18DocImagi,9,,,1,,,,,,,1,2,,1,,1,2,3,,1,,,,,,,,1,1,
68Trnscrpt,27,,,,,,,,,,1,4,1,,,,2,3,,,,,1,1,1,,,,,
57RecrdsMa,8,,,1,1,1,,,,1,1,4,,1,,1,2,3,1,,,,,1,,,,,,
01AcadAdvGr,23,,,1,,,,1,,1,,6,,,,,3,3,,,,,,1,,,,,,
07ClassCoor,5,,1,1,,,,,,,1,3,,,,,1,6,,1,,,,,,,,,,1
58ReprtPre,9,,,,1,1,1,,,1,,1,,1,,1,1,2,1,,,1,,2,,,,1,,
28GrdProce,17,,1,1,,1,,,,,1,6,,,,1,3,3,1,,,,,,,,,,,
45SupEqpPu,1,,,,1,1,1,,1,,1,,,1,1,1,2,2,,,,1,,,,1,,,,
49EndTrmPr,20,,1,1,,,,,,,1,4,,,,,3,3,,,,,,,,,,,,
20EnrLeadM,3,1,,,,,,1,,1,,,,,,,1,3,1,,,,1,6,1,1,,,,
33MailRoom,1,,,,,1,,,1,,,,,1,1,1,1,2,,1,,1,,,,1,,1,,1
51ClassSch,4,1,,1,,,,,,,,2,1,,,,2,2,,,1,,,,,,,,,
29ImplPoli,1,1,1,,,1,2,1,,1,1,5,,,,,1,3,1,,,,,2,,,,,,
35MktgBusR,4,1,,,,,,,,1,,1,,,,,1,2,,,,,,6,,1,,,,
69TrnfrCrd,26,,,,,,,,,,1,3,1,,,,2,1,,,,,,1,,,,,,
13StuSvcsT,7,,1,1,,1,,,,,1,8,1,,,,2,3,1,,,,,1,,,,,,
70TrblshtS,7,,,,1,1,,,1,1,1,1,,1,1,1,3,2,,,,,,1,,,,,,
06BldgMonSe,3,1,,,,1,,,,,1,,1,,,,1,4,,,,,,,,,2,,,1
10HRDocs,,1,,,1,1,2,,1,,1,3,,,,,1,1,1,,,1,,,,,,,,
47OvrseeSt,5,,1,1,,1,,1,,,1,4,1,,,,1,2,1,,,,,,,,,,,
11ComplRpt,2,1,,,1,1,1,,,1,,,,1,,1,,2,,,1,,,1,,,,1,,
62StaffSup,,1,,,,1,1,,1,,1,3,,,,,1,1,,,,,,2,,,,,,

 

[Martijn Lievaart]

On Wed, 21 Jul 2010 22:13:35 +0200, Martijn Lievaart wrote:

Here's a variant. Start out with all positions in a set, delete those
positions from the set where there are no people in a function we're
interested in. Print what is left.

#!/usr/bin/perl

use strict;
use warnings;

# I/O simplyfied
# Error handling omitted

my @in = qw/53ProcStuR 15DataEntr 54ProspStu 02AcadAdvUg 48PrtcipOr
68Trnscrpt 69TrnfrCrd 27GrdProcA/;

my $line = <DATA>;
chomp $line;
my @positioncodes = split /,/, $line;
shift @positioncodes;

my %positions = map { ($_, 1) } @positioncodes;

while ($line = <DATA>) {
my ($function, @people) = split /,/, $line;
next unless grep { $_ eq $function } @in; # skip uninteresting
for my $pos (@positioncodes) {
my $t = shift @people;
delete $positions{$pos} unless $t;
}
}

my @matches = keys %positions;

print "The positions that matched: @matches\n";

exit

__DATA__

(same as previous)

 

[sln]

On Wed, 21 Jul 2010 07:33:52 -0700, ccc31807 wrote:

[snip ccc]

Your output does not match your sample data. F.i. there is no position
code 1E.

[snip ccc]

OK, first note that we can skip any line that starts with a function code
we're not interested in. After that, we just count the number of non-zero
functions per position and print the ones that have the same number as
the query.

M4


#!/usr/bin/perl

use strict;
use warnings;

# I/O simplyfied
# Error handling omitted

my @in = qw/53ProcStuR 15DataEntr 54ProspStu 02AcadAdvUg 48PrtcipOr
68Trnscrpt 69TrnfrCrd 27GrdProcA/;

my $line = <DATA>;
chomp $line;
my @positioncodes = split /,/, $line;
shift @positioncodes; # skip fID column

my %positions;
while ($line = <DATA>) {

chomp $line;

Because ",,,,,\n" and if ("\n") is true
and the output is:
The positions that matched: 1ASD 1BDD 1AA 1ADSS 1ASC 1CLK3
where 1CLK3 does not match:
1CLK3
15DataEntr, ,1
53ProcStuR, ,
54ProspStu, ,1
02AcadAdvUg, ,
48PrtcipOr, ,
68Trnscrpt, ,
69TrnfrCrd, ,
27GrdProcA, ,

my ($function, @people) = split /,/, $line;

Here, every single line is split() even if it is not an
interresting line. I find it hard to believe there could be
millions of function id's, though there might be more position codes.

next unless grep { $_ eq $function } @in; # skip uninteresting

^
grep's gonna loop through all @in every time.
Would there be better performance if it were a hash?
next unless exists $in{ $function };

-sln

 

[sln]

I've solved this twice, but still am not happy with my solution. I can
post working code that meets the requirements, and I will do so. But
first, I'd like to listen to the wisdom of this august body.

I have reproduced a sample of the data file below. The top row
consists of position codes. The left column consists of function
codes. The cells consists of the number of people who have the
position identified in the top row that perform the function
identified in the left column.

The requirement is: return a list of all position codes where at least
one person with that position performs all the functions in the query.
A query can look like this:
----------------query (in.txt)----------------
53ProcStuR 15DataEntr 54ProspStu 02AcadAdvUg 48PrtcipOr 68Trnscrpt
69TrnfrCrd 27GrdProcA

Here is a sample run:
------------output-----------------
jobs1>perl crunch_data_2.plx
Enter the name of your data file (comma separated values): Consol.csv
Enter the name of the functions file (space separated values): in.txt

The functions you specified are: 53ProcStuR 15DataEntr 54ProspStu
02AcadAdvUg 48PrtcipOr 68Trnscrpt 69Trn frCrd 27GrdProcA

The positions that matched: fID 1BDD 1ADSS 1SC 1E 1SD 1ASD 1AA 1SSS
1ASC 1DSS
------------end output---------------------

I didn't see this post until now.
One thing, like Lievaart said, when you load the position codes into an array,
you have to shift out the first value because it throws off the results.

I don't know how big your data is, this is another method you can
benchmark to see how it is. If you need more information, like
how many people are available it would need counting code.
I imagine there is quite a few ways to do this.

-sln

-------------
use strict;
use warnings;

#
my %queryfuncs = map {$_ => ''} qw /
53ProcStuR 15DataEntr 54ProspStu 02AcadAdvUg
48PrtcipOr 68Trnscrpt 69TrnfrCrd 27GrdProcA /;

my $line = <DATA>;
defined $line or die "Can't find the position codes.";
chomp $line;
my @posncodes = split /,/, $line;
shift @posncodes;
@posncodes > 0 or die "0 number of position codes.";
my @flags = (1) x scalar @posncodes;

#
while( defined( $line = <DATA>))
{
chomp $line;
my ($func) = $line =~ /([^,]+),./;
next unless exists $queryfuncs{ $func };

my $ndx = 0;
for( split /,/, substr $line, length( $func) + 1 ) {
last if $ndx > $#flags;
$_ ||= 0;
$flags[$ndx++] = $flags[$ndx] && $_ > 0;
}
@flags[$ndx .. $#flags] = ();
}
for( 0 .. $#flags) {
print "$posncodes[$_]\n" if $flags[$_];
}

exit;

__DATA__
fID,1AA,1AAD,1AD,1ADA,1ADB,1ADBFA,1ADBO,1ADFG,1ADFS,1ADMAS,1ADOF,1ADSS,1AE,1AF,1AS,1AS2,1ASC,1ASD,1ASDSS,1ASST,1ATRD,1ATTD,1BD,1BDD,1BDR,1BDS,1BM,1CA,1CLK1,1CLK3
15DataEntr,31,1,1,1,1,1,1,,,1,1,5,1,1,1,1,4,6,1,1,1,1,1,3,1,1,,1,1,1
54ProspStu,31,,1,1,,,,,,1,1,5,1,,,,4,5,1,1,,,1,6,1,,,,1,1
03ClericSup,23,,,1,1,1,1,,1,1,,3,1,1,1,1,2,4,,1,1,1,,1,1,1,,1,1,1
48PrtcipOr,27,,1,1,,1,,,1,1,1,4,1,1,,1,3,5,1,,1,,,4,,,,,,
53ProcStuR,36,,1,1,,,,,,1,1,3,1,1,,,4,3,,,,,1,2,1,,,,,
42MPlnRcrt,12,1,,,,,,1,,,,3,,,,,1,5,,1,1,,1,7,1,1,,,,1
52PrepStuF,22,,,1,1,,,,,1,1,2,,1,,1,3,4,,,,,,1,,,,,,
55RcvScanD,13,,,,,1,,,,1,,,,1,,1,1,2,,1,1,1,,,,,,1,1,1
24FileDevM,15,,,1,1,1,,,,1,1,4,,1,1,1,2,3,1,1,,,,3,,,,1,1,1
56RecepDut,12,,,,,,,,,1,,,,1,,,1,2,,1,1,1,,,,,,1,1,1
65TestProc,18,,,1,,,,1,,,1,3,1,,,,4,2,,,1,,,2,,,,,,1
04AdmssProc,18,,,1,,,,,,1,1,3,1,1,,,3,3,,,,,,2,,,,,1,
02AcadAdvUg,30,,,1,,,,1,,,1,4,1,1,,,4,3,,,,,,2,,,,,,
27GrdProcA,23,1,,1,1,,1,,1,1,,6,,1,1,,2,4,1,1,,,,2,1,,,,,
09CommInvPR,8,1,1,,,,,1,,1,,2,,,,,2,4,1,1,,,1,7,1,1,,,,1
17RetenStr,7,1,1,1,,,,,,1,,4,,,,,2,6,1,,,,1,6,1,1,,,,
16RecrtStr,7,1,,,,,,1,,1,,1,1,,,,1,5,,,,,1,7,1,1,,,,
18DocImagi,9,,,1,,,,,,,1,2,,1,,1,2,3,,1,,,,,,,,1,1,
68Trnscrpt,27,,,,,,,,,,1,4,1,,,,2,3,,,,,1,1,1,,,,,
57RecrdsMa,8,,,1,1,1,,,,1,1,4,,1,,1,2,3,1,,,,,1,,,,,,
01AcadAdvGr,23,,,1,,,,1,,1,,6,,,,,3,3,,,,,,1,,,,,,
07ClassCoor,5,,1,1,,,,,,,1,3,,,,,1,6,,1,,,,,,,,,,1
58ReprtPre,9,,,,1,1,1,,,1,,1,,1,,1,1,2,1,,,1,,2,,,,1,,
28GrdProce,17,,1,1,,1,,,,,1,6,,,,1,3,3,1,,,,,,,,,,,
45SupEqpPu,1,,,,1,1,1,,1,,1,,,1,1,1,2,2,,,,1,,,,1,,,,
49EndTrmPr,20,,1,1,,,,,,,1,4,,,,,3,3,,,,,,,,,,,,
20EnrLeadM,3,1,,,,,,1,,1,,,,,,,1,3,1,,,,1,6,1,1,,,,
33MailRoom,1,,,,,1,,,1,,,,,1,1,1,1,2,,1,,1,,,,1,,1,,1
51ClassSch,4,1,,1,,,,,,,,2,1,,,,2,2,,,1,,,,,,,,,
29ImplPoli,1,1,1,,,1,2,1,,1,1,5,,,,,1,3,1,,,,,2,,,,,,
35MktgBusR,4,1,,,,,,,,1,,1,,,,,1,2,,,,,,6,,1,,,,
69TrnfrCrd,26,,,,,,,,,,1,3,1,,,,2,1,,,,,,1,,,,,,
13StuSvcsT,7,,1,1,,1,,,,,1,8,1,,,,2,3,1,,,,,1,,,,,,
70TrblshtS,7,,,,1,1,,,1,1,1,1,,1,1,1,3,2,,,,,,1,,,,,,
06BldgMonSe,3,1,,,,1,,,,,1,,1,,,,1,4,,,,,,,,,2,,,1
10HRDocs,,1,,,1,1,2,,1,,1,3,,,,,1,1,1,,,1,,,,,,,,
47OvrseeSt,5,,1,1,,1,,1,,,1,4,1,,,,1,2,1,,,,,,,,,,,
11ComplRpt,2,1,,,1,1,1,,,1,,,,1,,1,,2,,,1,,,1,,,,1,,
62StaffSup,,1,,,,1,1,,1,,1,3,,,,,1,1,,,,,,2,,,,,,

 

[Martijn Lievaart]

chomp $line;

Because ",,,,,\n" and if ("\n") is true and the output is:
The positions that matched: 1ASD 1BDD 1AA 1ADSS 1ASC 1CLK3 where
1CLK3 does not match:

Good catch.

Here, every single line is split() even if it is not an interresting
line. I find it hard to believe there could be millions of function
id's, though there might be more position codes.

"Make it right first, optimize later". But this is an obvious
optimazation, yes.

^
grep's gonna loop through all @in every time. Would there be better
performance if it were a hash? next unless exists $in{ $function };

Same, but as @in is relatively small, I doubt it's going to make a
(noticable) difference.

M4