matching multiple ip addresses

[mhoer]

Im fairly new to perl and i have a problem. I am sorting a firewall log
into a plain text file. I have it doing everything except for 1 thing
and that is to see a list source and destination IP addresses. I got it
to print out the data in neat columns.
The only thing i need is to see if multiple attempts come from or
are directed to the same ip address. Then the count how many times for
each mutiple address. If someone could point me in the right direction
or give me an idea would be great.



Example of firewall log:
"Number" "Date" "Time" "Product" "Interface" "Origin" "Type" "Action"
"Service" "Source" "Destination" "Protocol" "Rule" "Source Port" "User"
"Information"
"1" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719"
"" ""
"2" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Accept" "imap" "DT017484" "emh1.otc.edu" "tcp" "30" "4036" "" ""
"3" "19Sep2003" "17:00:05" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Drop" "389" "172.16.16.179" "192.0.0.8" "udp" "92" "1068" "" ""


######### Here is what i have so far...

#!/usr/bin/perl
# fwlog.plx
use warnings;
use strict;


my $counter = -1;#this is -1 because the first line is not a actual
line that is needed
my $numDrops = 0;
my $numAccepts = 0;
my $numRejects = 0;
my @field; #this is for separating the different fields.

#these are for input and output.
open INPUT, "firewalllog2.txt" or die "Couldn't open file: $!\n";
open OUTPUT, ">fireout.txt" or die "Couldn't open file: $!\n";

while(<INPUT>){
$counter++;#used to find the total
@field = split /" "/, $_;#this breaks up the file into fields from the
spaces

print OUTPUT $field[5],"\t\t";#outputs the origin or name
print OUTPUT $field[9]," " x (40-length($field[9]));#outputs the
source
print OUTPUT $field[10],"\n";#outputs the destination

if ($field[7] =~ /Drop/) {
$numDrops++;#finds how many drops there are.
}
if ($field[7] =~ /Accept/) {
$numAccepts++;#finds how many accepts there are.
}
if ($field[7] =~ /Reject/){
$numRejects++;#finds how many rejects there are
}
}
# The total of logs, drops, acceps,and rejects
print OUTPUT "\n\nTotal: ",$counter,"\n\n";
print OUTPUT "Action:\n";
print OUTPUT "\tDrops (",$numDrops,")\n";
print OUTPUT "\tAccepts (",$numAccepts,")\n";
print OUTPUT "\tRejects (",$numRejects,")\n";

#still needs to find same sources and destinations and tell how many
there are.

 

[Brian McCauley]

Subject: matching multiple ip addresses

This is a Perl newgroup. Your subject line should describe what you are
doing from the point of view of Perl, not from the perspective of your
application domain

Im fairly new to perl and i have a problem. I am sorting a firewall log
into a plain text file. I have it doing everything except for 1 thing
and that is to see a list source and destination IP addresses. I got it
to print out the data in neat columns.
The only thing i need is to see if multiple attempts come from or
are directed to the same ip address. Then the count how many times for
each mutiple address. If someone could point me in the right direction
or give me an idea would be great.

The way to count the instances of distinct values of something is:

$counter{$something}++;

Example of firewall log:
"Number" "Date" "Time" "Product" "Interface" "Origin" "Type" "Action"
"Service" "Source" "Destination" "Protocol" "Rule" "Source Port" "User"
"Information"
"1" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719"
"" ""
"2" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Accept" "imap" "DT017484" "emh1.otc.edu" "tcp" "30" "4036" "" ""
"3" "19Sep2003" "17:00:05" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Drop" "389" "172.16.16.179" "192.0.0.8" "udp" "92" "1068" "" ""


######### Here is what i have so far...

#!/usr/bin/perl
# fwlog.plx
use warnings;
use strict;

Excelent start. Well done, you are in the top 20% of first-timers already.

my $counter = -1;#this is -1 because the first line is not a actual

I would actually be temped to initialize counter to 0 read the first

my $numDrops = 0;
my $numAccepts = 0;
my $numRejects = 0;

This may be OK but it smells to me like you should really be using a
hash $num{Drops}, $num{Accepts}, $num{Rejects}

my %num;

Or if you really want to initialize maybe

my %num = (Drops => 0, Accepts => 0, Rejects => 0 );

my @field; #this is for separating the different fields.

I think you are suffering from premature declaration. You should always
declare all variables in the smallest applicable scope. (Note this is
not Perl specific, it applies to programming in general).

#these are for input and output.
open INPUT, "firewalllog2.txt" or die "Couldn't open file: $!\n";
open OUTPUT, ">fireout.txt" or die "Couldn't open file: $!\n";

Excellent.

And to throw away the first line...

while(<INPUT>){
$counter++;#used to find the total
@field = split /" "/, $_;#this breaks up the file into fields from the
spaces

There are two ways to split. split() and m//g. The former is useful if
you have a pattern that describes what you want to skip, the latter if
you have a pattern that describes what you want to keep.

my @fields = m/"(.*?)"/g;

print OUTPUT $field[5],"\t\t";#outputs the origin or name
print OUTPUT $field[9]," " x (40-length($field[9]));#outputs the
source
print OUTPUT $field[10],"\n";#outputs the destination

Consider using printf.

printf OUTPUT "%s\t\t%-40s%s", @field[5,9,10];

if ($field[7] =~ /Drop/) {
$numDrops++;#finds how many drops there are.
}
if ($field[7] =~ /Accept/) {
$numAccepts++;#finds how many accepts there are.
}
if ($field[7] =~ /Reject/){
$numRejects++;#finds how many rejects there are
}
}

If you'd used %num hash ...

$num{$field[7]}++;

print OUTPUT "\n\nTotal: ",$counter,"\n\n";
print OUTPUT "Action:\n";
print OUTPUT "\tDrops (",$numDrops,")\n";
print OUTPUT "\tAccepts (",$numAccepts,")\n";
print OUTPUT "\tRejects (",$numRejects,")\n";

IMHO interpolation looks tidier.

print OUTPUT "\n\nTotal: $counter\n\n";
print OUTPUT "Action:\n";
print OUTPUT "\tDrops ($num{Drops})\n";
print OUTPUT "\tAccepts ($num{Accepts})\n";
print OUTPUT "\tRejects ($num{Rejects})\n";

 

[John W. Krahn]

Im fairly new to perl and i have a problem. I am sorting a firewall log
into a plain text file. I have it doing everything except for 1 thing
and that is to see a list source and destination IP addresses. I got it
to print out the data in neat columns.
The only thing i need is to see if multiple attempts come from or
are directed to the same ip address. Then the count how many times for
each mutiple address. If someone could point me in the right direction
or give me an idea would be great.


Example of firewall log:
"Number" "Date" "Time" "Product" "Interface" "Origin" "Type" "Action"
"Service" "Source" "Destination" "Protocol" "Rule" "Source Port" "User"
"Information"
"1" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719"
"" ""
"2" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Accept" "imap" "DT017484" "emh1.otc.edu" "tcp" "30" "4036" "" ""
"3" "19Sep2003" "17:00:05" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Drop" "389" "172.16.16.179" "192.0.0.8" "udp" "92" "1068" "" ""


######### Here is what i have so far...

#!/usr/bin/perl
# fwlog.plx
use warnings;
use strict;


my $counter = -1;#this is -1 because the first line is not a actual
line that is needed
my $numDrops = 0;
my $numAccepts = 0;
my $numRejects = 0;
my @field; #this is for separating the different fields.

#these are for input and output.
open INPUT, "firewalllog2.txt" or die "Couldn't open file: $!\n";
open OUTPUT, ">fireout.txt" or die "Couldn't open file: $!\n";

while(<INPUT>){
$counter++;#used to find the total
@field = split /" "/, $_;#this breaks up the file into fields from the
spaces

print OUTPUT $field[5],"\t\t";#outputs the origin or name
print OUTPUT $field[9]," " x (40-length($field[9]));#outputs the
source
print OUTPUT $field[10],"\n";#outputs the destination

if ($field[7] =~ /Drop/) {
$numDrops++;#finds how many drops there are.
}
if ($field[7] =~ /Accept/) {
$numAccepts++;#finds how many accepts there are.
}
if ($field[7] =~ /Reject/){
$numRejects++;#finds how many rejects there are
}
}
# The total of logs, drops, acceps,and rejects
print OUTPUT "\n\nTotal: ",$counter,"\n\n";
print OUTPUT "Action:\n";
print OUTPUT "\tDrops (",$numDrops,")\n";
print OUTPUT "\tAccepts (",$numAccepts,")\n";
print OUTPUT "\tRejects (",$numRejects,")\n";

#still needs to find same sources and destinations and tell how many
there are.

You could write it something like this (untested):


#!/usr/bin/perl
# fwlog.plx
use warnings;
use strict;


#these are for input and output.
open INPUT, '<', 'firewalllog2.txt' or die "Couldn't open file: $!\n";
open OUTPUT, '>', 'fireout.txt' or die "Couldn't open file: $!\n";

my ( %action, %source, %destination );
while ( <INPUT> ) {
#this breaks up the file into fields from the spaces
my @field = split /" "/;

#outputs the origin or name, source and destination
printf OUTPUT "%s\t\t%-40s%s\n", @field[ 5, 9, 10 ];

$action{ $field[ 7 ] }++;
$source{ $field[ 9 ] }++;
$destination{ $field[ 10 ] }++;
}
# The total of logs, drops, accepts,and rejects
printf OUTPUT <<FORMAT, $. - 1, @action{ qw/ Drop Accept Reject / };

Total: %d

Action:
Drops (%d)
Accepts (%d)
Rejects (%d)
FORMAT

print "\nSource IP Count\n";
printf "%-15s %d\n", $_, $source{ $_ } for keys %source;
print "\nDestination IP Count\n";
printf "%-15s %d\n", $_, $destination{ $_ } for keys %destination;

__END__



John

 

[Anno Siegel]

Im fairly new to perl and i have a problem. I am sorting a firewall log
into a plain text file. I have it doing everything except for 1 thing
and that is to see a list source and destination IP addresses. I got it
to print out the data in neat columns.
The only thing i need is to see if multiple attempts come from or
are directed to the same ip address. Then the count how many times for
each mutiple address. If someone could point me in the right direction
or give me an idea would be great.

Whenever the question is, "How many times does this or that string
appear?", the answer in Perl is a hash. Two hashes in your case.

Example of firewall log:
"Number" "Date" "Time" "Product" "Interface" "Origin" "Type" "Action"
"Service" "Source" "Destination" "Protocol" "Rule" "Source Port" "User"
"Information"
"1" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719"
"" ""
"2" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Accept" "imap" "DT017484" "emh1.otc.edu" "tcp" "30" "4036" "" ""
"3" "19Sep2003" "17:00:05" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
"Log" "Drop" "389" "172.16.16.179" "192.0.0.8" "udp" "92" "1068" "" ""

Your news software has broken up the lines. It's a bit of guesswork
to piece them together again.

######### Here is what i have so far...

#!/usr/bin/perl
# fwlog.plx
use warnings;
use strict;


my $counter = -1;#this is -1 because the first line is not a actual
line that is needed

Again, line too long, but this time you are in control. Best not to
post lines longer than 72 characters to Usenet.

Further, if you are going to throw the first line away, do so. Don't
rely on the fact that the rest of the program will ignore it. That
may change some day.

However, the header line you got there is pure gold. Giving names
to the data fields, it allows you to write your program in a more
self-documenting manner. See my rewrite of your program below for
how to do that.

my $numDrops = 0;
my $numAccepts = 0;
my $numRejects = 0;
my @field; #this is for separating the different fields.

#these are for input and output.

Needless comment. You *named* them INPUT and OUTPUT.

open INPUT, "firewalllog2.txt" or die "Couldn't open file: $!\n";
open OUTPUT, ">fireout.txt" or die "Couldn't open file: $!\n";

You should mention the file name in the error message so the user knows
which file could not be opened.

while(<INPUT>){
$counter++;#used to find the total
@field = split /" "/, $_;#this breaks up the file into fields from the
spaces

The simple split may not be good enough. The log format looks a lot like
it has some escaping mechanism up its sleeve if a string should contain
quote marks. If so, Text::Balanced or Text:arsewords may be the
answer.

print OUTPUT $field[5],"\t\t";#outputs the origin or name
print OUTPUT $field[9]," " x (40-length($field[9]));#outputs the
source
print OUTPUT $field[10],"\n";#outputs the destination

if ($field[7] =~ /Drop/) {
$numDrops++;#finds how many drops there are.
}
if ($field[7] =~ /Accept/) {
$numAccepts++;#finds how many accepts there are.
}
if ($field[7] =~ /Reject/){
$numRejects++;#finds how many rejects there are
}
}
# The total of logs, drops, acceps,and rejects
print OUTPUT "\n\nTotal: ",$counter,"\n\n";
print OUTPUT "Action:\n";
print OUTPUT "\tDrops (",$numDrops,")\n";
print OUTPUT "\tAccepts (",$numAccepts,")\n";
print OUTPUT "\tRejects (",$numRejects,")\n";

#still needs to find same sources and destinations and tell how many
there are.

I have adapted your program below to do this.

To count how often a destination $dest appears, use a hash %destcount
and increment the value $destcount{ $dest} as each comes by. Same for
the sources. Since there are two such hashes to deal with, I have
used a subroutine to report their content.

Another change, as mentioned, is the use of the header line to collect
possible hash keys in @keys. For that purpose, your array @fields has
been replaced by a hash %field. It has the same function, but instead
of accessing fields through magic numbers, it accesses them through
meaningful strings.

open OUTPUT, '>&', 'STDOUT' or die "Can't dup STDOUT"; # for simplicity

my @keys;
@keys = /"([^"]+)"/g for scalar <DATA>;

my $counter = 0;
my $numDrops = 0;
my $numAccepts = 0;
my $numRejects = 0;
my ( %field, %sourcecount, %destcount);
while(<DATA>){
++ $counter;
@field{ @keys} = /"([^"]+)"/g;

print "$field{ Origin}\t\t";
print $field{ Source}, " " x (40-length( $field{ Source}));
print $field{ Destination},"\n";
++ $destcount{ $field{ Destination}};
++ $sourcecount{ $field{ Source}};

for ( $field{ Action} ) {
++ $numDrops if /Drop/;
++ $numAccepts if /Accept/;
++ $numRejects if /Reject/;
}
}
print "\n\nTotal: ",$counter,"\n\n";
print "Action:\n";
print "\tDrops (",$numDrops,")\n";
print "\tAccepts (",$numAccepts,")\n";
print "\tRejects (",$numRejects,")\n";

report_count( "N of attempts by source", \ %sourcecount);
report_count( "N of attempts by destination", \ %destcount);

sub report_count {
my ( $msg, $count) = @_;
my $any;
print "$msg:\n";
for ( sort keys %$count ) {
next if $count->{ $_} < 2;
++ $any;
print "$_: $count->{ $_}\n";
}
print "No multiple attempts\n" unless $any;
}

__DATA__
"Number" "Date" "Time" "Product" "Interface" "Origin" "Type" "Action" "Service" "Source" "Destination" "Protocol" "Rule" "Source Port" "User" "Information"
"1" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW" "Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719" "" ""
"2" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW" "Log" "Accept" "imap" "DT017484" "emh1.otc.edu" "tcp" "30" "4036" "" ""
"3" "19Sep2003" "17:00:05" "VPN-1 & FireWall-1" "N100016" "OTC-FW" "Log" "Drop" "389" "172.16.16.179" "192.0.0.8" "udp" "92" "1068" "" ""
"4" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW" "Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719" "" ""

 

[A. Sinan Unur]

(e-mail address removed) wrote:
....


Excelent start. Well done, you are in the top 20% of first-timers
already.

Ditto. Thanks to the OP.

....

Excellent.

And to throw away the first line...

It seems to me $counter only really counts the total lines. In that
case, can't the whole thing be reduced to:
#! /usr/bin/perl

use strict;
use warnings;

while(<DATA>) {
next unless $.;
print;
}

print "$. lines read";

__END__
1
2
3
4

Or, would it better to localize $. before the while loop?

{
local $.;
while(<DATA>) {
next unless $.;
print;
}
print "$. lines read";
}

Sinan

 

[Peter Sundstrom]

Im fairly new to perl and i have a problem. I am sorting a firewall log
into a plain text file. I have it doing everything except for 1 thing
and that is to see a list source and destination IP addresses. I got it
to print out the data in neat columns.
The only thing i need is to see if multiple attempts come from or
are directed to the same ip address. Then the count how many times for
each mutiple address. If someone could point me in the right direction
or give me an idea would be great.

Im fairly new to perl and i have a problem. I am sorting a firewall log
into a plain text file. I have it doing everything except for 1 thing
and that is to see a list source and destination IP addresses. I got it
to print out the data in neat columns.
The only thing i need is to see if multiple attempts come from or
are directed to the same ip address. Then the count how many times for
each mutiple address. If someone could point me in the right direction
or give me an idea would be great.

Sounds like fwlogum might be useful for you.

See http://www.ginini.com/software/fwlogsum/

Please note that you should not use the code from this as an example for
learning. It was originally written back
in the days of Perl 4, and is currently being completely rewritten to be
much better structured.