Parse log files for data

[Dan]

I have a bunch of custom log files with database queries in them, and
would like to extract all the insert and update queries so I can
recreate some lost data.

I'm pretty sure this is the perfect exercise for a perl script. I
just don't have the vocabulary to write it myself. But this is all it
needs to do:

* Open, in order, all log files named yyyymmdd.log in the current
directory. (where yyyymmdd is what you think it is)

* For each file, look at each line, and if it begins with " query=
INSERT" or " query = UPDATE", write that entire line starting from the
word "INSERT" or "UPDATE" to the output file.

I'm not sure what version of perl I have or if that matters, but my
system is Fedora Core 2, so whatever comes with that...

Any takers? The 2 minutes of your time would be greatly appreciated.

 

[Mladen Gogala]

I have a bunch of custom log files with database queries in them, and
would like to extract all the insert and update queries so I can
recreate some lost data.

I'm pretty sure this is the perfect exercise for a perl script. I
just don't have the vocabulary to write it myself. But this is all it
needs to do:

* Open, in order, all log files named yyyymmdd.log in the current
directory. (where yyyymmdd is what you think it is)

* For each file, look at each line, and if it begins with " query=
INSERT" or " query = UPDATE", write that entire line starting from the
word "INSERT" or "UPDATE" to the output file.

find mydir -name logfile -exec cat {} \;|perl -ne 'if (/\s+query =([UI].*)/) { print "$1\n"; }'

 

[Dan]

ls -1 *.log | grep '/.*/'

returns nothing.

 

[Dan]

That will print the whole line. He wants to print just the part after
INSERT or UPDATE.

Not just after, but including. If it grabs the whole line that's
fine. I can easily remove the " query= " from the beginning of each
line.

But at the moment it's not grabbing anything. It just lists all the
files at the end.

Here's a sample entry from one of the log files (just the beginning of
each line):

===00:23:54 on Mon July 26, 2004===
HTTP_REFERER=
HTTP_HOST= www.vi...
REQUEST_URI= /ult...
HTTP_USER_AGENT= Moz...
REMOTE_ADDR= 203...
query= INSERT INTO sc...

Spacing is slightly different, so I modified the regex accordingly to
(/^ query= (INSERT|UPDATE)/x) but it still found nothing.

 

[Anno Siegel]

I have a bunch of custom log files with database queries in them, and
would like to extract all the insert and update queries so I can
recreate some lost data.

I'm pretty sure this is the perfect exercise for a perl script. I
just don't have the vocabulary to write it myself. But this is all it
needs to do:

* Open, in order, all log files named yyyymmdd.log in the current
directory. (where yyyymmdd is what you think it is)

* For each file, look at each line, and if it begins with " query=
INSERT" or " query = UPDATE", write that entire line starting from the
word "INSERT" or "UPDATE" to the output file.

Under Unix that's a job for (e)grep. Untested:

egrep -h '^query = (UPDATE|INSERT)' *.log >output

Anno

 

[Joe Smith]

ls -1 *.log | grep '/.*/'

Sorry about that. Grep does not use //. But the argument needs quoting.

ls -1 *.log | grep '.*'