Parsing a log file

[CG]

I am looking for a way to parse a simple log file to get the
information in a format that I can use. I would like to use python,
but I am just beginning to learn how to use it. I am not a programmer,
but have done some simple modifications and revisions of scripts. I am
willing to attempt this on my own, if someone can point me in the right
direction (any example scripts that do similar things would be
helpful). This doesn't have to be Python, but I need a cross-platform
solution (i.e. Perl or some other kind of script). I just wanted to
try Python because I like the concept of it.

Here is my scenario:
I have a program that connects and disconnects to a server. It writes
a simple log file like this:

08-13-2005 13:19:37:564 Program: CONNECTED to 'Server'
08-13-2005 15:40:08:313 Program: DISCONNECTED from 'Server'
08-13-2005 15:45:39:234 Program: CONNECTED to 'Server'
08-13-2005 15:55:18:113 Program: DISCONNECTED from 'Server'
08-13-2005 16:30:57:264 Program: CONNECTED to 'Server'
08-13-2005 16:59:46:417 Program: DISCONNECTED from 'Server'
08-13-2005 17:10:33:264 Program: CONNECTED to 'Server'
08-13-2005 18:25:26:316 Program: DISCONNECTED from 'Server'
08-13-2005 18:58:13:564 Program: CONNECTED to 'Server'
08-13-2005 19:29:10:715 Program: DISCONNECTED from 'Server'

What I basically want to do is end up with a text file that can be
easily imported into a database with a format like this (or I guess it
could be written in a SQL script form that could write directly to a
database like Mysql):

Connect_Date Connect_Time Disconnect_date Disconnect_time User
------------ ------------ --------------- --------------- -------
08-13-2005 13:19:37 08-13-2005 15:40:08 John
08-13-2005 15:45:39 08-13-2005 15:55:18 John
08-13-2005 16:30:57 08-13-2005 16:59:46 John
08-13-2005 17:10:33 08-13-2005 18:25:26 John
08-13-2005 18:58:13 08-13-2005 19:29:10 John

Here are some notes about this:
* the username would come from the log file name (i.e.
John_Connect.log)
* I don't need the fractions of seconds in the timestamps
* I only need date, time, and connect or disconnect, the other info is
not important
* If it is possible to calculate the elapsed time between Connect and
Disconnect and create a new field with that data, that would help (but
I can easily do that with SQL queries)
* This log file layout seems to be consistent
* There may not be a "disconnect" statement if the log file is read
while connected, so the next time it would have to insert the
disconnect information. The file will be read quite regularly, so this
is very likely.
* This would eventually need to be done without intervention (maybe
every 5 minutes).

I am open to other ideas or existing programs and am flexible about the
final solution.

Thanks,
Clint

 

[Andreas Kostyrka]

Am Samstag, den 13.08.2005, 14:01 -0700 schrieb CG:

Well, you have described your problem nicely. One thing that's missing
is how to deal with incorrect input. (For example missing connect or
disconnect messages).

Furthermore, you can now:
a) try to find somebody who writes it for you. How you motivate that
person is another question.
b) try to hack some solution yourself. Start with doing the python
tutorial?

Andreas

I am looking for a way to parse a simple log file to get the
information in a format that I can use. I would like to use python,
but I am just beginning to learn how to use it. I am not a programmer,
but have done some simple modifications and revisions of scripts. I am
willing to attempt this on my own, if someone can point me in the right
direction (any example scripts that do similar things would be
helpful). This doesn't have to be Python, but I need a cross-platform
solution (i.e. Perl or some other kind of script). I just wanted to
try Python because I like the concept of it.

Here is my scenario:
I have a program that connects and disconnects to a server. It writes
a simple log file like this:

08-13-2005 13:19:37:564 Program: CONNECTED to 'Server'
08-13-2005 15:40:08:313 Program: DISCONNECTED from 'Server'
08-13-2005 15:45:39:234 Program: CONNECTED to 'Server'
08-13-2005 15:55:18:113 Program: DISCONNECTED from 'Server'
08-13-2005 16:30:57:264 Program: CONNECTED to 'Server'
08-13-2005 16:59:46:417 Program: DISCONNECTED from 'Server'
08-13-2005 17:10:33:264 Program: CONNECTED to 'Server'
08-13-2005 18:25:26:316 Program: DISCONNECTED from 'Server'
08-13-2005 18:58:13:564 Program: CONNECTED to 'Server'
08-13-2005 19:29:10:715 Program: DISCONNECTED from 'Server'

What I basically want to do is end up with a text file that can be
easily imported into a database with a format like this (or I guess it
could be written in a SQL script form that could write directly to a
database like Mysql):

Connect_Date Connect_Time Disconnect_date Disconnect_time User
------------ ------------ --------------- --------------- -------
08-13-2005 13:19:37 08-13-2005 15:40:08 John
08-13-2005 15:45:39 08-13-2005 15:55:18 John
08-13-2005 16:30:57 08-13-2005 16:59:46 John
08-13-2005 17:10:33 08-13-2005 18:25:26 John
08-13-2005 18:58:13 08-13-2005 19:29:10 John

Here are some notes about this:
* the username would come from the log file name (i.e.
John_Connect.log)
* I don't need the fractions of seconds in the timestamps
* I only need date, time, and connect or disconnect, the other info is
not important
* If it is possible to calculate the elapsed time between Connect and
Disconnect and create a new field with that data, that would help (but
I can easily do that with SQL queries)
* This log file layout seems to be consistent
* There may not be a "disconnect" statement if the log file is read
while connected, so the next time it would have to insert the
disconnect information. The file will be read quite regularly, so this
is very likely.
* This would eventually need to be done without intervention (maybe
every 5 minutes).

I am open to other ideas or existing programs and am flexible about the
final solution.

Thanks,
Clint

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBC/zsYHJdudm4KnO0RArj+AKDO9t4P7Sc/I6aAFcznfeWG5/nvqgCgjAIk
SnheKzvZcopetsUB/isosq8=
=z8EC
-----END PGP SIGNATURE-----

 

[John Machin]

CG wrote:
[snip]

What I basically want to do is end up with a text file that can be
easily imported into a database with a format like this (or I guess it
could be written in a SQL script form that could write directly to a
database like Mysql):

Connect_Date Connect_Time Disconnect_date Disconnect_time User
------------ ------------ --------------- --------------- -------
08-13-2005 13:19:37 08-13-2005 15:40:08 John [snip]
* I don't need the fractions of seconds in the timestamps

(1) Famous last words.
(2) What do you gain my throwing information away? Nothing! On input,
record what you are given. You can always round/truncate on output.

* I only need date, time, and connect or disconnect, the other info is
not important

Think about date and time as ONE piece of info. Use a "datetime" object
in Python, not a "date" and a "time". Same story with the columns in
your database.

* If it is possible to calculate the elapsed time between Connect and
Disconnect and create a new field with that data, that would help (but
I can easily do that with SQL queries)

and you will be able to do that even more easily if you use one "datetime".

A couple of quick silly questions: What do you do if servers are in
different timezones? What if "John" connects before a daylight saving
change and disconnects afterwards? Any chance of your using ISO standard
format for representing dates?

 

[CG]

Thanks Andreas,

In your first paragraph, you ask about incorrect input. I guess it is
possible, but without that information, my collection of the data is
useless, so I really don't know what I would do with that.

As for the other stuff, I can hack the data in other ways, such as with
VBA and MSAccess, which I am more familiar with, but I am trying to
move to Linux and want to do it right the first time. I figure Perl is
the more common language for this kind of stuff, but I did want to try
to learn some Python while I am at it. I have started the tutorial,
but being a businessman, time is an issue, which, if I had an example
script that did a similar thing, I can learn by doing that (I am
looking for something similar now).

I do live in a low-labor cost country, so I can hire someone to do it
for a small amount of money, but Python people are a little harder to
find.

Thanks for the comments,
Clint

 

[CG]

John,

Your comments are very helpful. I will take the datetime stamp as the
way to go. I don't have a need to throw away the time info, it is

You said:

What do you do if servers are in different
timezones?

This is all inhouse in a non-daylight savings country and would not be
an issue

You also said:

Any chance of your using ISO standard format
for representing dates?

I think I have very little control over the actual logfile data. I
seem to be able to control what info it collects, but I don't think I
can change the formatting.

Thanks,
Clint

 

[googleboy]

I am similarly not a programmer but am trying to learn python to do
tasks like this. I would read through the regular expressions
tutorial. You could probably easily read in all teh lines of the log
file, and then split them up by " " (spaces)..

If you're right about the lines all being consistent, that should
easily handle each line.

From there you could amost certainly drop off the trailling

milliseconds on the timestamps and do the simple data manipulation
you'd like.

here are a couple of links:

http://www.amk.ca/python/howto/regex/
http://gnosis.cx/publish/programming/regular_expressions.html

HTH

googleboy

 

[Andreas Kostyrka]

Completly untested:

#!/usr/bin/env python

import sys, datetime

user = sys.argv[1]

starttime = None
for l in sys.stdin:
flds = l.strip().split()
datestr, timestr, prog, op, to, sname = flds
month, day, year = [int(x) for x in datestr.split("-", 2)]
hour, min, sec, ms = [int(x) for x in timestr.split(":")]
timestamp = datetime.datetime(year, month, day, hour, min, sec)
if op == 'CONNECTED':
assert starttime is None
starttime = timestamp
elif op == 'DISCONNECTED':
assert starttime is not None
endtime = timestamp
sql = "insert into data (start, end, user) value (%r, %r, %r);"
print sql % (starttime, endtime, user)
else:
raise AssertationError("%r is not a valid line" % l)



Am Sonntag, den 14.08.2005, 07:31 -0700 schrieb CG:

Thanks Andreas,

In your first paragraph, you ask about incorrect input. I guess it is
possible, but without that information, my collection of the data is
useless, so I really don't know what I would do with that.

As for the other stuff, I can hack the data in other ways, such as with
VBA and MSAccess, which I am more familiar with, but I am trying to
move to Linux and want to do it right the first time. I figure Perl is
the more common language for this kind of stuff, but I did want to try
to learn some Python while I am at it. I have started the tutorial,
but being a businessman, time is an issue, which, if I had an example
script that did a similar thing, I can learn by doing that (I am
looking for something similar now).

I do live in a low-labor cost country, so I can hire someone to do it
for a small amount of money, but Python people are a little harder to
find.

Thanks for the comments,
Clint

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBC/7N5HJdudm4KnO0RAtMtAJ9i/+y83Y/ISJOUmhW22YvHjBuz1wCePLFq
3NMs2bO+4nqgNscSouhN208=
=579W
-----END PGP SIGNATURE-----

 

[Andreas Kostyrka]

Completly untested:

#!/usr/bin/env python

import sys, datetime

user = sys.argv[1]

starttime = None
for l in sys.stdin:
flds = l.strip().split()
datestr, timestr, prog, op, to, sname = flds
month, day, year = [int(x) for x in datestr.split("-", 2)]
hour, min, sec, ms = [int(x) for x in timestr.split(":")]
timestamp = datetime.datetime(year, month, day, hour, min, sec)
if op == 'CONNECTED':
assert starttime is None
starttime = timestamp
elif op == 'DISCONNECTED':
assert starttime is not None
endtime = timestamp
sql = "insert into data (start, end, user) value (%r, %r, %r);"
print sql % (starttime, endtime, user)
else:
raise AssertationError("%r is not a valid line" % l)



Am Sonntag, den 14.08.2005, 07:31 -0700 schrieb CG:

Thanks Andreas,

In your first paragraph, you ask about incorrect input. I guess it is
possible, but without that information, my collection of the data is
useless, so I really don't know what I would do with that.

As for the other stuff, I can hack the data in other ways, such as with
VBA and MSAccess, which I am more familiar with, but I am trying to
move to Linux and want to do it right the first time. I figure Perl is
the more common language for this kind of stuff, but I did want to try
to learn some Python while I am at it. I have started the tutorial,
but being a businessman, time is an issue, which, if I had an example
script that did a similar thing, I can learn by doing that (I am
looking for something similar now).

I do live in a low-labor cost country, so I can hire someone to do it
for a small amount of money, but Python people are a little harder to
find.

Thanks for the comments,
Clint

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBC/7N5HJdudm4KnO0RAtMtAKDooZ+aqUQjGgRlJUPDOzCkm6MeRwCfbXTr
1Xl2sb6Fn9fuq0wM46t/jM0=
=pe/a
-----END PGP SIGNATURE-----