tailing a log file using c ...

[Tony Stock]

Hi all,

problem - need to read a log (text) file looking for certain entries,
this file will still be written to by another 3rd party process, and
hence constantly require monitoring (dare I say tailing ?)

Is it possible / practical / efficient to 'tail' a log file using c.


Questions are :

1. If I was reading file and I reached EOF, would I see new file
writes - or would I have to re-open file to see new additions to it ?

2. Would I use unix 'tail' command fed into a c program ?

3. Any advice appreciated ?

Cheers

Tony

 

[Ekkehard Morgenstern]

1. If I was reading file and I reached EOF, would I see new file
writes - or would I have to re-open file to see new additions to it ?

Depends on the operating system. On many UNIXes, and on Windows, yes, you
will see the new writes.

You have to open the file in a shared mode using unbuffered I/O (open(),
read(), close()).

Copy the data from read() to an internal buffer and print it when it's ready
(complete line).

Keep the file open and attempt to read() beyond the EOF. If an error is
returned, no new data is available, otherwise read as much as you can
process.

2. Would I use unix 'tail' command fed into a c program ?

You could also do that, calling system() to issue a tail command and
retrieving its output. But the other solution is simpler.

 

[Simon Biber]

It's not guaranteed in standard C. When you open the file it may be
completely locked down so no other program could append log entries
to it.

Depends on the operating system. On many UNIXes, and on Windows,
yes, you will see the new writes.

You have to open the file in a shared mode using unbuffered I/O
(open(), read(), close()).

Unbuffered I/O is not part of Standard C, and is off-topic on comp.lang.c.
The open, read, close functions can be discussed on comp.unix.programmer,
even if you are writing code for Windows, since they are POSIX functions.

 

[Rouben Rostamian]

Unbuffered I/O is not part of Standard C, and is off-topic on comp.lang.c.
The open, read, close functions can be discussed on comp.unix.programmer,
even if you are writing code for Windows, since they are POSIX functions.

I don't think that true. The standard defines setvbuf() to set/change
the buffering mode of a stream.

#include <stdio.h>
int setvbuf(FILE *stream, char *buf, int mode , size_t size);

 

[Floyd Davidson]

Hi all,

problem - need to read a log (text) file looking for certain entries,
this file will still be written to by another 3rd party process, and
hence constantly require monitoring (dare I say tailing ?)

Is it possible / practical / efficient to 'tail' a log file using c.
Yes.

Questions are :

1. If I was reading file and I reached EOF, would I see new file
writes - or would I have to re-open file to see new additions to it ?

If you clear the eof condition and re-read the file, you'll see
new data.

2. Would I use unix 'tail' command fed into a c program ?

If you had asked this on comp.unix.programmer the answer would
be to read the man page for popen(), but...

3. Any advice appreciated ?

#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>

int main(int argc, char *argv[])
{
char buf[BUFSIZ];
fpos_t pos;
FILE *fp;

if (argc != 2) {
exit(EXIT_FAILURE);
}

if (NULL == (fp = fopen(argv[1], "r"))) {
fprintf(stderr,"Cannot find file: %s\n", argv[1]);
exit(EXIT_FAILURE);
}

if ( 0 != fseek(fp, 0, SEEK_END)) {
fprintf(stderr,"Cannot read file: %s\n", argv[1]);
exit(EXIT_FAILURE);
}

if (-1 == fgetpos(fp, &pos)) {
fprintf(stderr,"Cannot read file: %s\n", argv[1]);
exit(EXIT_FAILURE);
}

for (; {
fgets(buf, BUFSIZ, fp);
if (ferror(fp)) {
fprintf(stderr,"Error reading file: %s\n", argv[1]);
exit(EXIT_FAILURE);
}
if (feof(fp)) {
fsetpos(fp, &pos);
clearerr(fp);
#if 0
/* non-ansi code! */
sleep(1); /* give up the cpu to other processes */
#endif
continue;
}
fgetpos(fp, &pos);
printf("%s", buf);
fflush(stdout);
}

return EXIT_SUCCESS;
}

 

[T.M. Sommers]

Hi all,

problem - need to read a log (text) file looking for certain entries,
this file will still be written to by another 3rd party process, and
hence constantly require monitoring (dare I say tailing ?)

Is it possible / practical / efficient to 'tail' a log file using c.

Questions are :

1. If I was reading file and I reached EOF, would I see new file
writes - or would I have to re-open file to see new additions to it ?

2. Would I use unix 'tail' command fed into a c program ?

Off-topic here, but 'tail -f logfile | your_program' would be the
simplest, and most flexible, solution. Or use the syslog facility.

 

[Tony Stock]

Thanks for the responses - food for thought.

It will be on a unix platform (unknown at present though)

Just a little confused, one poster said that it needs to be UNBUFFERED
io (i.e. open / read / close), but the sample code posted uses a
normal fileopen (fopen) ????


Thanks

Tony

 

[Floyd Davidson]

Thanks for the responses - food for thought.

It will be on a unix platform (unknown at present though)

Just a little confused, one poster said that it needs to be UNBUFFERED
io (i.e. open / read / close), but the sample code posted uses a
normal fileopen (fopen) ????

Data will be delivered in chunks, and the specifics depend on
whether you use setvbuf (or not) to change the buffering.

A file (as opposed to a terminal) accessed by fopen() will default
to block buffering; hence, you won't see any changes until the
total equals the block size (probably BUFSIZ, defined in stdio.h,
and likely to be 4k or 8k.).

If your logs are line oriented (almost certainly true) you'll
probably want to use setvbuf to change access to line buffered, in
which case each newline will cause data to be available. Otherwise
a single line log entry might not be available until other log
entries occur in sufficient numbers to fill the buffer when block
buffering is used.

The other alternative is to turn off buffering (or use non-portable
low level unbuffered i/o functions), in which case the size of the
chunk will be 1 character, and you're program can happily thrash
away at each and every byte...

 

[Dan Pop]

I don't think that true. The standard defines setvbuf() to set/change
the buffering mode of a stream.

#include <stdio.h>
int setvbuf(FILE *stream, char *buf, int mode , size_t size);

This only suppresses the buffering performed by the standard C library.
Sometimes, you may need to suppress the buffering performed by the OS,
too, and setvbuf is very unlikely to help (although, in theory, it could).

But when people talk about unbuffered I/O (or raw I/O or low level I/O)
they usually mean I/O performed using the OS primitives, rather than the
(relatively high level) <stdio.h> stuff.

Dan

 

[Floyd Davidson]

Data will be delivered in chunks, and the specifics depend on
whether you use setvbuf (or not) to change the buffering.

A file (as opposed to a terminal) accessed by fopen() will default
to block buffering; hence, you won't see any changes until the
total equals the block size (probably BUFSIZ, defined in stdio.h,
and likely to be 4k or 8k.).

If your logs are line oriented (almost certainly true) you'll
probably want to use setvbuf to change access to line buffered, in
which case each newline will cause data to be available. Otherwise
a single line log entry might not be available until other log
entries occur in sufficient numbers to fill the buffer when block
buffering is used.

The other alternative is to turn off buffering (or use non-portable
low level unbuffered i/o functions), in which case the size of the
chunk will be 1 character, and you're program can happily thrash
away at each and every byte...

Well, all of that sounds good... but it doesn't do that.

The code that I posted uses fgets() for input, and acts as described
for line buffering because of that. By changing the input to fread()
behavior as described for unbuffered input can be obtained.

Each read either reaches the end of a line or the end of the
buffer (which ever is important for the mode used) _or_ it hits
an end of file condition. Hence the new data is always available.

 

[CBFalconer]

Thanks for the responses - food for thought.

It will be on a unix platform (unknown at present though)

Just a little confused, one poster said that it needs to be UNBUFFERED
io (i.e. open / read / close), but the sample code posted uses a
normal fileopen (fopen) ????

This is OT on c.l.c. but FYI here is an extract from the info file
for GNU tail:

`-f'
`--follow[=HOW]'
Loop forever trying to read more characters at the end of the file,
presumably because the file is growing. This option is ignored
when reading from a pipe. If more than one file is given, `tail'
prints a header whenever it gets output from a different file, to
indicate which file that output is from.

There are two ways to specify how you'd like to track files with
this option, but that difference is noticeable only when a
followed file is removed or renamed. If you'd like to continue to
track the end of a growing file even after it has been unlinked,
use `--follow=descriptor'. This is the default behavior, but it
is not useful if you're tracking a log file that may be rotated
(removed or renamed, then reopened). In that case, use
`--follow=name' to track the named file by reopening it
periodically to see if it has been removed and recreated by some
other program.