Search a Large files backwards

[mud_saisem]

Hi there,

Could anybody tell me if there is a fast and efficient way to scan/
print a large files (1Gb+) backwards ?

I am not sure that the reverse function is the best way of doing it,
particularly for really large files.

I was thinking of maybe finding the file size in bytes and then
running through a for loop in a descending order printing the file
position with seek, but as you all properly know, that will not work
very well.

i would really like to be able to do this with out using any modules.

Any ideas ?

eg:

#!/usr/bin/perl

$filesize = -s $ARGV[0];

open(FILE, "$ARGV[0]") or die "$!\n";

for ($filesize = -s $ARGV[0]; $filesize >= 0; $filesize--)
{
seek(FILE,$filesize,0);
$line = <FILE>;
print $line;
}

close (FILE);

But my output is as follows

../file_byte_size.pl /tmp/test.txt

..
s.
ds.
rds.
ards.
wards.
kwards.
ckwards.
ackwards.
backwards.


e
le
ile
file
file
e file
he file
the file
the file
t the file
nt the file
int the file
rint the file
print the file
print the file
n print the file
an print the file
can print the file
can print the file
I can print the file

f
if
if
e if
ee if
see if
see if
o see if
to see if
to see if
e to see if
le to see if
ile to see if
file to see if

t
st
est
test
test
a test
a test
s a test
is a test
is a test
s is a test
is is a test
his is a test
This is a test

 

[Uri Guttman]

ms> Could anybody tell me if there is a fast and efficient way to scan/
ms> print a large files (1Gb+) backwards ?

ms> I am not sure that the reverse function is the best way of doing it,
ms> particularly for really large files.

ms> I was thinking of maybe finding the file size in bytes and then
ms> running through a for loop in a descending order printing the file
ms> position with seek, but as you all properly know, that will not work
ms> very well.

ms> i would really like to be able to do this with out using any modules.

why no modules? that is always a bad excuse. you can always cut/paste
the code in the worst case or use local::lib to install them locally.

so the answer is to use File::ReadBackwards

uri

 

[mud_saisem]

  ms> Could anybody tell me if there is a fast and efficient way to scan/
  ms> print a large files (1Gb+) backwards ?

  ms> I am not sure that the reverse function is the best way of doing it,
  ms> particularly for really large files.

  ms> I was thinking of maybe finding the file size in bytes and then
  ms> running through a for loop in a descending order printing the file
  ms> position with seek, but as you all properly know, that will not work
  ms> very well.

  ms> i would really like to be able to do this with out using any modules.

why no modules? that is always a bad excuse. you can always cut/paste
the code in the worst case or use local::lib to install them locally.

so the answer is to use File::ReadBackwards

uri

The point of not using modules is because I would like to learn and
understand how to achieve this first. Simply using the module because
it is easy is not going to teach me anything.

I don't understand why you say not using the module is a bad excuse ?

 

[Uri Guttman]

ms> The point of not using modules is because I would like to learn
ms> and understand how to achieve this first. Simply using the module
ms> because it is easy is not going to teach me anything.

you didn't say that. it helps to know your motivation.

and you can learn how to do it well by studying the code in the
module. writing it for yourself may not show you all that you need to
learn to do it well. it happens to be written by me and it needs to do
more than a simple loop to handle all the cases and be fast and easy to
use.

ms> I don't understand why you say not using the module is a bad
ms> excuse ?

because we get people here who say they can't use modules and have many
poor excuses for why they can't. it is a common whine.

uri

 

[mud_saisem]

  >> why no modules? that is always a bad excuse. you can always cut/paste
  >> the code in the worst case or use local::lib to install them locally.
  >>
  >> so the answer is to use File::ReadBackwards

  ms> The point of not using modules is because I would like to learn
  ms> and understand how to achieve this first. Simply using the module
  ms> because it is easy is not going to teach me anything.

you didn't say that. it helps to know your motivation.

and you can learn how to do it well by studying the code in the
module. writing it for yourself may not show you all that you need to
learn to do it well. it happens to be written by me and it needs to do
more than a simple loop to handle all the cases and be fast and easy to
use.

  ms> I don't understand why you say not using the module is a bad
  ms> excuse ?

because we get people here who say they can't use modules and have many
poor excuses for why they can't. it is a common whine.

uri

Yes, I have no problems with modules or using modules. I just wanted
to find away to do it without a module.
I don't want to be a perl programer that has to rely on other
programmers work or solutions, even though they are developed from
very good programmers such as yourself.

I must admit that I am quite surprised that a developer with your
calibre is in google groups offering support and help to others.

I will take your advise onboard and download your module and study the
code. I am not sure how much of it I will follow, but I will give it a
shot anyway.

 

[sreservoir]

Yes, I have no problems with modules or using modules. I just wanted
to find away to do it without a module.
I don't want to be a perl programer that has to rely on other
programmers work or solutions, even though they are developed from
very good programmers such as yourself.

problem: you can't. somebody else wrote and probably compiled your perl.

I must admit that I am quite surprised that a developer with your
calibre is in google groups offering support and help to others.

it's usenet. and most of us are volunteers.

I will take your advise onboard and download your module and study the
code. I am not sure how much of it I will follow, but I will give it a
shot anyway.

it is comment-a-line code; I find it easy to understand, but a bit
difficult to read. matter of opinion, though.

 

[Uri Guttman]

ms> Yes, I have no problems with modules or using modules. I just
ms> wanted to find away to do it without a module. I don't want to be
ms> a perl programer that has to rely on other programmers work or
ms> solutions, even though they are developed from very good
ms> programmers such as yourself.

relying on other people's work is the mark of a good programmer. you do
it all the time just by using perl itself or your OS, etc. there is no
way anyone could write up all the code they would want to use today. i
know http well enough but i wouldn't replace LWP with my own code. i
might modify it or offer patches or feature requests, etc.

ms> I must admit that I am quite surprised that a developer with your
ms> calibre is in google groups offering support and help to others.

why not? larry himself used to inhabit this group a decade ago. there
are plenty of top perl coders who help out in one of the many possible
forums (usenet, web, irc, mailing lists, etc.).

ms> I will take your advise onboard and download your module and study
ms> the code. I am not sure how much of it I will follow, but I will
ms> give it a shot anyway.

the code is very straightforward and does exactly what you want with all
the bugs fixed and edge conditions handled. it is better to learn from a
working module than to struggle to write your own. the whole concept of
reading lines backwards requires some deeper understanding of files and
i/o which would take you a good while to learn if you don't know it
yet.

uri

 

[Uri Guttman]

s> problem: you can't. somebody else wrote and probably compiled your perl.

s> it's usenet. and most of us are volunteers.

s> it is comment-a-line code; I find it easy to understand, but a bit
s> difficult to read. matter of opinion, though.

and which is more important - reading or understanding code? you have to
realize i have been coding for 36 years and doing perl for 17. my style
(commenting included) has evolved to make sure anyone can understand
what i wrote with the least amount of effort. it may be more verbose
than some but the comments are written to explain WHY i did something as
the code is WHAT i did.

here is a tidbit to learn:

code (all the text in a source file) is a record of all your logical
decisions that you made when implementing a program. it is the one place
where your ego should shine forth. you want readers of your code to be
impressed with how much care you put into it thinking about them.

uri