Program slows as I run it

[Daniel Miller]

Hi, I wrote a program that basically takes one element at a time from a
large file (>one million lines) and searches some arrays etc. and spits out
data. Problem is that after I start it, it slowly grinds to a crawl as it
runs. I've re-written it several ways so as not to chew up my RAM (windows
says over 700MB free!) Any thoughts on how to approach this problem? Is this
a CPU cache problem?

Thanks

Dan

 

[Scott W Gifford]

Hi, I wrote a program that basically takes one element at a time from a
large file (>one million lines) and searches some arrays etc. and spits out
data. Problem is that after I start it, it slowly grinds to a crawl as it
runs. I've re-written it several ways so as not to chew up my RAM (windows
says over 700MB free!) Any thoughts on how to approach this problem? Is this
a CPU cache problem?

It sounds to me like a problem with your algorithm. Can you describe
in more detail what your program is doing?

-----ScottG.

 

[Gunnar Hjalmarsson]

Hi, I wrote a program that basically takes one element at a time
from a large file (>one million lines) and searches some arrays
etc. and spits out data.

Then we know *exactly* what the reason for your problem is...

Problem is that after I start it, it slowly grinds to a crawl as it
runs. I've re-written it several ways so as not to chew up my RAM
(windows says over 700MB free!) Any thoughts on how to approach
this problem?

Of course: Re-write it again.

 

[gnari]

Hi, I wrote a program that basically takes one element at a time from a
large file (>one million lines) and searches some arrays etc. and spits out
data. Problem is that after I start it, it slowly grinds to a crawl as it
runs. I've re-written it several ways so as not to chew up my RAM (windows
says over 700MB free!) Any thoughts on how to approach this problem? Is this
a CPU cache problem?

use hashes instead of the arrays

gnari

 

[ctcgag]

Hi, I wrote a program that basically takes one element at a time from a
large file (>one million lines) and searches some arrays etc.

Are the arrays static once initially set, or do they change throughout?

and spits
out data. Problem is that after I start it, it slowly grinds to a crawl
as it runs. I've re-written it several ways so as not to chew up my RAM
(windows says over 700MB free!) Any thoughts on how to approach this
problem? Is this a CPU cache problem?

It is kind of hard to say when you don't how us any of the progam.

95.54% chance the problem is you are using arrays when you should use
hashes.

2.87% chance the you are making some other poor algorithmic choice.

1.58% chance that the problem is simply too big for you machine's
britches. 0.01% chance that the problem lies solely in the CPU cache.

Xho

 

[Daniel Miller]

Thanks for the input, even with the lack of information I think most of you
guessed correctly. Turns out that as each record was processed I was saving
it in an array and then reitterating over that array with the next record
(to make sure there were no duplicates) explaining why it was slowing
exponentially as the second array grew in size. Anyway, thanks for you
thoughts. I'll have to get more comfortable with hashes.

Dan

 

[Patrice Auffret]

Thanks for the input, even with the lack of information I think most of
you
guessed correctly. Turns out that as each record was processed I was
saving
it in an array and then reitterating over that array with the next record
(to make sure there were no duplicates) explaining why it was slowing
exponentially as the second array grew in size. Anyway, thanks for you
thoughts. I'll have to get more comfortable with hashes.

If you want to have arrays with unique elements:
use Tie::Array::Unique;