Slow output

[subhabangalore]

Dear Group,
I am Sri Subhabrata Banerjee writing from India. I am running a small program which exploits around 12 1 to 2 KB .txt files. I am using MS Windows XP Service Pack 3 and Python 2.6 where IDLE is GUI. The text is plain ASCII text. The RAM of the machine is around 2 GB. To run the program the machine is becoming dead slow and it is executing only after several minutes. I am shocked as I have run hugely loaded Statistical Learning stuff only on 516MBRAM on Python using Windows XP only. And that too on a laptop.
I am not getting. I discussed the issue with my system administrator and heincreased the RAM to 8GB result improved by 30% but not satisfactory speed.. May any one suggest me what may be the likely issue?

Thanking You in Advance,
Subhabrata Banerjee.

 

[MRAB]

Dear Group,
I am Sri Subhabrata Banerjee writing from India. I am running a small
program which exploits around 12 1 to 2 KB .txt files. I am using MS
Windows XP Service Pack 3 and Python 2.6 where IDLE is GUI. The text
is plain ASCII text. The RAM of the machine is around 2 GB. To run
the program the machine is becoming dead slow and it is executing
only after several minutes. I am shocked as I have run hugely loaded
Statistical Learning stuff only on 516MB RAM on Python using Windows
XP only. And that too on a laptop. I am not getting. I discussed the
issue with my system administrator and he increased the RAM to 8GB
result improved by 30% but not satisfactory speed. May any one
suggest me what may be the likely issue?

Could you post the program here (you say that it's small) so that we
can see what you're trying to do.

 

[Jean-Michel Pichavant]

Could you post the program here (you say that it's small) so that we
can see what you're trying to do.

Additionaly, you may want to look at
http://docs.python.org/library/profile.html
You may spot quite quickly where your program is dying. It's worth the
try, it's doesn't take long to setup the profile.

JM

 

[Steven D'Aprano]

Dear Group,
I am Sri Subhabrata Banerjee writing from India. I am running a small
program which exploits around 12 1 to 2 KB .txt files. I am using MS
Windows XP Service Pack 3 and Python 2.6 where IDLE is GUI. The text is
plain ASCII text. The RAM of the machine is around 2 GB. To run the
program the machine is becoming dead slow and it is executing only
after several minutes.

[...]
Could you post the program here (you say that it's small) so that we can
see what you're trying to do.

I bet that he is building up long strings using repeated string
concatenation:

s = ''
for item in many_items:
s += item


instead of accumulating them into a list, then joining them all at once.

Repeated string concatenation can be *painfully* slow, especially under
Windows. (For some reason, the details of Windows memory management
sometimes prevents the string concat optimization from working.)

 

[alex23]

May any one suggest me what may be the likely issue?

In situations like this, it always helps to see your code, especially
if you can reduce it down to only the part doing the loading.

One thing that can help reduce memory usage is to replace lists/list
comprehensions with generators. For example, this loads the entire
file into memory:

for line in open('big.txt').readlines():
# do something to line

While this only loads one line at a time:

for line in open('big.txt'):
# do something to line

You can also do the same with multiple files by creating a generator
to return their content a line at a time:

filenames = ['a.txt', 'b.txt', 'c.txt']
files = (open(name) for name in filenames)
lines = (line for file in files for line in file)

for line in lines:
# do something to line

I highly recommend David Beazley's "Generator Tricks for Systems
Programmers" for more techniques like this.

 

[subhabangalore]

Dear Group,
I am Sri Subhabrata Banerjee writing from India. I am running a small program which exploits around 12 1 to 2 KB .txt files. I am using MS Windows XP Service Pack 3 and Python 2.6 where IDLE is GUI. The text is plain ASCII text. The RAM of the machine is around 2 GB. To run the program the machineis becoming dead slow and it is executing only after several minutes. I amshocked as I have run hugely loaded Statistical Learning stuff only on 516MB RAM on Python using Windows XP only. And that too on a laptop.
I am not getting. I discussed the issue with my system administrator and he increased the RAM to 8GB result improved by 30% but not satisfactory speed. May any one suggest me what may be the likely issue?

Thanking You in Advance,
Subhabrata Banerjee.

Dear Group,
Thank you for your kind reply and teaching me some nice coding options. I would port your suggestions and see if it improves. Unfortuantely I can not share the code as it has slight propietary issue, but I was debugging the code on myself, I was seeing that if I write the outputs in the file then itis giving fast results, but if I print it to the console -which is a list of big size, but well even I print all the files of 1KB each I have 12 files only, why a print command would make it so slow? I have searched processed made big n-grams big calculations across even UTF-8 encoding data sets Python gave result in fragment of seconds.
Regards,
Subhabrata Banerjee.

 

[Roel Schroeven]

(e-mail address removed) schreef:

Dear Group,
I am Sri Subhabrata Banerjee writing from India. I am running a small
program which exploits around 12 1 to 2 KB .txt files. I am using MS
Windows XP Service Pack 3 and Python 2.6 where IDLE is GUI. The text
is plain ASCII text. The RAM of the machine is around 2 GB. To run
the program the machine is becoming dead slow and it is executing
only after several minutes. I am shocked as I have run hugely loaded
Statistical Learning stuff only on 516MB RAM on Python using Windows
XP only. And that too on a laptop. I am not getting. I discussed the
issue with my system administrator and he increased the RAM to 8GB
result improved by 30% but not satisfactory speed. May any one
suggest me what may be the likely issue?

Does the program generate a lot of text? Do you run the program in IDLE?
IDLE gets very slow when it has to show a lot of text. Try running the
program from the command line to see if that offers any improvement.

Best regards,
Roel