Pickle Problem

[Fab86]

Hello,

I am new to using Python and am looking at exporting some of my code
into a seperate document.

The code I am using for the pickle is:

file = open('testdoc.txt', 'w')

pickle.dump(res1.total_results_available,file)
pickle.dump(res2.total_results_available,file)
pickle.dump(res3.total_results_available,file)
file.close()

res1.total_results_available and others are simply integers which are
recalled from the Yahoo Search API and they print fine cmd or console
but when trying to pickle them they are displayed like this:

I14
..I152000000
..I86000
..

But in console simply printing these attributes I get:

14
152000000
86000

Can anyone help?

Many thanks,

Fabien

 

[Chris Rebert]

Hello,

I am new to using Python and am looking at exporting some of my code
into a seperate document.

The code I am using for the pickle is:

file = open('testdoc.txt', 'w')

pickle.dump(res1.total_results_available,file)
pickle.dump(res2.total_results_available,file)
pickle.dump(res3.total_results_available,file)
file.close()

res1.total_results_available and others are simply integers which are
recalled from the Yahoo Search API and they print fine cmd or console
but when trying to pickle them they are displayed like this:

I14
.I152000000
.I86000
.

That's the contents of testdoc.txt after your program has written data
to it using pickle. With pickle's default settings, the version of the
format it uses (specifically, the oldest version) looks like that
(integers in base-10 prefixed with 'I', entries separated by a newline
and a period). It's *NOT* intended NOR guaranteed to be human-readable
(indeed, with alternate settings it uses a binary format which is
/unintelligible/ to the causal viewer). For human-readable
serialization, use the `json` module (among other possibilities).

But in console simply printing these attributes I get:

14
152000000
86000

That's normal. Pickle reads in the data from the file and deserializes
it back into the proper Python objects.

Can anyone help?

There's no problem to be solved here, just some explaining in order to
deepen your understanding.

Cheers,
Chris

 

[Fab86]

That's the contents of testdoc.txt after your program has written data
to it using pickle. With pickle's default settings, the version of the
format it uses (specifically, the oldest version) looks like that
(integers in base-10 prefixed with 'I', entries separated by a newline
and a period). It's *NOT* intended NOR guaranteed to be human-readable
(indeed, with alternate settings it uses a binary format which is
/unintelligible/ to the causal viewer). For human-readable
serialization, use the `json` module (among other possibilities).





That's normal. Pickle reads in the data from the file and deserializes
it back into the proper Python objects.




There's no problem to be solved here, just some explaining in order to
deepen your understanding.

Cheers,
Chris

Thank you for your reply.

Are you saying to take a look at the python json module then?

I am wanting to store the integers in a file so that I can then run it
through some software without having to edit it. Will json enable me
to do this?

Thanks again,

Fabien

 

[Hrvoje Niksic]

when trying to pickle them they are displayed like this:

I14
.I152000000
.I86000
.

But in console simply printing these attributes I get:

14
152000000
86000

Can anyone help?

Can you describe the problem in some detail? Everything seems to be
working fine, AFAICT; files written with pickle.dump() are supposed to
be read back with pickle.load(). Is that not working for you, or (I
am guessing) would you prefer a human-readable/editable on-disk
representation of your data?

 

[Fab86]

Fantastic, just what I was looking for Andrew.

Many thanks,

Fabien

 

[Fab86]

Thanks, this seems like a simpler way to do it.

I plan on recording 200 values to this file from the outcome of 200
Yahoo searches. Is there any type of loop I can make to do this or do
I have to have a line like "print >> f, res1.total_results_available"
200 times?

Regards,

Fabien

 

[odeits]

Thanks, this seems like a simpler way to do it.

I plan on recording 200 values to this file from the outcome of 200
Yahoo searches. Is there any type of loop I can make to do this or do
I have to have a line like "print >> f, res1.total_results_available"
200 times?

Regards,

Fabien

How are you getting res1, res2? in most cases you could just use a for
loop

for res in results:
print >> f, res.total_results_available

 

[Fab86]

I am getting res1 and res2 etc from this code:

srch1 = WebSearch(app_id=YahooKey)
srch1.query = "avoir site:.al"
res1 = srch1.parse_results()

srch2 = WebSearch(app_id=YahooKey)
srch2.query = "avoir site:.fr"
res2 = srch2.parse_results()

After identifying res1, I then use the total_results_available class
which saves res1 as an integer.

This is happening 200 times.

How could I make a for loop to do this?

Thanks,

Fabien

 

[MRAB]

I am getting res1 and res2 etc from this code:

srch1 = WebSearch(app_id=YahooKey)
srch1.query = "avoir site:.al"
res1 = srch1.parse_results()

srch2 = WebSearch(app_id=YahooKey)
srch2.query = "avoir site:.fr"
res2 = srch2.parse_results()

After identifying res1, I then use the total_results_available class
which saves res1 as an integer.

This is happening 200 times.

How could I make a for loop to do this?

langs = ["al", "fr"]
for lang in langs:
srch = WebSearch(app_id=YahooKey)
srch.query = "avoir site:.%s" % lang
res = srch.parse_results()
print >> f, res.total_results_available


You might be able to move the "WebSearch" line out of the loop.

Anyway, have you read through a tutorial?

 

[Fab86]

I am getting res1 and res2 etc from this code:

srch1 = WebSearch(app_id=YahooKey)
srch1.query = "avoir site:.al"
res1 = srch1.parse_results()

srch2 = WebSearch(app_id=YahooKey)
srch2.query = "avoir site:.fr"
res2 = srch2.parse_results()

After identifying res1, I then use the total_results_available class
which saves res1 as an integer.

This is happening 200 times.

How could I make a for loop to do this?

langs = ["al", "fr"]
for lang in langs:
     srch = WebSearch(app_id=YahooKey)
     srch.query = "avoir site:.%s" % lang
     res = srch.parse_results()
     print >> f, res.total_results_available

You might be able to move the "WebSearch" line out of the loop.

Anyway, have you read through a tutorial?

thanks for this, works perfectly just as I wanted it to.

Again, thanks to everyone who has helped me.

Fabien

 

[Fab86]

langs = ["al", "fr"]
for lang in langs:
     srch = WebSearch(app_id=YahooKey)
     srch.query = "avoir site:.%s" % lang
     res = srch.parse_results()
     print >> f, res.total_results_available

You might be able to move the "WebSearch" line out of the loop.

Anyway, have you read through a tutorial?

thanks for this, works perfectly just as I wanted it to.

Again, thanks to everyone who has helped me.

Fabien

I am having a bit on an issue getting my program to work. The online
database which I am trying to contact keep timing out meaning I can
not carry out my 200 searches without being interupted.

I believe that the solution would be to include an exception handler
with a timer delay if this error occurs so wait a bit then carry on
however having spent hours looking at this I can not get my head
around how to do such a thing.

I am running with this code:

f = open('TestRun1.txt', 'w')
domains = ["nf", "se", "au", "ca", "nl", "be", "is", "us", "jp", "ie",
"ch", "uk", "fi", "at", "lu", "fr", "dk", "nc", "de", "es", "it",
"il", "hk", "gr", "sg", "pt", "si", "mk", "bb", "cy", "mt", "cz",
"bn", "ar", "sc", "ee", "pl", "hu", "sh", "bh", "lt", "sk", "cl",
"kw", "cr", "uy", "qa", "hr", "ae", "lv", "bs", "cu", "mx", "tt",
"aq", "bg", "ru", "ly", "my", "mo", "pa", "by", "to", "mu", "al",
"ba", "sr", "ve", "ro", "ua", "lc", "br", "co", "om", "ws", "th",
"sa", "kz", "jm", "lb", "fj", "am", "ph", "mv", "pe", "tm", "vc",
"tr", "py", "jo", "az", "tn", "gr", "cl", "dm", "lk", "ge", "do",
"bz", "ec", "im", "ps", "sv", "gy", "cv", "sy", "uz", "dz", "gq",
"kg", "id", "vn", "md", "bo", "hn", "tj", "mm", "ni", "za", "eg",
"gp", "ga", "st", "sb", "ma", "na", "in", "bw", "vu", "kh", "gh",
"mm", "pg", "bt", "la", "km", "sz", "bd", "sd", "np", "cm", "pk",
"tg", "cd", "ls", "ug", "zw", "ke", "ye", "mg", "ng", "mr", "ht",
"dj", "gm", "er", "sn", "tl", "rw", "gn", "bj", "tz", "ci", "zm",
"mw", "ao", "td", "cd", "cf", "et", "mz", "gw", "bi", "ml", "bf",
"ne", "sl", "af", "ad", "ck", "kp", "gl", "va", "ir", "ki", "lr",
"li", "mh", "fm", "mc", "nr", "nu", "pw", "pr", "sm", "rs", "so",
"tw", "tv", "eh"]
srch = WebSearch(app_id=YahooKey)
for domain in domains:
srch.query = "avoir site:.%s" % domain
res = srch.parse_results()
print >> f, res.total_results_available


f.close()

Any help would be greatly appreciated.

Fabien

 

[Gabriel Genellina]

I am having a bit on an issue getting my program to work. The online
database which I am trying to contact keep timing out meaning I can
not carry out my 200 searches without being interupted.

I believe that the solution would be to include an exception handler
with a timer delay if this error occurs so wait a bit then carry on
however having spent hours looking at this I can not get my head
around how to do such a thing.

Exactly. How to handle exceptions:
http://docs.python.org/tutorial/errors.html

Use time.sleep() to wait for some time:
http://docs.python.org/library/time.html#time.sleep

 

[Fab86]

Exactly. How to handle exceptions:http://docs.python.org/tutorial/errors.html

Use time.sleep() to wait for some time:http://docs.python.org/library/time.html#time.sleep

Thanks for that I will give it a read.

Do I need to know what error I am getting?

IDLE is giving me this:

Traceback (most recent call last):
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\test6.py", line
13, in <module>
res = srch.parse_results()
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 765, in parse_results
xml = self.get_results()
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 738, in get_results
stream = self.open()
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 723, in open
raise SearchError(err)
SearchError: service temporarily unavailable [C:28]

 

[Gabriel Genellina]

Thanks for that I will give it a read.

Do I need to know what error I am getting?

Usually it's best to be as specific as possible.

IDLE is giving me this:

Traceback (most recent call last):
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\test6.py", line
13, in <module>
res = srch.parse_results()
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 765, in parse_results
xml = self.get_results()
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 738, in get_results
stream = self.open()
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 723, in open
raise SearchError(err)
SearchError: service temporarily unavailable [C:28]

That means that, in line 13 of test6.py [first line in the stack trace],
after several function calls [intermediate lines in the stack trace]
leading to function open in the search package [last line of the stack
trace], a SearchError exception was raised.
If you want to catch that exception:

try:
... some code ...
except SearchError:
... code to handle the exception ...

How to "spell" exactly the exception name should appear in the
documentation; might be yahoo.SearchError, or yahoo.search.SearchError, or
yahoo.errors.SearchError, or similar.

 

[Fab86]

En Tue, 03 Mar 2009 16:50:25 -0200, Fab86 <[email protected]> escribió:

Thanks for that I will give it a read.

Do I need to know what error I am getting?

Usually it's best to be as specific as possible.

IDLE is giving me this:

Traceback (most recent call last):
  File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\test6.py", line
13, in <module>
    res = srch.parse_results()
  File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 765, in parse_results
    xml = self.get_results()
  File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 738, in get_results
    stream = self.open()
  File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 723, in open
    raise SearchError(err)
SearchError: service temporarily unavailable [C:28]

That means that, in line 13 of test6.py [first line in the stack trace],  
after several function calls [intermediate lines in the stack trace]  
leading to function open in the search package [last line of the stack  
trace], a SearchError exception was raised.
If you want to catch that exception:

try:
    ... some code ...
except SearchError:
   ... code to handle the exception ...

How to "spell" exactly the exception name should appear in the  
documentation; might be yahoo.SearchError, or yahoo.search.SearchError, or  
yahoo.errors.SearchError, or similar.

I have been trying except SearchError: however I get the error:

Traceback (most recent call last):
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\timeDelay.py",
line 19, in <module>
except SearchError:
NameError: name 'SearchError' is not defined

I have searched all documents for terms along the lines of searcherror
but am finding nothing..

Any ideas?

 

[Tim Wintle]

I have been trying except SearchError: however I get the error:

Traceback (most recent call last):
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\timeDelay.py",
line 19, in <module>
except SearchError:
NameError: name 'SearchError' is not defined

I have searched all documents for terms along the lines of searcherror
but am finding nothing..

Try looking at the file
C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\timeDelay.py

and see where it imports SearchError from (or defines it)

Exceptions are just classes, so you'll have to import and reference it
like you'd reference any other class you import.

Hope that helps


Tim Wintle

 

[MRAB]

En Tue, 03 Mar 2009 16:50:25 -0200, Fab86 <[email protected]> escribió:

En Tue, 03 Mar 2009 16:39:43 -0200, Fab86 <[email protected]>
escribió:
I am having a bit on an issue getting my program to work. The online
database which I am trying to contact keep timing out meaning I can
not carry out my 200 searches without being interupted.
I believe that the solution would be to include an exception handler
with a timer delay if this error occurs so wait a bit then carry on
however having spent hours looking at this I can not get my head
around how to do such a thing.
Exactly. How to handle
exceptions:http://docs.python.org/tutorial/errors.html
Use time.sleep() to wait for some
time:http://docs.python.org/library/time.html#time.sleep
Thanks for that I will give it a read.
Do I need to know what error I am getting?

Usually it's best to be as specific as possible.

IDLE is giving me this:
Traceback (most recent call last):
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\test6.py", line
13, in <module>
res = srch.parse_results()
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 765, in parse_results
xml = self.get_results()
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 738, in get_results
stream = self.open()
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 723, in open
raise SearchError(err)
SearchError: service temporarily unavailable [C:28]

That means that, in line 13 of test6.py [first line in the stack trace],
after several function calls [intermediate lines in the stack trace]
leading to function open in the search package [last line of the stack
trace], a SearchError exception was raised.
If you want to catch that exception:

try:
... some code ...
except SearchError:
... code to handle the exception ...

How to "spell" exactly the exception name should appear in the
documentation; might be yahoo.SearchError, or yahoo.search.SearchError, or
yahoo.errors.SearchError, or similar.

I have been trying except SearchError: however I get the error:

Traceback (most recent call last):
File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\timeDelay.py",
line 19, in <module>
except SearchError:
NameError: name 'SearchError' is not defined

I have searched all documents for terms along the lines of searcherror
but am finding nothing..

Any ideas?

It's defined in the module you imported to get the search functionality.

 

[Fab86]

En Tue, 03 Mar 2009 16:50:25 -0200, Fab86 <[email protected]> escribió:
En Tue, 03 Mar 2009 16:39:43 -0200, Fab86 <[email protected]>  
escribió:
I am having a bit on an issue getting my program to work. The online
database which I am trying to contact keep timing out meaning I can
not carry out my 200 searches without being interupted.
I believe that the solution would be to include an exception handler
with a timer delay if this error occurs so wait a bit then carry on
however having spent hours looking at this I can not get my head
around how to do such a thing.
Exactly. How to handle  
exceptions:http://docs.python.org/tutorial/errors.html
Use time.sleep() to wait for some  
time:http://docs.python.org/library/time.html#time.sleep
Thanks for that I will give it a read.
Do I need to know what error I am getting?
Usually it's best to be as specific as possible.
IDLE is giving me this:
Traceback (most recent call last):
  File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\test6.py", line
13, in <module>
    res = srch.parse_results()
  File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 765, in parse_results
    xml = self.get_results()
  File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 738, in get_results
    stream = self.open()
  File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\yahoo\search
\__init__.py", line 723, in open
    raise SearchError(err)
SearchError: service temporarily unavailable [C:28]
That means that, in line 13 of test6.py [first line in the stack trace],  
after several function calls [intermediate lines in the stack trace]  
leading to function open in the search package [last line of the stack  
trace], a SearchError exception was raised.
If you want to catch that exception:
try:
    ... some code ...
except SearchError:
   ... code to handle the exception ...
How to "spell" exactly the exception name should appear in the  
documentation; might be yahoo.SearchError, or yahoo.search.SearchError, or  
yahoo.errors.SearchError, or similar.

I have been trying except SearchError: however I get the error:

Traceback (most recent call last):
  File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\timeDelay.py",
line 19, in <module>
    except SearchError:
NameError: name 'SearchError' is not defined

I have searched all documents for terms along the lines of searcherror
but am finding nothing..

Any ideas?

It's defined in the module you imported to get the search functionality.

I imported:
from yahoo.search.web import WebSearch

However there is nothing re SearchError in that doc or in the .py.

I can only find a reference to SearchError in the __init__ file as a
class called SearchError

 

[Gabriel Genellina]

I imported:
from yahoo.search.web import WebSearch

However there is nothing re SearchError in that doc or in the .py.

I can only find a reference to SearchError in the __init__ file as a
class called SearchError

The __init__.py indicates a package
<http://docs.python.org/tutorial/modules.html#packages>
You didn't tell the package name (the name of the directory containing
__init__.py) so this is somewhat generic. If the package name is foo, use:
from foo import SearchError
If foo is a subpackage under bar, use:
from bar.foo import SearchError

It *might* be:
from yahoo.search.web import WebSearch
or perhaps:
from yahoo.search import WebSearch

You can enter those lines in the interactive interpreter to discover the
right form. (This really ought to have been documented)

 

[Fab86]

En Tue, 03 Mar 2009 23:11:30 -0200, Fab86 <[email protected]> escribió:







The __init__.py indicates a package  
<http://docs.python.org/tutorial/modules.html#packages>
You didn't tell the package name (the name of the directory containing  
__init__.py) so this is somewhat generic. If the package name is foo, use:
 from foo import SearchError
If foo is a subpackage under bar, use:
 from bar.foo import SearchError

It *might* be:
 from yahoo.search.web import WebSearch
or perhaps:
 from yahoo.search import WebSearch

You can enter those lines in the interactive interpreter to discover the  
right form. (This really ought to have been documented)

Ok, I managed to import the correct error class (was in a non expected
place)

Just as I thought I was finished, I encountered a final problem. I am
running a while loop which is constantly adding search results to a
file, like this (print >> f, res.total_results_available). I have put
an exception in that if it times out, it simply tries again until its
finished. The problem is that the loop re-write all new results
continuing on from the previous searches. I would like to somehow
delete all in that file and start again.

I thought this could simply be achieved by putting f.close() in the
exception and then it re-writes it however I am getting this error:

Traceback (most recent call last):
File "/home/csunix/scs5fjnh/FYProj/Python/pYsearch-3.1/test9.py",
line 17, in <module>
print >> f, res.total_results_available
ValueError: I/O operation on closed file

Is there another way rather than closing the file? Is it possible to
delete all within the file?

Thanks