While loop

[Fab86]

Hello,

I am currently working on my program which send queries to Yahoo and
then saves them into a flatfile. The problem I have is that I need to
conduct 200 searches and Yahoo typically times out during the search
with an error. I have caught the error and told it to time.sleep(10)
however I can only think of deleting the flatfile and starting again
hoping that it will not time out again.

Is it possible to get the program to catch the exception, wait 10
seconds, then carry of from where it was rather than starting again?

If so, could someone please inform me how to do this?

Thanks,

Fabien

 

[Marco Mariani]

Is it possible to get the program to catch the exception, wait 10
seconds, then carry of from where it was rather than starting again?

something like this? probably works in PASCAL as well

i=0
while i < len(stuff):
try:
do_with(stuff)
except SomeError:
sleep(10)
continue
i+=1

 

[Fab86]

Is it possible to get the program to catch the exception, wait 10
seconds, then carry of from where it was rather than starting again?

something like this? probably works in PASCAL as well

i=0
while i < len(stuff):
   try:
      do_with(stuff)
   except SomeError:
      sleep(10)
      continue
   i+=1

using sleep and then continue just makes the search start from the
first search term like before.. Would it be easier to understand if I
posted sections of my code?

i = 0
k = open('blah', 'w')
domains = ["au", "ca", "nl", "be", "...]
srch = WebSearch(app_id=YahooKey)

while i<200:
try:
for domain in domains:
srch.query = "test site:.%s" % domain
res = srch.parse_results()
print >> k, res.total_results_available
i = i + 1

except SearchError:

(I currently close then reopen document here then restart i to 0)

Any ideas?

 

[Fab86]

something like this? probably works in PASCAL as well

i=0
while i < len(stuff):
   try:
      do_with(stuff)
   except SomeError:
      sleep(10)
      continue
   i+=1

using sleep and then continue just makes the search start from the
first search term like before.. Would it be easier to understand if I
posted sections of my code?

i = 0
k = open('blah', 'w')
domains = ["au", "ca", "nl", "be", "...]
srch = WebSearch(app_id=YahooKey)

while i<200:
    try:
        for domain in domains:
            srch.query = "test site:.%s" % domain
            res = srch.parse_results()
            print >> k, res.total_results_available
            i = i + 1

    except SearchError:

(I currently close then reopen document here then restart i to 0)

Any ideas?

All sorted now, thanks.

 

[Andre Engels]

Is it possible to get the program to catch the exception, wait 10
seconds, then carry of from where it was rather than starting again?

something like this? probably works in PASCAL as well

i=0
while i < len(stuff):
   try:
      do_with(stuff)
   except SomeError:
      sleep(10)
      continue
   i+=1

using sleep and then continue just makes the search start from the
first search term like before.. Would it be easier to understand if I
posted sections of my code?

i = 0
k = open('blah', 'w')
domains = ["au", "ca", "nl", "be", "...]
srch = WebSearch(app_id=YahooKey)

while i<200:
   try:
       for domain in domains:
           srch.query = "test site:.%s" % domain
           res = srch.parse_results()
           print >> k, res.total_results_available
           i = i + 1

   except SearchError:

(I currently close then reopen document here then restart i to 0)

Any ideas?

What you did not tell us was the inside loop. Just put code like
Tino's inside this inner loop:


while i<200:
for domain in domains:
srch.query = "test site:.%s" % domain
while true:
try:
res = srch.parse_results()
break
except SearchError:
time.sleep(10)
print >> k, res.total_results_available
i = i + 1


Or, if you want to to retry a maximum number of times:


max_retries = 50
while i<200:
for domain in domains:
srch.query = "test site:.%s" % domain
retries = 0
while true:
try:
res = srch.parse_results()
break
except SearchError:
if retries < max_retries:
time.sleep(10)
retries += 1
else:
raise TooManyRetriesError
print >> k, res.total_results_available
i = i + 1

 

[Chris Rebert]

using sleep and then continue just makes the search start from the
first search term like before.. Would it be easier to understand if I
posted sections of my code?

i = 0
k = open('blah', 'w')
domains = ["au", "ca", "nl", "be", "...]
srch = WebSearch(app_id=YahooKey)

while i<200:
   try:
       for domain in domains:
           srch.query = "test site:.%s" % domain
           res = srch.parse_results()
           print >> k, res.total_results_available
           i = i + 1

   except SearchError:

(I currently close then reopen document here then restart i to 0)

Any ideas?

You need to narrow the `try` and put it inside another while-loop:

while i<200:
for domain in domains:
srch.query = "test site:.%s" % domain
while True:
try:
res = srch.parse_results()
except SearchError:
pass
else:
break
print >> k, res.total_results_available
i += 1

Cheers,
Chris

 

[J Kenneth King]

Is it possible to get the program to catch the exception, wait 10
seconds, then carry of from where it was rather than starting again?

something like this? probably works in PASCAL as well

i=0
while i < len(stuff):
   try:
      do_with(stuff)
   except SomeError:
      sleep(10)
      continue
   i+=1

using sleep and then continue just makes the search start from the
first search term like before.. Would it be easier to understand if I
posted sections of my code?

i = 0
k = open('blah', 'w')
domains = ["au", "ca", "nl", "be", "...]
srch = WebSearch(app_id=YahooKey)

while i<200:
try:
for domain in domains:
srch.query = "test site:.%s" % domain
res = srch.parse_results()
print >> k, res.total_results_available
i = i + 1

except SearchError:

(I currently close then reopen document here then restart i to 0)

Any ideas?

You should check out the sched module for scheduling tasks.

<code>

import sched, time

domains = ["au", "ca", "nl", "be", "ru", "us", "de"]
s = sched.scheduler(time.time, time.time)

def do_query(query, domain):
search.query = "%s site:.%s" % (query, domain)
try:
result = search.parse_results()
except SearchError, e:
print >> sys.stderr, e
else:
print >> k, result

for domain in domains:
s.enter(2, 1, do_query, domain)

s.run()

</code>

YMMV