how to scrape url out of href

[homepricemaps]

i need to scrape a url out of an href. it seems that people recommend
that i use beautiful soup but had some problems.

does anyone have sample code for scraping the actual url out of an href
like this one

<a href="http://www.cnn.com" target="_blank">

 

[Paul Rubin]

does anyone have sample code for scraping the actual url out of an href
like this one

<a href="http://www.cnn.com" target="_blank">

If you've got the tag by itself like that, just use a regexp to get
the href out.

 

[homepricemaps]

sorry paul-i'm an extremely beginner programmer, if that! ;-) can you
give me an example?

thanks in advance

 

[Mike Meyer]

i need to scrape a url out of an href. it seems that people recommend
that i use beautiful soup but had some problems.

What problem are you having with BeautifulSoup? It's working fine for
here.

does anyone have sample code for scraping the actual url out of an href
like this one

<a href="http://www.cnn.com" target="_blank">

The following fragment works fine for me:

linktext = soup.fetchText('Next')
if not linktext:
return pages
else:
url = linktext[0].findParent('a')['href']


So you probably want something like:

for anchor in soup.fetch('a', {'target': '_blank'}):
print anchor['href']


<mike

 

[homepricemaps]

mike's code worked like a charm. i have one more question. i have an
href which looks like this:

<td class="all">
<a class="btn" name="D1" href="http://www.cnn.com">
</a>

i thought i would use this code to get the href out but it fails, gives
me a keyerror:

for incident in row('td', {'class':'all'}):
n = incident.findNextSibling('a', {'class': 'btn'})
link = incident.findNextSibling['href'] + "','"


any idea what i'm doing wrong here with the syntax? thanks in advance

 

[Kent Johnson]

mike's code worked like a charm. i have one more question. i have an
href which looks like this:

<td class="all">
<a class="btn" name="D1" href="http://www.cnn.com">
</a>

i thought i would use this code to get the href out but it fails, gives
me a keyerror:

for incident in row('td', {'class':'all'}):
n = incident.findNextSibling('a', {'class': 'btn'})
link = incident.findNextSibling['href'] + "','"


any idea what i'm doing wrong here with the syntax? thanks in advance

ISTM that <a class="btn"> is a child of <td>, not a sibling, and
findNextSibling is a method, not an indexable element. Try
n = incident('a', {'class': 'btn'})
link = n['href'] + "','"

Kent

 

[homepricemaps]

hey ken thanks for writing. when i try that i get told

KeyError: 'href'

 

[homepricemaps]

actuall the full error is this:


File "/home/felafela/BeautifulSoup.py", line 301, in __getitem__
return self._getAttrMap()[key]
KeyError: 'href'