Getting URL's

[defcon8]

How do I get all the URL's in a page?

 

[Ju Hui]

use
htmlparser or regular expression

 

[defcon8]

Thanks

 

[Paul McGuire]

How do I get all the URL's in a page?

pyparsing comes with a simple example that does this, too.

-- Paul
Download pyparsing at http://sourceforge.net/projects/pyparsing

 

[softwindow]

it is difficult to get all URL's in a page
you can use sgmllib module to parse html files
can get the standard href .

 

[Paul McGuire]

it is difficult to get all URL's in a page

<snip>

Is this really so hard?:

-----------------
from pyparsing import Literal,Suppress,CharsNotIn,CaselessLiteral,\
Word,dblQuotedString,alphanums,SkipTo,makeHTMLTags
import urllib

# extract all <a> anchor tags - makeHTMLTags defines a
# fairly robust pair of match patterns, not just "<tag>","</tag>"
linkOpenTag,linkCloseTag = makeHTMLTags("a")
link = linkOpenTag + \
SkipTo(linkCloseTag).setResultsName("body") + \
linkCloseTag.suppress()

# read the HTML source from some random URL
serverListPage = urllib.urlopen( "http://www.google.com" )
htmlText = serverListPage.read()
serverListPage.close()

# use the link grammar to scan the HTML source
for toks,strt,end in link.scanString(htmlText):
print toks.startA.href,"->",toks.body

-----------------
Prints:
/url?sa=p&pref=ig&pval=2&q=http://www.google.com/ig?hl=en ->
Personalized Home
https://www.google.com/accounts/Login?continue=http://www.google.com/&hl=en ->
Sign in
/imghp?hl=en&tab=wi&ie=UTF-8 -> Images
http://groups.google.com/grphp?hl=en&tab=wg&ie=UTF-8 -> Groups
http://news.google.com/nwshp?hl=en&tab=wn&ie=UTF-8 -> News
http://froogle.google.com/frghp?hl=en&tab=wf&ie=UTF-8 -> Froogle
/maphp?hl=en&tab=wl&ie=UTF-8 -> Maps
/intl/en/options/ -> more&nbsp;&raquo;
/advanced_search?hl=en -> Advanced Search
/preferences?hl=en -> Preferences
/language_tools?hl=en -> Language Tools
/intl/en/ads/ -> Advertising&nbsp;Programs
/services/ -> Business Solutions
/intl/en/about.html -> About Google


-- Paul