Help with regular expressions

[dmbkiwi]

I have a problem. I have written a python based theme for a linux app
called superkaramba, which is effectively an engine for desktop applets
that utilises python as its theming language. The script basically parses
weather websites, and displays the info in a visually appealing way.

A couple of other people have contributed code to this project,
particularly relating to the parsing of the websites. Unfortunately, it
is not parsing one particular part of the website properly. This is
because it is expecting the data to be in a certain form, and occasionally
it is in a different form. Unfortunately this causes the entire script to
fail to run.

Unfortunately, I know very little about regular expressions and can't get
hold of the person who wrote this part of the script. The other issue I am
struggling with is that there are no error messages as to what's going
wrong, which makes it more difficult to code round the issue.

The issue comes down to a couple of lines in the html for the web page.
The following lines parse correctly:

<TD ALIGN=LEFT VALIGN=TOP CLASS=obsInfo1>Wind:</TD>
<TD ALIGN=LEFT VALIGN=TOP CLASS=obsInfo2>From the Northeast at 6&nbsp;mph</TD>

these don't:

<TD ALIGN=LEFT VALIGN=TOP CLASS=obsInfo1>Wind:</TD>
<TD ALIGN=LEFT VALIGN=TOP CLASS=obsInfo2>calm&nbsp;</TD>

the relevant portion of the python script is as follows:

print '==================================================='
p_current = r'''(?isx) # Ignore case, Dot matches all, Verbose
wxicons/52/(?P<icon>\d*?)\.gif # Icon
..*?obsTempTextA>(?P<temp>\d*?)&deg; # Temp
..*?obsTextA>(?P<sky>.*?)</b> # Sky
..*?Feels\sLike<br>(?P<heat>.*?)&deg; # Heat
..*?UV\sIndex:.*?Info2>(?P<uv>.*?)&nbsp
..*?Dew\sPoint:.*?Info2>(?P<dew>.*?)&deg;
..*?Humidity:.*?Info2>(?P<hum>\d+)
..*?Visibility:.+?Info2>(?P<vis>.*?)</td>
..*?Pressure:.+?Info2>(?P<baro>.*?)\sinches\sand\s(?P<change>.*?)</td>
..*?Wind:.+?Info2>(?P<wind>.*?)\sat\s(?P<speed>\d*?)&nbsp;
'''


match = re.search(p_current, data1)
if match:
now.icon(match.group('icon'))
now.temperature(match.group('temp'), 'F')
now.relative_heat(match.group('heat'), 'F')
now.sky(match.group('sky'))
now.uv(match.group('uv'))
now.dewpoint(match.group('dew'), 'F')
now.humidity(match.group('dew'))
now.visibility(match.group('vis'))
now.pressure(match.group('baro'), 'inHg')
now.pressure_change(match.group('change'))
mywind = match.group('wind')
now.wind(mywind.replace('From the ', ''))
now.wind_speed(match.group('speed'), 'mph')

Obviously the issue is that the regular expression expects "at", and in
the second line of the html that doesn't parse, there is no at.

The question I have, is how do I go about fixing this. What I want is to
test to see if the line does or doesn't contain an "at", and if not,
change it to contain an "at". I'm just not sure how to code the RE in
python to do this.

Any help would be appreciated.

Matt

 

[Sybren Stuvel]

dmbkiwi enlightened us with:

A couple of other people have contributed code to this project,
particularly relating to the parsing of the websites.
Unfortunately, it is not parsing one particular part of the website
properly. This is because it is expecting the data to be in a
certain form, and occasionally it is in a different form.
Unfortunately this causes the entire script to fail to run.

You seem to expect old HTML. Why not use XHTML only ('tidy' can
convert between them) and use a regular XML parser? Much, much, much
easier! And you won't have to be afraid of messing up your regular
expressions ;-)

Sybren

 

[dmbkiwi]

dmbkiwi enlightened us with:

You seem to expect old HTML. Why not use XHTML only ('tidy' can
convert between them) and use a regular XML parser? Much, much, much
easier! And you won't have to be afraid of messing up your regular
expressions ;-)

Sybren

XML would be nice, but unfortunately I have no choice as to the markup
language used by the site. It's a website on the world wide web, not a
site overwhich I have any control. My regular expressions are at the
mercy of the developers of that site.

Any other suggestions?

Matt

 

[John J. Lee]

XML would be nice, but unfortunately I have no choice as to the markup
language used by the site. It's a website on the world wide web, not a
site overwhich I have any control. My regular expressions are at the
mercy of the developers of that site.

You misunderstand. HTMLTidy (or its descendant, tidylib) reads ugly,
non-conformant HTML and spits out clean, conformant XHTML (or HTML).

uTidylib is a ctypes wrapper of tidylib.

import tidy
from cStringIO import StringIO
tidydoc = tidy.parseString(html)
s = StringIO()
tidydoc.write(s)
tidied_html = s.getvalue()


mxTidy is a wrapper of a shared-library-ized HTMLTidy.

from mx.Tidy import tidy
tidied_html = tidy(html)[2]


John