Problem using Optional pyparsing

[Nathan Harmston]

Hi,

I know this isnt the pyparsing list, but it doesnt seem like there is
one. I m trying to use pyparsing to parse a file however I cant get
the Optional keyword to work. My file generally looks like this:

ALIGNMENT 1020 YS2-10a02.q1k chr09 1295 42 141045
142297 C 1254 95.06 1295 reject_bad_break 0

or this:

ALIGNMENT 36 YS2-10a08.q1k chrm 208 165 10745
10788 C 44 95.45 593 reject_low 10,14

and my grammar work well for these lines, however somethings the row looks like:
ALIGNMENT 53 YS2-10b03.p1k chr12 180 125 1067465
1067520 C 56 98.21 532|5,2 reject_low 25

So I try to parse the 532 using

from pyparsing import *

integer = Word( nums )
float = Word( nums+".")
identifier = Word( alphanums+"-_." )

alignment = Literal("ALIGNMENT ").suppress()
row_1 = integer.setResultsName("row_1")#.setParseAction(make_int)
src_id = identifier.setResultsName("src_id")
dest_id = identifier.setResultsName("dest_id")
src_start = integer.setResultsName("src_start")#.setParseAction(make_int)
src_stop = integer.setResultsName("src_stop")#.setParseAction(make_int)
dest_start = integer.setResultsName("dest_start")#.setParseAction(make_int)
dest_stop = integer.setResultsName("dest_stop")#.setParseAction(make_int)
row_8 = oneOf("F C").setResultsName("row_8")
length = integer.setResultsName("length")#.setParseAction(make_int)
percent_id = float.setResultsName("percent_id")#.setParseAction(make_float)
row_11 = integer + Optional(Literal("|") + commaSeparatedList )
)#.setResultsName("row_11")#.setParseAction(make_int)
result = Word(alphas+"_").setResultsName("result")
row_13 = commaSeparatedList.setResultsName("row_13")

def make_alilines_status_parser():
return alignment + row_1 + src_id + dest_id + src_start + src_stop
+ dest_start + dest_stop + row_8 + length + percent_id + row_11 +
result + row_13

def parse_alilines_status(ifile):
alilines = make_alilines_status_parser()
for l in ifile:
yield alilines.parseString( l )

However my parser always fails on lines of type 3. Does anyone know
why the Optional part is not working.

Many Thanks in advance

Nathan

 

[Peter Otten]

I know this isnt the pyparsing list, but it doesnt seem like there is
one. I m trying to use pyparsing to parse a file however I cant get
the Optional keyword to work. My file generally looks like this:

ALIGNMENT  1020  YS2-10a02.q1k chr09     1295       42    141045
142297   C    1254 95.06 1295 reject_bad_break 0

or this:

ALIGNMENT  36    YS2-10a08.q1k chrm      208      165     10745
10788   C      44 95.45 593 reject_low 10,14

and my grammar work well for these lines, however somethings the row looks like:
ALIGNMENT  53    YS2-10b03.p1k chr12      180      125   1067465
1067520   C      56 98.21 532|5,2 reject_low 25

So I try to parse the 532 using

from pyparsing import *

integer = Word( nums )
float = Word( nums+".")
identifier = Word( alphanums+"-_." )

alignment = Literal("ALIGNMENT ").suppress()
row_1 = integer.setResultsName("row_1")#.setParseAction(make_int)
src_id = identifier.setResultsName("src_id")
dest_id = identifier.setResultsName("dest_id")
src_start = integer.setResultsName("src_start")#.setParseAction(make_int)
src_stop = integer.setResultsName("src_stop")#.setParseAction(make_int)
dest_start = integer.setResultsName("dest_start")#.setParseAction(make_int)
dest_stop = integer.setResultsName("dest_stop")#.setParseAction(make_int)
row_8 = oneOf("F C").setResultsName("row_8")
length = integer.setResultsName("length")#.setParseAction(make_int)
percent_id = float.setResultsName("percent_id")#.setParseAction(make_float)
row_11 = integer + Optional(Literal("|") + commaSeparatedList )
)#.setResultsName("row_11")#.setParseAction(make_int)
result = Word(alphas+"_").setResultsName("result")
row_13 = commaSeparatedList.setResultsName("row_13")

def make_alilines_status_parser():
    return alignment + row_1 + src_id + dest_id + src_start + src_stop
+ dest_start + dest_stop + row_8 + length + percent_id + row_11 +
result + row_13

def parse_alilines_status(ifile):
    alilines = make_alilines_status_parser()
    for l in ifile:
        yield alilines.parseString( l )

However my parser always fails on lines of type 3. Does anyone know
why the Optional part is not working.

The commaSeparatedList includes the rest of the line into its last item:
(['a', 'b c'], {})

You can fix this by defining your own delimitedList that doesnt accept
whitespace, e. g.:
(['a', 'b'], {})

Peter

 

[Paul McGuire]

Hi,

I know this isnt the pyparsing list, but it doesnt seem like there is
one. I m trying to use pyparsing to parse a file however I cant get
the Optional keyword to work.

<snip>

Thanks, Peter, your comments are dead-on.

Pyparsing-related posts crop up here every so often, usually with me
as the culprit. But to address your question, I've added a "Getting
Help" page to the pyparsing wiki, with links to the various mailing
lists and support pages on SourceForge. Fortunately, I don't think
this group minds an occasional trip down Pyparsing Lane (there've been
no serious complaints so far).

Some other suggestions/comments on your parser:
- Good use of results names. For debugging, you can print out the
parse results by using the dump() method - this gives you this kind of
output:
[1020, 'YS2-10a02.q1k', 'chr09', 1295, 42, 141045, 142297, 'C',
1254, 95.060000000000002, 1295, 'reject_bad_break', '0']
- dest_id: chr09
- dest_start: 141045
- dest_stop: 142297
- length: 1254
- percent_id: 95.06
- result: reject_bad_break
- row_1: 1020
- row_11: [1295]
- row_13: ['0']
- row_8: C
- src_id: YS2-10a02.q1k
- src_start: 1295
- src_stop: 42
This may give you some ideas on some more meaningful names ('row_8'
looks like an obvious candidate for replacement, for instance).

- If you are using pyparsing 1.4.7, you can abbreviate your calls to
setResultsName to just ("name"), that is, this:
src_id = identifier.setResultsName("src_id")
can be written as just:
src_id = identifier("src_id")

- Don't use the name 'float' for the expression for a real number,
since this masks the builtin Python type float, and more importantly,
the builtin Python conversion function float (which you will want to
use in the parse action for this expression). Perhaps a name like
'real' or 'realNumber' would do.

- There is no need for the trailing space in defining
Literal("ALIGNMENT").

- It is simpler to attach the parse actions to integer and real
themselves, rather than on each line where they are used. If you end
up with some integer field that you don't want converted, you can
define it in situ, like this:
zipCode = Word(nums) # don't use integer for this, don't want it
converted
or define a intString expression that does not do the conversion.

- With these changes, your grammar section becomes a little more
readable:
make_int = lambda t: int(t[0])
make_float = lambda t: float(t[0])
integer = Word( nums ).setParseAction(make_int)
real = Word( nums+".").setParseAction(make_float)
identifier = Word( alphanums+"-_." )

alignment = Literal("ALIGNMENT").suppress()
row_1 = integer("row_1")
src_id = identifier("src_id")
dest_id = identifier("dest_id")
src_start = integer("src_start")
src_stop = integer("src_stop")
dest_start = integer("dest_start")
dest_stop = integer("dest_stop")
row_8 = oneOf("F C")("row_8")
length = integer("length")
percent_id = real("percent_id")
row_11 = (integer +
Optional(Literal("|") +
delimitedList(Word(nums))("subItems"))
)("row_11")
result = Word(alphas+"_")("result")
row_13 = commaSeparatedList("row_13")

(although all the ()'s make row_11 a little tough to follow - this
might deserve getting broken up into several lines.) And yes, I know
that PEP8 says not to align '=' signs on successive assignment
statments, but I truly believe it does help readability at times.

-- Paul