CSV Parser and unusual (?) linesterminator - misunderstanding?

[Tino Lange]

Hi!

I'm trying to use the csv Parser included with Python. Field Delimiter is
"|", Line Delimiter is "#". Unfortunately it doesn't work as expected. The
parser seems to just ignore the 'lineterminator'?

Here's some example:

$ cat test.py
#! /usr/bin/env python

import sys, csv, cStringIO

class SpecialCSVDialect(csv.Dialect):
delimiter = '|'
lineterminator = '#'
quotechar = '"'
doublequote = True
skipinitialspace = False
quoting = csv.QUOTE_MINIMAL

csv.register_dialect("SpecialCSV", SpecialCSVDialect)

memfile = cStringIO.StringIO("1a|1b|1c|1d#2a|2b|2c|2d#3a|3b|3c|3d#")
cfile = csv.reader(memfile, dialect="SpecialCSV")

while 1:
try:
data = cfile.next()
except csv.Error, (errmsg):
print >> sys.stderr, "SpecialCSVError '%s' - aborting...!" % (errmsg)
sys.exit()
except StopIteration:
break
print data

$ ./test.py
['1a', '1b', '1c', '1d#2a', '2b', '2c', '2d#3a', '3b', '3c', '3d#']
$

I would have been expecting that the parser returns three lines, i. e.

['1a', '1b', '1c', '1d']
['2a', '2b', '2c', '2d']
['3a', '3b', '3c', '3d']

Any hints what I'm doing wrong here?

Thanks

Tino

 

[Peter Otten]

I'm trying to use the csv Parser included with Python. Field Delimiter is
"|", Line Delimiter is "#". Unfortunately it doesn't work as expected. The
parser seems to just ignore the 'lineterminator'?

The csv reader accepts '\r' '\r\n' or '\n' as line endings, even mixed in
the same file. This behaviour is hardcoded. Only the writer uses the
lineterminator specified in the dialect.

The following workaround might suffice

memfile = (s + "\n" for s in "1a|1b|1c|1d#2a|2b|2c|2d#3a|3b|3c
3d#".split("#") if s)

Note that the reader expects a row iterator, so the split operation would be
necessary even if arbitrary line terminators were recognized.

Peter

 

[Peter Otten]

memfile = (s + "\n" for s in "1a|1b|1c|1d#2a|2b|2c|2d#3a|3b|3c
3d#".split("#") if s)

I just found out that you need not add "\n" to the line.

Peter

 

[Tino Lange]

The csv reader accepts '\r' '\r\n' or '\n' as line endings, even mixed in
the same file. This behaviour is hardcoded. Only the writer uses the
lineterminator specified in the dialect.

Boah ... Really?

a) this is not in the documentation ... or did I oversee something?

b) this is really unacceptable, or? At least we here have many CSV's with
other lineterminators than '\n'.

Is this going to be changed? Is someone working on it? Or are patches for SF
wanted?

Cheers,

Tino

 

[skip]

Tino> Is this going to be changed? Is someone working on it? Or are
Tino> patches for SF wanted?

A patch that removes this constraint would be helpful.

Skip

 

[Peter Otten]

Boah ... Really?

a) this is not in the documentation ... or did I oversee something?

It's only in the development version of the documentation, see

http://docs.python.org/dev/lib/csv-fmt-params.html

b) this is really unacceptable, or? At least we here have many CSV's with
other lineterminators than '\n'.

Do you need the elaborate (read: odd) escaping techniques of the original
CSV for these? Or would [row.split(fieldsep) for row in
data.split(linesep)] work just as well?

Is this going to be changed? Is someone working on it? Or are patches for
SF wanted?

I would think so.

Peter