genfromtxt and comment identifier

[simona bellavista]

Hi All,

I have a problem with reading data from a file using genfromtxt of
numpy module.

I have prepared a minimal example similar to the ones presented in

http://docs.scipy.org/doc/numpy/user/basics.io.genfromtxt.html#splitting-the-lines-into-columns

The script is

import numpy as np
from StringIO import StringIO
file = open('esempio.dat')
for line in file.xreadlines() :
if not line : break
print line
np.genfromtxt(StringIO(line), comments="#", delimiter=",")

I have a data file - esempio.dat - like the following:

#
# Skip me !
# Skip me too !
1, 2
3, 4
5, 6 #This is the third line of the data
7, 8
# And here comes the last line
9, 0
"""

The code is breaking at the first line, it looks like it doesn't
recognize # like comment identifier.

The error in the python interpreter is

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "esempio.py", line 7, in <module>
np.genfromtxt(StringIO(line), comments="#", delimiter=",")
File "/opt/numpy/1.5.1/lib/python2.6/site-packages/numpy/lib/
npyio.py", line 1174, in genfromtxt
raise IOError('End-of-file reached before encountering data.')
IOError: End-of-file reached before encountering data.


It is clear that I haven't understood something, what am I doing
wrong?

Cheers

 

[Peter Otten]

Hi All,

I have a problem with reading data from a file using genfromtxt of
numpy module.

I have prepared a minimal example similar to the ones presented in

http://docs.scipy.org/doc/numpy/user/basics.io.genfromtxt.html#splitting- the-lines-into-columns

The script is

import numpy as np
from StringIO import StringIO
file = open('esempio.dat')
for line in file.xreadlines() :
if not line : break
print line
np.genfromtxt(StringIO(line), comments="#", delimiter=",")

I have a data file - esempio.dat - like the following:

#
# Skip me !
# Skip me too !
1, 2
3, 4
5, 6 #This is the third line of the data
7, 8
# And here comes the last line
9, 0
"""

The code is breaking at the first line, it looks like it doesn't
recognize # like comment identifier.

The error in the python interpreter is

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "esempio.py", line 7, in <module>
np.genfromtxt(StringIO(line), comments="#", delimiter=",")
File "/opt/numpy/1.5.1/lib/python2.6/site-packages/numpy/lib/
npyio.py", line 1174, in genfromtxt
raise IOError('End-of-file reached before encountering data.')
IOError: End-of-file reached before encountering data.


It is clear that I haven't understood something, what am I doing
wrong?

The examples use StringIO to simulate a file, but you are wrapping every
line of your actual file. The first "simulated file" is then

StringIO("#\n")

i. e. it contains only a comment, no data -- and that's what genfromtxt()
complains about. Read the file in one fell swoop and you should be OK:

import numpy as np

with open('esempio.dat') as instream:
print np.genfromtxt(instream, comments="#", delimiter=",")