IOError:[Errno 27] File too large

[ch.valderanis]

Hi,

Relatively newcomer here.
The following code fails with the above error:
python version used 2.6.2 under linux

filestring='somestring'
for files in glob.glob('*'):
f2=open(files.replace('.xml','.sub'),'w')
f2.write(filestring+files)
f2.close()


The glob commands returns around 10k files.
I want to create 10k new files. the filename should have a different extension (sub instead of xml)and the content of the new file should be some filestring with the addition of the original filename.

I am pretty sure that the files is less than 1kB. Is there another reason for the operation to fail?

Any other comments,remarks welcomed.

Thanks,
Makis

 

[Steven D'Aprano]

Hi,

Relatively newcomer here.
The following code fails with the above error: python version used 2.6.2
under linux

Which part of the code fails? Please copy and paste the entire traceback,
starting with the line "Traceback (most recent call last)" and ending
with the complete error message.

Which file does it fail on?

filestring='somestring'
for files in glob.glob('*'):
f2=open(files.replace('.xml','.sub'),'w')
f2.write(filestring+files)
f2.close()

Some other comments:

You use the name "files" to represent a single filename, rather than
multiple files. That is misleading, a poor choice of name.

"f2" is also a poor name.

You try to change the file extension using string.replace method. This is
risky, and will fail if some file has ".xml" in the filename apart from
the file extension. Worse, if you have a file *without* ".xml", your code
will over-write that file. Better to use the os.path.splitext function
for this.

Is filestring really a constant string like in the snippet above? Is
there any change that it could be an enormous string?


Re-writing your code snippet:


import os
filestring = 'somestring'
for filename in glob.glob('*.xml'):
newname = os.path.splitext(filename)[0] + '.sub'
newfile = open(newname, 'w')
newfile.write(filestring + filename)
newfile.close()

I am pretty sure that the files is less than 1kB.

"Pretty sure"? You need to be 100% certain. We can't tell you whether
this is the case or not.

Is there another reason for the operation to fail?

We don't know which operation has failed. That's why you need to show the
complete traceback.

 

[Dennis Lee Bieber]

Hi,

Relatively newcomer here.
The following code fails with the above error:
python version used 2.6.2 under linux

filestring='somestring'
for files in glob.glob('*'):
f2=open(files.replace('.xml','.sub'),'w')

"Danger Will Robinson, Danger"

What do you expect to have happen for files that did NOT have ".xml"
somewhere in the name. If you only want to do this for *.xml you should
probably ask for that in the glob() call.

Also, the standard library has functions for splitting file names
into prefix and extension...

f2.write(filestring+files)
f2.close()

I am pretty sure that the files is less than 1kB. Is there another reason for the operation to fail?

Any other comments,remarks welcomed.

Provide the full traceback, so we can confirm what line produced the
error.

 

[ch.valderanis]

Dear Steven,

Thank you very much both for your answer and of course your comments. They are taken into account.
I found out that when I touch FILENAME.sub in the command line, I get the same error. So I guess it isn't a problem with the language but rather deeper. I will solve this first before I come back.
Just for completeness:

Which part of the code fails? Please copy and paste the entire traceback,
starting with the line "Traceback (most recent call last)" and ending
with the complete error message.

The code is corrected according to your advice:
Here it the error:

Traceback (most recent call last):
File "createsubmitfiles.py", line 12, in <module>
newfile=open(newname,'w')
IOError: [Errno 27] File too large: 'FILENAME.sub;'

I have changed the actual filename reported by the traceback.
FILENAME.xml does exist before the code is run
FILENAME.sub is not created
FILENAME has a length of 160 characters
I do have space to write all the files in the directory

Is filestring really a constant string like in the snippet above? Is
there any change that it could be an enormous string?

Yes. It is a constant string

"Pretty sure"? You need to be 100% certain. We can't tell you whether
this is the case or not.

When I do an ls -l before I run the code and after the code has failed, all I see are files with size <1kB.

Makis

 

[Steven D'Aprano]

Traceback (most recent call last):
File "createsubmitfiles.py", line 12, in <module>
newfile=open(newname,'w')
IOError: [Errno 27] File too large: 'FILENAME.sub;'

I have changed the actual filename reported by the traceback.
FILENAME.xml does exist before the code is run FILENAME.sub is not
created
FILENAME has a length of 160 characters I do have space to write all the
files in the directory

You are not trying to create a file called "FILENAME.sub". You are trying
to create a file called "FILENAME.sub;", notice the semi-colon.

Taking a wild guess, I think that you are using a Samba share on a Linux
server. A file "FILENAME.xml;" was accidentally creating on this share
from the Linux filesystem layer, since Linux will allow you to use
semicolons in file names. But samba enforces the same restrictions as
Windows, so when you access the file system through sambda, it gives an
error when you try to create a new file "FILENAME.sub;".

This is a wild guess, and could be completely wrong. Please come back and
tell us the solution when you have solved it.



Good luck!

 

[ch.valderanis]

Taking a wild guess, I think that you are using a Samba share on a Linux

server. A file "FILENAME.xml;" was accidentally creating on this share

from the Linux filesystem layer, since Linux will allow you to use

semicolons in file names. But samba enforces the same restrictions as

Windows, so when you access the file system through sambda, it gives an

error when you try to create a new file "FILENAME.sub;".

This is a wild guess, and could be completely wrong. Please come back and

tell us the solution when you have solved it.

So, let me report the solution.
All the files were in an afs filesystem. afs has a limit for the total length of the filenames per directory. for more details take a look at http://www.inf.ed.ac.uk/systems/AFS/topten.html#Tip13
In my case 10k files with ~160characters per filename were just hitting the limit. The error message is basically misleading.

Makis

 

[Dave Angel]

So, let me report the solution.
All the files were in an afs filesystem. afs has a limit for the total length of the filenames per directory. for more details take a look at http://www.inf.ed.ac.uk/systems/AFS/topten.html#Tip13
In my case 10k files with ~160characters per filename were just hitting the limit. The error message is basically misleading.

Makis

Aha! So the file that was too large was the directory file.