fcntl problems

[mhearne808[insert-at-sign-here]gmail[insert-dot-he]

I'm having a number of problems with the fcntl module. First off, my
system info:

Mac OS X
Darwin igskcicglthearn.cr.usgs.gov 8.10.1 Darwin Kernel Version
8.10.1: Wed May 23 16:33:00 PDT 2007; root:xnu-792.22.5~1/RELEASE_I386
i386 i386
Python 2.5.1 (built from source)

OK, the weirdness:

First of all, if I try this:
file = open("counter.txt","w+")
fcntl.flock(file.fileno(), fcntl.LOCK_NB)

I get this:
---------------------------------------------------------------------------
<type 'exceptions.IOError'> Traceback (most recent call
last)
/Users/mhearne/src/python/<ipython console> in <module>()
<type 'exceptions.IOError'>: [Errno 9] Bad file descriptor

However, if I try this:
fcntl.flock(file.fileno(), fcntl.LOCK_EX)

I get no errors.

Proceeding forward with the locked file, let's say I do the above in
Python interactive Process A. Then in python interactive Process B, I
repeat the "open" function on the same file with the same
permissions. Then, in each process, I write some text to the file
using the write() method. After closing the file in both processes,
the only text I see in the file is from Process B!

According to my Python Cookbook:
"Exclusive lock: This denies all _other_ processes both read and write
access to the file."

I seem to be experiencing the reverse of that description. Is this my
lack of understanding, or have I discovered a bug?

Thanks,

Mike

 

[mhearne808[insert-at-sign-here]gmail[insert-dot-he]

I'm having a number of problems with the fcntl module. First off, my
system info:

Mac OS X
Darwin igskcicglthearn.cr.usgs.gov 8.10.1 Darwin Kernel Version
8.10.1: Wed May 23 16:33:00 PDT 2007; root:xnu-792.22.5~1/RELEASE_I386
i386 i386
Python 2.5.1 (built from source)

OK, the weirdness:

First of all, if I try this:
file = open("counter.txt","w+")
fcntl.flock(file.fileno(), fcntl.LOCK_NB)

I get this:
---------------------------------------------------------------------------
<type 'exceptions.IOError'> Traceback (most recent call
last)
/Users/mhearne/src/python/<ipython console> in <module>()
<type 'exceptions.IOError'>: [Errno 9] Bad file descriptor

However, if I try this:
fcntl.flock(file.fileno(), fcntl.LOCK_EX)

I get no errors.

Proceeding forward with the locked file, let's say I do the above in
Python interactive Process A. Then in python interactive Process B, I
repeat the "open" function on the same file with the same
permissions. Then, in each process, I write some text to the file
using the write() method. After closing the file in both processes,
the only text I see in the file is from Process B!

According to my Python Cookbook:
"Exclusive lock: This denies all _other_ processes both read and write
access to the file."

I seem to be experiencing the reverse of that description. Is this my
lack of understanding, or have I discovered a bug?

Thanks,

Mike

I've been doing some experiments, and here are some specific examples
to try. I'll designate the two interactive python processes as PA and
PB. Both processes were started in the same directory. Here goes:
PA: import fcntl
PA: f = open("foo.txt","w+")
PA: fcntl.flock(f.fileno(),fcntl.LOCK_EX)
PA: f.write("text1\n")
PB: f = open("foo.txt","w+")
PB: f.write("text2\n")
PA: f.close()
PB: f.close()

contents of foo.txt are:
text2

Second experiment:

PA: f = open("foo.txt","w+")
PA: fcntl.flock(f.fileno(),fcntl.LOCK_EX)
PB: f = open("foo.txt","w+")
PA: f.write("text1\n")
PB: f.write("text2\n")
PA: f.write("text3\n")
PB: f.close()
PA: f.write("text4\n")
PA: f.close()

contents of foo.txt are:
text1
text3
text4

Third experiment:
PA: f = open("foo.txt","w+")
PA: fcntl.flock(f.fileno(),fcntl.LOCK_EX)
PA: f.write("text1\n")
PB: f = open("foo.txt","w+")
PB: f.write("text2\n")
PB: f.close()
PA: f.close()

contents of foo.txt are:
text1

Fourth experiment:
PA: f = open("foo.txt","w+")
PA: f.write("text1\n")
PB: f = open("foo.txt","w+")
PB: f.write("text2\n")
PB: f.close()
PA: f.close()

contents of foo.txt are:
text1

From these last two experiments I can only conclude that file locking

isn't doing a durned thing.

What's going on?

--Mike

 

[Miles]

I'm having a number of problems with the fcntl module.

Read this first: http://linux.die.net/man/2/flock

First of all, if I try this:
file = open("counter.txt","w+")
fcntl.flock(file.fileno(), fcntl.LOCK_NB)

I get this:
---------------------------------------------------------------------------
<type 'exceptions.IOError'> Traceback (most recent call
last)
/Users/mhearne/src/python/<ipython console> in <module>()
<type 'exceptions.IOError'>: [Errno 9] Bad file descriptor

That should be:

Proceeding forward with the locked file, let's say I do the above in
Python interactive Process A. Then in python interactive Process B, I
repeat the "open" function on the same file with the same
permissions. Then, in each process, I write some text to the file
using the write() method. After closing the file in both processes,
the only text I see in the file is from Process B!

This is due to two issues: caching and file position. When you open
the file in both processes as 'w+', they are both positioned at the
*current* EOF, but from that point on the offset is not externally
influenced. The correct sequence of events should be:
- open file in mode w+
- obtain exclusive lock
- f.seek(0, 2) # (to end of file)
- write to file
- f.flush() # or f.close()
- release lock

Is this my lack of understanding, or have I discovered a bug?

If you find yourself asking this question, it's too often the former

-Miles

 

[Miles]

Sorry, that last quote-only reply was accidental.

I've been doing some experiments, and here are some specific examples
to try.

[snipped examples]

From these last two experiments I can only conclude that file locking
isn't doing a durned thing.

What's going on?

File locking isn't doing a durned thing in those cases because you're
only obtaining the lock from a single process.

According to my Python Cookbook:
"Exclusive lock: This denies all _other_ processes both read and write
access to the file."

This is only for mandatory locking; POSIX flock is advisory locking,
which states: "Only one process may hold an exclusive lock for a given
file at a given time." Advisory locks don't have any effect on
processes that don't use locks. Mandatory locks are kernel enforced,
but non-POSIX and not available in Mac OS X.

-Miles

 

[mhearne808[insert-at-sign-here]gmail[insert-dot-he]

Sorry, that last quote-only reply was accidental.

I've been doing some experiments, and here are some specific examples
to try.

[snipped examples]

From these last two experiments I can only conclude that file locking
isn't doing a durned thing.

What's going on?

File locking isn't doing a durned thing in those cases because you're
only obtaining the lock from a single process.

According to my Python Cookbook:
"Exclusive lock: This denies all _other_ processes both read and write
access to the file."

This is only for mandatory locking; POSIX flock is advisory locking,
which states: "Only one process may hold an exclusive lock for a given
file at a given time." Advisory locks don't have any effect on
processes that don't use locks. Mandatory locks are kernel enforced,
but non-POSIX and not available in Mac OS X.

-Miles

I think I'm still confused. Maybe I should explain the behavior that
I want, and then figure out if it is possible.

I have a script that will be run from a cron job once a minute. One
of the things this script will do is open a file to stash some
temporary results. I expect that this script will always finish its
work in less than 15 seconds, but I didn't want to depend on that.
Thus I started to look into file locking, which I had hoped I could
use in the following fashion:

Process A opens file foo
Process A locks file foo
Process A takes more than a minute to do its work
Process B wakes up
Process B determines that file foo is locked
Process B quits in disgust
Process A finishes its work

Since I couldn't figure out file locking, I decided to just have
Process A create a "pid" file in the directory - analogous to the
"Occupied" sign on an airplane bathroom. This works, but it seems a
little hacky.

--Mike

 

[Hrvoje Niksic]

I think I'm still confused.

What Miles tried to tell you is that you should call fcnt.flock from
both PA and PB. In the example you posted, you failed to call it from
PB. No lock call, so no locking happened.

I have a script that will be run from a cron job once a minute. One
of the things this script will do is open a file to stash some
temporary results. I expect that this script will always finish its
work in less than 15 seconds, but I didn't want to depend on that.

Thus I started to look into file locking, which I had hoped I could
use in the following fashion:

Process A opens file foo
Process A locks file foo
Process A takes more than a minute to do its work
Process B wakes up
Process B determines that file foo is locked
Process B quits in disgust
Process A finishes its work

File locking supports that scenario, as you suspected. You need to
use flock with LOCK_EX|LOCK_NB. If the call succeeds, you got the
lock. If you get an exception whose errno is EWOULDBLOCK, you quit in
disgust.

 

[Miles]

I have a script that will be run from a cron job once a minute. One
of the things this script will do is open a file to stash some
temporary results. I expect that this script will always finish its
work in less than 15 seconds, but I didn't want to depend on that.
Thus I started to look into file locking, which I had hoped I could
use in the following fashion:

Process A opens file foo
Process A locks file foo
Process A takes more than a minute to do its work
Process B wakes up
Process B determines that file foo is locked
Process B quits in disgust
Process A finishes its work

That would look like (untested):

import fcntl, sys
f = open('foo', 'w+')
try:
fcntl.flock(f.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
except IOError, e:
if e.args[0] == 35:
sys.exit(1)
else:
raise
f.seek(0, 2) # seek to end
# do your thing with the file
f.flush()
fcntl.flock(f.fileno(), fcntl.LOCK_UN)
f.close()

-Miles

 

[mhearne808[insert-at-sign-here]gmail[insert-dot-he]

I have a script that will be run from a cron job once a minute. One
of the things this script will do is open a file to stash some
temporary results. I expect that this script will always finish its
work in less than 15 seconds, but I didn't want to depend on that.
Thus I started to look into file locking, which I had hoped I could
use in the following fashion:

Process A opens file foo
Process A locks file foo
Process A takes more than a minute to do its work
Process B wakes up
Process B determines that file foo is locked
Process B quits in disgust
Process A finishes its work

That would look like (untested):

importfcntl, sys
f = open('foo', 'w+')
try:
fcntl.flock(f.fileno(),fcntl.LOCK_EX |fcntl.LOCK_NB)
except IOError, e:
if e.args[0] == 35:
sys.exit(1)
else:
raise
f.seek(0, 2) # seek to end
# do your thing with the file
f.flush()fcntl.flock(f.fileno(),fcntl.LOCK_UN)
f.close()

-Miles

I tested that, and it works! Thanks!

Looking at my flock(3) man page, I'm guessing that "35" is the error
code for EWOULDBLOCK. Which system header file am I supposed to look
in to figure that magic number out?

I would make the argument that this module could be either more
pythonic, or simply documented more completely. The open source
response, of course, would be "go for it!".

--Mike

 

[Miles]

Looking at my flock(3) man page, I'm guessing that "35" is the error
code for EWOULDBLOCK. Which system header file am I supposed to look
in to figure that magic number out?

I got the error number by looking at the IOError exception raised when
playing with the interactive interpreter, but I really should have
written:

from errno import EWOULDBLOCK
....
if e.args[0] == EWOULDBLOCK:
....

- Miles

 

[Michael J. Fromberger]

"mhearne808[insert-at-sign-here]gmail[insert-dot-here]com"

Looking at my flock(3) man page, I'm guessing that "35" is the error
code for EWOULDBLOCK. Which system header file am I supposed to look
in to figure that magic number out?

On a MacOS system, you can find them in /usr/include/sys/errno.h
On a Linux system, try /usr/include/asm-generic/errno.h

However, if you're writing in Python, you will probably have an easier
time using the "errno" module, e.g.,

] import errno
] errno.errorcode[35]
'EDEADLOCK'

Note that some codes have multiple names (e.g., EAGAIN and EWOULDBLOCK)
so that this lookup may not return exactly the name you're expecting.

Cheers,
-M

 

[Lawrence D'Oliveiro]

except IOError, e:
if e.args[0] == 35:

Why not

except IOError, (ErrNo, Msg) :
if ErrNo == errno.EAGAIN :