more os.walk() issues... probably user error

[rbt]

This function is intended to remove unwanted files and dirs from
os.walk(). It will return correctly *IF* I leave the 'for fs in
fs_objects' statement out (basically leave out the entire purpose of the
function).

It's odd, when the program goes into that statment... even when only a
'pass', and nothing else is present, nothing is returned. Why is that?
I'm testing Python 2.4 on Linux x86 and WinXP. Results are the same on
either platform.

def build_clean_list(self, path):

file_skip_list = ['search_results.txt']
dir_skip_list = ['dev', 'proc', 'Temporary Internet Files']

fs_objects = os.walk(path, topdown=True)
## for fs in fs_objects:
##
## for f in fs[2]:
## if f in file_skip_list:
## print f
## fs[2].remove(f)
##
## for d in fs[1]:
## if d in dir_skip_list:
## print d
## fs[1].remove(d)

return fs_objects

 

[rbt]

This function is intended to remove unwanted files and dirs from
os.walk(). It will return correctly *IF* I leave the 'for fs in
fs_objects' statement out (basically leave out the entire purpose of the
function).

It's odd, when the program goes into that statment... even when only a
'pass', and nothing else is present, nothing is returned. Why is that?
I'm testing Python 2.4 on Linux x86 and WinXP. Results are the same on
either platform.

def build_clean_list(self, path):

file_skip_list = ['search_results.txt']
dir_skip_list = ['dev', 'proc', 'Temporary Internet Files']

fs_objects = os.walk(path, topdown=True)
## for fs in fs_objects:
##
## for f in fs[2]:
## if f in file_skip_list:
## print f
## fs[2].remove(f)
##
## for d in fs[1]:
## if d in dir_skip_list:
## print d
## fs[1].remove(d)

return fs_objects

Just to clarify, it's wrong of me to say that 'nothing is returned'...
in either case, this is what is returned:

Here's what was returned and its type:

 

[wittempj]

That's an easy one: fs_objects is not modified by your ode, so you get
it back as created by os.walk

 

[Dan Perl]

This function is intended to remove unwanted files and dirs from
os.walk(). It will return correctly *IF* I leave the 'for fs in
fs_objects' statement out (basically leave out the entire purpose of the
function).

It's odd, when the program goes into that statment... even when only a
'pass', and nothing else is present, nothing is returned. Why is that?
I'm testing Python 2.4 on Linux x86 and WinXP. Results are the same on
either platform.

def build_clean_list(self, path):

file_skip_list = ['search_results.txt']
dir_skip_list = ['dev', 'proc', 'Temporary Internet Files']

fs_objects = os.walk(path, topdown=True)
## for fs in fs_objects:
##
## for f in fs[2]:
## if f in file_skip_list:
## print f
## fs[2].remove(f)
##
## for d in fs[1]:
## if d in dir_skip_list:
## print d
## fs[1].remove(d)

return fs_objects

Just to clarify, it's wrong of me to say that 'nothing is returned'... in
either case, this is what is returned:

Here's what was returned and its type:

What do you mean by not being able to iterate over the returned object?
What kind of error are you getting? Have you tried to debug the code?

BTW, os.walk indeed returns a generator. You should familiarize yourself
with generators and iterators if you haven't done so yet.

 

[Dan Perl]

def build_clean_list(self, path):

file_skip_list = ['search_results.txt']
dir_skip_list = ['dev', 'proc', 'Temporary Internet Files']

fs_objects = os.walk(path, topdown=True)
## for fs in fs_objects:
##
## for f in fs[2]:
## if f in file_skip_list:
## print f
## fs[2].remove(f)
##
## for d in fs[1]:
## if d in dir_skip_list:
## print d
## fs[1].remove(d)

return fs_objects

Rather as an aside, the idiom for using os.walk is
for dirpath, dirnames, dirfiles in os.walk(path):
for f in dirnames:
if f in file_skip_list:
print f
filenames.remove(f)
if d in dir_skip_list:
print d
dirnames.remove(f)

More crucially for your code, returning the generator object after having
iterated all the way through it will not do you any good. The generator has
an internal state that puts it at "the end of the iteration" so you cannot
use it to iterate again.

 

[Kent Johnson]

This function is intended to remove unwanted files and dirs from
os.walk(). It will return correctly *IF* I leave the 'for fs in
fs_objects' statement out (basically leave out the entire purpose of
the function).

It's odd, when the program goes into that statment... even when only a
'pass', and nothing else is present, nothing is returned. Why is that?
I'm testing Python 2.4 on Linux x86 and WinXP. Results are the same on
either platform.

def build_clean_list(self, path):

file_skip_list = ['search_results.txt']
dir_skip_list = ['dev', 'proc', 'Temporary Internet Files']

fs_objects = os.walk(path, topdown=True)

fs_objects is a generator, not a list. This loop is exhausting fs_objects, so when you return
fs_objects is at the end of iteration, there is nothing left.

## for fs in fs_objects:
##
## for f in fs[2]:
## if f in file_skip_list:
## print f
## fs[2].remove(f)
##
## for d in fs[1]:
## if d in dir_skip_list:
## print d
## fs[1].remove(d)

Add this here:
yield fs

and take out the return. This turns build_clean_list() into a generator function and you will be
able to iterate the result.

Kent

 

[rbt]

This function is intended to remove unwanted files and dirs from
os.walk(). It will return correctly *IF* I leave the 'for fs in
fs_objects' statement out (basically leave out the entire purpose of
the function).

It's odd, when the program goes into that statment... even when only
a 'pass', and nothing else is present, nothing is returned. Why is
that? I'm testing Python 2.4 on Linux x86 and WinXP. Results are the
same on either platform.

def build_clean_list(self, path):

file_skip_list = ['search_results.txt']
dir_skip_list = ['dev', 'proc', 'Temporary Internet Files']

fs_objects = os.walk(path, topdown=True)

fs_objects is a generator, not a list. This loop is exhausting
fs_objects, so when you return fs_objects is at the end of iteration,
there is nothing left.

That makes sense. Thanks for the explanation. I've never used generators
before.

## for fs in fs_objects:
##
## for f in fs[2]:
## if f in file_skip_list:
## print f
## fs[2].remove(f)
##
## for d in fs[1]:
## if d in dir_skip_list:
## print d
## fs[1].remove(d)

Add this here:
yield fs

and take out the return. This turns build_clean_list() into a generator
function and you will be able to iterate the result.

I'll try this.

Will the changes I made (file and dir removals from os.walk()) be
reflected in the generator object? Is it safe to remove objects this way
and pass the results in a generator on to another function? Sorry for
all the questions, I just like to fully understand something before I
start doing it with confidence.

rbt

 

[Kent Johnson]

## for fs in fs_objects:
##
## for f in fs[2]:
## if f in file_skip_list:
## print f
## fs[2].remove(f)
##
## for d in fs[1]:
## if d in dir_skip_list:
## print d
## fs[1].remove(d)

Will the changes I made (file and dir removals from os.walk()) be
reflected in the generator object? Is it safe to remove objects this way
and pass the results in a generator on to another function? Sorry for
all the questions, I just like to fully understand something before I
start doing it with confidence.

Yes. The docs for os.walk() explicitly state, "When topdown is true, the caller can modify the
dirnames list in-place (perhaps using del or slice assignment), and walk() will only recurse into
the subdirectories whose names remain in dirnames."

So changes to the dir list affect the iteration; changes to the file list directly affect the value
you return to the caller.

Kent