Unexpected output while walking dirs

[Evan Carmi]

hi,

i am creating a program to go through a directory structure recursively
(including directories below it) and move all files that end in .msf to
a directory above the current level.

the code i have so far is as follows:

<code>

#!/usr/bin/python
#Author: Evan Carmi
#Date: 20061101
#Purpose: To uncorrupt Mozilla Thunderbird mail index files.
#Version: 2.04
top = 'f:\\test\\mail'


import os, time, itertools

#walk and find all files
allfiles = []
for root, dirs, files in os.walk(top, topdown=False):
for name in files:
allfiles.append(os.path.join(root, name))
#remove all non .msf files
index = []
for match in allfiles:
if match.endswith('.msf'):
index.append(match)
print index
#need to fix the problem with adding folders to thunderbird
indexdest = []
indexdest = ['%s\\..\\..\\%s\\%s\\%s' % (x , time.strftime('%Y%m%d%H%M%S'),
os.path.basename(os.path.normpath(x+'\\..')), os.path.basename(x)) for
x in index]
#\\.. to remove the ending and than basename to add it back on
indexdest = [os.path.normpath(i) for i in indexdest]
indexdest = [i.replace('Mail', 'backups-msf') for i in indexdest]

for a, b in itertools.izip(index, indexdest):
os.renames(a, b)

</code>

if the directory structure is:

-test
-Mail
-Local Folders
-mail.binarymanipulations.com

and there are no additional directories inside of Local Folders than everything
works and the script moves the .msf files to a directory on the same level as
-Mail named backups-msf.

the problem occurs when the directory structure:
-test
-Mail
-Local Folders
-mail.binarymanipulations.com

has additional directories inside of Local Folders. This results in very odd
behavior and the directory structure looks like:

-test
-Mail
-Local Folders
-mail.binarymanipulations.com
-20061228005643
-Local Folders
-mail.binarymanipulations.com

after running the script.
i hope that my problem is clear now, if you have any other information that
would be helpful please tell me.
I am testing this on a w2k machine with python 2.4

thanks,
Evan

 

[wittempj]

hi,

i am creating a program to go through a directory structure recursively
(including directories below it) and move all files that end in .msf to
a directory above the current level.

the code i have so far is as follows:

<code>

#!/usr/bin/python
#Author: Evan Carmi
#Date: 20061101
#Purpose: To uncorrupt Mozilla Thunderbird mail index files.
#Version: 2.04
top = 'f:\\test\\mail'

import os, time, itertools

#walk and find all files
allfiles = []
for root, dirs, files in os.walk(top, topdown=False):
for name in files:
allfiles.append(os.path.join(root, name))
#remove all non .msf files
index = []
for match in allfiles:
if match.endswith('.msf'):
index.append(match)
print index
#need to fix the problem with adding folders to thunderbird
indexdest = []
indexdest = ['%s\\..\\..\\%s\\%s\\%s' % (x , time.strftime('%Y%m%d%H%M%S'),
os.path.basename(os.path.normpath(x+'\\..')), os.path.basename(x)) for
x in index]
#\\.. to remove the ending and than basename to add it back on
indexdest = [os.path.normpath(i) for i in indexdest]
indexdest = [i.replace('Mail', 'backups-msf') for i in indexdest]

for a, b in itertools.izip(index, indexdest):
os.renames(a, b)

</code>

if the directory structure is:

-test
-Mail
-Local Folders
-mail.binarymanipulations.com

and there are no additional directories inside of Local Folders than everything
works and the script moves the .msf files to a directory on the same level as
-Mail named backups-msf.

the problem occurs when the directory structure:
-test
-Mail
-Local Folders
-mail.binarymanipulations.com

has additional directories inside of Local Folders. This results in very odd
behavior and the directory structure looks like:

-test
-Mail
-Local Folders
-mail.binarymanipulations.com
-20061228005643
-Local Folders
-mail.binarymanipulations.com

after running the script.
i hope that my problem is clear now, if you have any other information that
would be helpful please tell me.
I am testing this on a w2k machine with python 2.4

thanks,
Evan

If I do get this correct - you have files like \test\Mail\somename.msf
and \test\Mail\somedirectory\someothername.msf, these files you want to
move to \test\backup\timestamp\somename.msf and
\test\backup\timestamp\somedirectory\someothername.msf.

In your code you start with collecting allfiles, and then you remove
all except *,msf files. You can do this in one step to, by collecting
only the wanted files in a listby applying the glob command from module
glob :

allfiles = []
for root, dirs, files in os.walk(top, topdown=False):
targetfiles = glob.glob(os.path.join(root,'*.msf'))
allfiles += targetfiles

Now allfiles contains the list of .msf files down from directory
specified in variable top. From here I would create a list of 2-tuples,
with as first element the full original filename, and as second element
the desired backupname. Note that in your code you call the clock in a
for loop, this might result in more directories.

backuproot = os.path.join(os.path.dirname(top), 'backup-msf',
time.strftime('%Y%m%d%H%M%S'))
backupfilenames = []
for f in allfiles:
backupfilenames.append((f, os.path.join(backuproot,
os.path.basename(f))))

 

[Evan Carmi]

If I do get this correct - you have files like \test\Mail\somename.msf
and \test\Mail\somedirectory\someothername.msf, these files you want to
move to \test\backup\timestamp\somename.msf and
\test\backup\timestamp\somedirectory\someothername.msf.

In your code you start with collecting allfiles, and then you remove
all except *,msf files. You can do this in one step to, by collecting
only the wanted files in a listby applying the glob command from module
glob :

allfiles = []
for root, dirs, files in os.walk(top, topdown=False):
targetfiles = glob.glob(os.path.join(root,'*.msf'))
allfiles += targetfiles

Now allfiles contains the list of .msf files down from directory
specified in variable top. From here I would create a list of 2-tuples,
with as first element the full original filename, and as second element
the desired backupname. Note that in your code you call the clock in a
for loop, this might result in more directories.

backuproot = os.path.join(os.path.dirname(top), 'backup-msf',
time.strftime('%Y%m%d%H%M%S'))
backupfilenames = []
for f in allfiles:
backupfilenames.append((f, os.path.join(backuproot,
os.path.basename(f))))

From here it is easy to do the move.

Thanks so much for your help, you got me started and I finished it with some
help from others as well. For anyone interested the working code is:

--
#!/usr/bin/python
#Author: Evan Carmi
#Date: 20060102
#Purpose: To uncorrupt Mozilla Thunderbird mail index files.
#Version: 1.00
import os, time, glob

srcRoot = 'f:\\test\\mail'
backupRoot = os.path.join(os.path.dirname(srcRoot), 'backup-msf',
time.strftime('%Y%m%d%H%M%S'))

for root, dirs, files in os.walk(srcRoot, topdown=False):
sources = glob.glob(os.path.join(root,'*.msf'))
pairs = []

for source in sources:
# If srcRoot is /foo/bar and source is /foo/bar/baz, let relativeSource
equal /baz
#let relativeSource be equal to the remainder of source when you take
away len(srcRoot)
idx = len(srcRoot)
relativeSource = source[idx:]

# Then let destination equal /quux/baz if backupRoot is /quux
destination = backupRoot + relativeSource
# relativeSource begins with a path separator, so os.path.join will
misinterpret it.

pair = (source, destination)
pairs.append(pair)

for pair in pairs:
os.renames(*pair)
# This is functionally equivalent to os.renames(pair[0], pair[1])