Hardlink sub-directories and files

[loial]

I am trying to hardlink all files in a directory structure using
os.link.

This works fine for files, but the directory also contains sub-
directories (which themselves contain files and sub-directories).
However I do not think it is possible to hard link directories ?

So presumably I would need to do a mkdir for each sub-directory
encountered?
Or is there an easier way to hardlink everything in a directory
structure?.

The requirement is for hard links, not symbolic links

 

[Peter Otten]

I am trying to hardlink all files in a directory structure using
os.link.

This works fine for files, but the directory also contains sub-
directories (which themselves contain files and sub-directories).
However I do not think it is possible to hard link directories ?

So presumably I would need to do a mkdir for each sub-directory
encountered?
Or is there an easier way to hardlink everything in a directory
structure?.

The requirement is for hard links, not symbolic links

You cannot make hardlinks for directories, that's a restriction imposed by
the operating system.

Look for shutil.copytree() for a template of what you are trying to do; you
might even monkepatch it

# untested
copy2 = shutil.copy2
shutil.copy2 = os.link
try:
shutil.copytree(source, dest)
finally:
shutil.copy2 = copy2

if you are OK with a quick-and-dirty solution.

 

[Thomas Jollans]

I am trying to hardlink all files in a directory structure using
os.link.

This works fine for files, but the directory also contains sub-
directories (which themselves contain files and sub-directories).
However I do not think it is possible to hard link directories ?

So presumably I would need to do a mkdir for each sub-directory
encountered?
Or is there an easier way to hardlink everything in a directory
structure?.

The requirement is for hard links, not symbolic links

Yes, you have to mkdir everything. However, there is an easier way:

subprocess.Popen(['cp','-Rl','target','link'])

This is assuming that you're only supporting Unices with a working cp
program, but as you're using hard links, that's quite a safe bet, I
should think.

- Thomas

 

[Tim Chase]

I am trying to hardlink all files in a directory structure using
os.link.
Or is there an easier way to hardlink everything in a directory
structure?.

The requirement is for hard links, not symbolic links

While Peter & Thomas gave good answers, also be aware that
hard-links can't cross mount-points (an OS limitation). So if
you have something mounted under the directory you're trying to
hard-link-copy, attempting to create a hard-link will fail for
things within that mount.

-tkc

 

[Dennis Lee Bieber]

That is pretty true. I've heard of hardlinked directories on Solaris, but
that's kind of an exception to the general rule.

It's been 15 years, but I think one could create them on the Amiga.
Softlinks where the tricky one to use, since they were just data files
with full path to the linked file as the data -- applications basically
had to check if it was a softlink, if so, open/read/close then open the
linked path.

 

[Nobody]

However I do not think it is possible to hard link directories ?

Modern Unices disallow hard links to directories, as it makes the
directory "tree" not a tree, so anything which performs a recursive walk
must explicitly check for cycles to avoid infnite recursion.

Older systems which allowed hard links to directories required root
privilege to do so.

 

[Thomas Jollans]

On Tue, Aug 2, 2011 at 3:13 AM, Thomas Jollans <[email protected]

I am trying to hardlink all files in a directory structure using
os.link.

However I do not think it is possible to hard link directories ?

That is pretty true. I've heard of hardlinked directories on Solaris,
but that's kind of an exception to the general rule.

So presumably I would need to do a mkdir for each sub-directory
encountered?
Or is there an easier way to hardlink everything in a directory
structure?.

The requirement is for hard links, not symbolic links

Yes, you have to mkdir everything. However, there is an easier way:

subprocess.Popen(['cp','-Rl','target','link'])

This is assuming that you're only supporting Unices with a working cp
program, but as you're using hard links, that's quite a safe bet, I
should think.


A little more portable way:

$ cd from; find . -print | cpio -pdlv ../to
cpio: ./b linked to ../to/./b
../to/./b
cpio: ./a linked to ../to/./a
../to/./a
cpio: ./c linked to ../to/./c
../to/./c
../to/./d
cpio: ./d/1 linked to ../to/./d/1
../to/./d/1
cpio: ./d/2 linked to ../to/./d/2
../to/./d/2
cpio: ./d/3 linked to ../to/./d/3
../to/./d/3
0 blocks

However, you could do it without a shell command (IOW in pure python)
using os.path.walk().

Is it more portable? I don't actually have cpio installed on this
system. Which implementations of cp don't implement -R and -l? Of
course, the best way is probably implementing this in Python, either
with os.path.walk, or with a monkey-patched shutil.copytree, as Peter
suggested.

Thomas

 

[Grant Edwards]

In APUE, Richard Stevens says only root could do this,

Yep, in early versions of Solaris root could hard-link directories.
I did it once, and it's not something one did a second time. fsck
couldn't deal with it and pretty much fell over. IIRC, the only way
to recover was to clear several inodes manually and then let fsck
salvage things.

if it is supported by the system at all. In a footnote, he
additionally mentions he screwed up his filesystem by creating a loop
of hardlinked directories while writing that section of the book.

That sounds about right.

I suppose it is a good thing systems don't allow that now.

It wouldn't be a problem, except there are some important places in
Unix where it is assume that filesystems are trees. Hard linking
directories causes that assumption to be false.

 

[Thomas Jollans]

It wouldn't be a problem, except there are some important places in
Unix where it is assume that filesystems are trees. Hard linking
directories causes that assumption to be false.

It is generally assumed that ".." is clearly defined, i.e. that every
directory has a parent. With hard linked directories, this would break down.

 

[Thomas Jollans]

On Wed, Aug 3, 2011 at 2:47 AM, Thomas Jollans <[email protected]

Is it more portable? I don't actually have cpio installed on this
system.


Interesting. Of course, it's probably readily available to you. What
*ix are you seeing that doesn't include cpio by default?

Arch Linux - the base install is quite minimal. I just discovered that I
have a program called bsdcpio which is used by mkinitcpio (and possibly
other system scripts); no need for the GNU cpio. Curious.

Which implementations of cp don't implement -R and -l?


Probably most of them, except GNU and newer BSD.

Okay. While GNU libc manuals usually document how portable functions are
in detail, that's not true for the GNU coreutils manuals.

Thomas

 

[Ned Deily]

Yep, in early versions of Solaris root could hard-link directories.
I did it once, and it's not something one did a second time. fsck
couldn't deal with it and pretty much fell over. IIRC, the only way
to recover was to clear several inodes manually and then let fsck
salvage things.


That sounds about right.


It wouldn't be a problem, except there are some important places in
Unix where it is assume that filesystems are trees. Hard linking
directories causes that assumption to be false.

FWIW, Apple implemented directory hard links for HFS+ file systems in
Mac OS X 10.5 specifically to make the Time Machine incremental backup
system work efficiently. The hard directory links are, of course, not
meant to be used by the casual user. The section "Directory Hard Links"
in this blog entry from Amit Singh explains what restrictions are
enforced on directory hard links to prevent cycles:

http://osxbook.com/blog/2008/11/09/hfsdebug-40-and-new-hfs-features/

 

[Thomas Jollans]

Arch Linux - the base install is quite minimal. I just discovered that I
have a program called bsdcpio which is used by mkinitcpio (and possibly
other system scripts); no need for the GNU cpio. Curious.


I guess that makes some sense. If you want to really strip down an
install, removing cpio is a good candidate since it duplicates what's in
tar, and tar is more popular - especially for interactive use.


Okay. While GNU libc manuals usually document how portable functions are
in detail, that's not true for the GNU coreutils manuals.


I don't think cpio is in GNU coreutils. Also, I think GNU cpio is a
reimplementation, not the original.

Indeed. But cp is in the coreutils, and that was what we were talking about.

As for GNU cpio, that's simply what /usr/bin/cpio, if present, is
expected to be on a GNU/Linux system.

 

[Alexander Gattin]

Hello,

This works fine for files, but the directory
also contains sub- directories (which themselves
contain files and sub-directories). However I
do not think it is possible to hard link
directories ?

On some Unices it is, as I heard. But in general,
it's not always possible to even hardlink to an
ordinary file:
1) across filesystem boundary
2) on GRSEC system, if you don't own the original
file