VFAT + Linux/Knoppix bug?

[lbrtchx]

Hi,
~
I just noticed a bug the while coding, that made me spend quite a big
of time eyeballing, cleaning up and mentally remapping my code and I
believe to have reduced to a bug caused by some nasty VFAT + Linux/
Knoppix impedance
~
1._ I am using Linux/Knoppix 5.1.1 based on the 2.6.19 Linux kernel
~
sh-3.1# su -
root@Knoppix:~# uname -r
2.6.19
~
2._ which I booted with the cheatcode
~
"knoppix acpi=off noapic testcd init 2"
~
because the Linux/knoppix version 5.1.1 based on the 2.6.19 fills the
dmesg output with errors:
CPU0: 01(01)
APIC error on CPU0: 01(01) (<- lots of these)
spurious APIC interrupt on CPU#0, should never happen. (also, but not
as many as the previous one)
~
3._ I dumped the output of dmesg
~
root!tty1:/# dmesg > 00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt
~
4._ I tried mounting the vfat fs /dev/hdb6 using
~
root!tty1:/# mount /media/hdb6
~
but then while moving/copying files to directories in this fs I would
get
~
root!tty1:/# mv *.txt /media/hdb6/bugs
root!tty1:/# mv failed to preserve ownership for '/media/hdb6/bugs/
00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt' Operation not permitted
~
mounting the fs using the stanza
~
root!tty1:/# mount -t vfat /dev/hdb6 /media/hdb6
~
appears to rid you of these ownership transfer losses
~
5._ I then started X via
~
root!tty1:/# startx
~
6._ opened a shell went to a folder in the vfat fs and created files
with the same name but with the difference of a lower and upper case
extension
~
root@Knoppix:/media/hdb6/prjx/bugs# echo "test lower case extension" >
test.txt
root@Knoppix:/media/hdb6/prjx/bugs# echo "test upper case extension" >
test.TXT
~
7._ but even if the file with the upper case version is there (which
you only see by invoking it with its full name)
~
root@Knoppix:/media/hdb6/prjx/bugs# ls -l test.TXT
-rwxr-xr-x 1 root root 26 May 7 06:40 test.TXT
~
Linux is not 'seeing it' if you wild card expressions or just list
all files
~
root@Knoppix:/media/hdb6/prjx/bugs# ls -l *.TXT
ls: *.TXT: No such file or directory
root@Knoppix:/media/hdb6/prjx/bugs# ls -l
total 176
-rwxr-xr-x 1 root root 12889 May 7 05:59
00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt
-rwxr-xr-x 1 root root 12889 May 7 05:59 02_dmesg_mount_-
t_vfat_dev_hdb6_media_hdb6.txt
-rwxr-xr-x 1 root root 29501 May 7 05:05
1st_trial_00_dmesg_after_boot.txt
-rwxr-xr-x 1 root root 650 May 7 05:12
1st_trial_04_fdisk_dev_hdb.txt
-rwxr-xr-x 1 root root 1542 May 7 06:39 LinuxVFatBug00.txt
-rwxr-xr-x 1 root root 1342 May 7 06:35 LinuxVFatBug00.txt~
-rwxr-xr-x 1 root root 3687 May 7 05:59 cp--help.txt
-rwxr-xr-x 1 root root 208 May 7 06:02 md5sum00.txt
-rwxr-xr-x 1 root root 526 May 7 06:02 md5sum02.txt
-rwxr-xr-x 1 root root 26 May 7 06:40 test.txt
~
root@Knoppix:/media/hdb6/prjx/bugs# ls -l *.txt
-rwxr-xr-x 1 root root 12889 May 7 05:59
00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt
-rwxr-xr-x 1 root root 12889 May 7 05:59 02_dmesg_mount_-
t_vfat_dev_hdb6_media_hdb6.txt
-rwxr-xr-x 1 root root 29501 May 7 05:05
1st_trial_00_dmesg_after_boot.txt
-rwxr-xr-x 1 root root 650 May 7 05:12
1st_trial_04_fdisk_dev_hdb.txt
-rwxr-xr-x 1 root root 1542 May 7 06:39 LinuxVFatBug00.txt
-rwxr-xr-x 1 root root 3687 May 7 05:59 cp--help.txt
-rwxr-xr-x 1 root root 208 May 7 06:02 md5sum00.txt
-rwxr-xr-x 1 root root 526 May 7 06:02 md5sum02.txt
-rwxr-xr-x 1 root root 26 May 7 06:40 test.txt
~
root@Knoppix:/media/hdb6/prjx/bugs# md5sum *.txt
4d0a8122383095005be55c8daed3aac9
00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt
4d0a8122383095005be55c8daed3aac9 02_dmesg_mount_-
t_vfat_dev_hdb6_media_hdb6.txt
0291b2e145614198136599426f012e42 1st_trial_00_dmesg_after_boot.txt
3120b0f6172325eb4fad5c502b390b7e 1st_trial_04_fdisk_dev_hdb.txt
8380bad18b88152267686976bcbf5213 LinuxVFatBug00.txt
f7dd8b7cc5d6115be6b09186106862d1 cp--help.txt
7b9be97a510ba89b8d06cdff28e02ed8 md5sum00.txt
ec0a023c6a8c31c47232640a16f27ea4 md5sum02.txt
76744305c32618225193b5556c8b091e test.txt
~
root@Knoppix:/media/hdb6/prjx/bugs# md5sum *.TXT
md5sum: *.TXT: No such file or directory
~
Obviously this impedance regarding file names/extensions would
transpire to the JVM, since it works on top of the OS services +
utilities and the OS + VFAT FS
~
However what I don't quite get is why a class implementing
http://java.sun.com/javase/6/docs/api/index.html such as:
~
// __
class DirFilter implements FilenameFilter{
public boolean accept(File FlDir, String aFl){
boolean IsAccept = (FlDir.isDirectory());
if(IsAccept){
File Fl = new File(FlDir, aFl);
IsAccept = Fl.exists();
if(IsAccept){
aFl = Fl.getAbsolutePath();
if(!IsAll){
IsAccept = false;
for(int i = 0; (i < aExtAr.length) && !IsAccept; ++i){
IsAccept = aFl.endsWith(aExtAr);
}
}// (!IsAll)
}// (Fl.exists())
}// (FlDir.isDirectory())
// __
return(IsAccept);
}
}
~
would not pick all files with both extensions
~
when you go
~
DirFilter FlsFltr = new DirFilter();
File[] XFls = FlDir.listFiles(FlsFltr);
~
even if it detected in its accept implemented method both extensions
just fine!!!
~
I degugged and eyeballed it carefully, but you should check it out
yourself
~
I think there is impedance among the three parts:
~
* the OS Linux honoring this MS FAT32 requirements file system
filenames in which you get filenames stored in a case-sensitive
manner, but
filenames being treated as if they were case-insensitive. So this
might be creating this file name "ghosts" problem
* the JVM not knowing what to do with this conflict
~
I code mostly Java and I need to use FAT32 to keep the files in my
external drive easily accessible when I am working in a Linux,
Solaris, WIndows or MacOS box, but this filename thing is creating
problems
~
How do people keep their data portable regarding file system names?
Just using all lower or uppercase?
~
Thanks
lbrtchx

 

[Andreas Leitgeb]

[f'up-to ignored. post here, read here]

(solution to problem further down in this post)

root!tty1:/# mv failed to preserve ownership for '/media/hdb6/bugs/
00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt' Operation not permitted

This is irrelevant. (this is not critizism, but information)
It means, that vfat doesn't support changing of file-owner
as attemted by the "mv" command.

root!tty1:/# mount -t vfat /dev/hdb6 /media/hdb6
appears to rid you of these ownership transfer losses

No, it doesn't. If you now no longer get these warnings,
this must have a different reason.

root@Knoppix:/media/hdb6/prjx/bugs# echo "test lower case extension" >
test.txt
root@Knoppix:/media/hdb6/prjx/bugs# echo "test upper case extension" >
test.TXT

Both were targetted to the *same* file!

vfat is not case-sensitive, but case-preserving.
the first echo created a file "test.txt", the second
recognized test.TXT as being already there, and overwrote
its contents. test.txt now contains "... upper ...".
Since it was already there, it preserved the filename
under which it was created (not last accessed)

If you then access it with a differently cased name, the
system will actually find and access it, but if you want
to apply wildcards, then unix-case-sensitivity creeps in,
and it won't find files that don't match the pattern in
a case-sensitive way.

root@Knoppix:/media/hdb6/prjx/bugs# ls -l test.TXT
-rwxr-xr-x 1 root root 26 May 7 06:40 test.TXT

root@Knoppix:/media/hdb6/prjx/bugs# ls -l *.TXT
ls: *.TXT: No such file or directory

....$ ls -l test.*
-rwxr-xr-x 1 root root 26 May 7 06:40 test.txt

Obviously this impedance regarding file names/extensions would
transpire to the JVM, since it works on top of the OS services +
utilities and the OS + VFAT FS

sure it does. but the rules are easy:
you can access a file on a case-insensitive filesystem with
any case you like.
If you use wildcards, the wildcard-matching is case-sensitive.

IsAccept = aFl.endsWith(aExtAr);

This is the culprit: here you do the case-sensitive filtering yourself!

If you need to deal with case-insensitive filesystems, you had better
do it so:
IsAccept = aFl.toLowerCase().endsWith(aExtAr.toLowerCase());
(or likewise with toUpperCase() in both places.)

How do people keep their data portable regarding file system names?
Just using all lower or uppercase?

If you need to select by extension, and the extension of the files
could be any case, you need extra care for filtering (making the
filtering itself case-insensitive).

I don't think there is a way to do this automatically for
files on case-insensitive filesystems only.

PS: I've seen that one cannot rename a file on a case-INsensitive
filesystem from one case to another. Not even under Windows.
you have to rename it to a dummy name first, then rename it back
to proper case.

 

[Mark Thornton]

IsAccept = aFl.endsWith(aExtAr);

This is the culprit: here you do the case-sensitive filtering yourself!

If you need to deal with case-insensitive filesystems, you had better
do it so:
IsAccept = aFl.toLowerCase().endsWith(aExtAr.toLowerCase());
(or likewise with toUpperCase() in both places.)

How do people keep their data portable regarding file system names?
Just using all lower or uppercase?

If you need to select by extension, and the extension of the files
could be any case, you need extra care for filtering (making the
filtering itself case-insensitive).

This is harder to do correctly than it might seem. The Windows
comparison of filenames is not the same as a Java case insensitive
comparison. Examples where it differs include various Turkish
dotted/dotless letter I. The best you can do is to convert the file
names to UPPER case and compare that. Do NOT convert to lower case, the
result is different for those Turkish characters and some others.

If your operating system is Mac OSX, which by default also has a case
insensitive file system, then the file name comparison is probably
different.

Mark Thornton

 

[Andreas Leitgeb]

IsAccept = aFl.endsWith(aExtAr);

This is the culprit: here you do the case-sensitive filtering yourself!
If you need to deal with case-insensitive filesystems, you had better
do it so:
IsAccept = aFl.toLowerCase().endsWith(aExtAr.toLowerCase());
(or likewise with toUpperCase() in both places.)

This is harder to do correctly than it might seem. The Windows
comparison of filenames is not the same as a Java case insensitive
comparison. Examples where it differs include various Turkish
dotted/dotless letter I. The best you can do is to convert the file
names to UPPER case and compare that.

If you go to international territory, you really can't win, anyway ;-/

I didn't know about these OS(*)-peculiarities. I really thought, that
unicode was clear about case-mapping and character-equivalence, and
that windows just like java would adhere to unicode standard (with
all its immanent faults).

Anyway, thanks for the hint of transforming to uppercase. It seems
to be at least better than lowercase-folding, when we target
international scripts.

Another example: In greek there are also two versions of sigma which
then translate to same capital. I don't know whether windows
considers the two lowercase versions equivalent or not.
Greek also becomes fun with accented (tonos) vowels - are these
equivalent to their base-versions in Windows?
Einai ola ellhnika gia mena...

(*) actually they are human-language-peculiarities, which some pieces of
software account for better than others, or interpret differently.