Trying to design low level hard disk manipulation program

[ragtag99]

I posted this on comp.lang.asm.x86, alt.os.development, comp.arch,
comp.lang.c++

Im working with windows xp professional, NTFS and programming with
MASM, c++ (free compiler) or visual basic 6.0

=== question 1

Primarily Im trying to design a program that has full control over a
hard disk. What it needs to do is find out what sectors haven't been
written to and be able to write anything there, but doesn't count
towards disk space, IOW the data is user defined garbage with no
consequense if overwritten.

As a very gum-and-bandaid idea for an open space hard drive sweep,
could i simply create a file the size of the remaining data in the hard
drive save it then delete it? Would this in effect fill all available
space with user defined garbage? Although ideally Id like to pinpoint
empty sectors and write to them.

=== question 2

Secondarily make a full delete, thus where a file was is now just
zeros. For this, if I were to open a file with any given language
specific 'open' methood re-write the open data with the exact data
length of zeros then resave it, would that accomplish this task? Or
would the data be saved in a new sector and pointed to that leaving the
original data intact but unattainable?
IOW if you open a file from the hard disk make changes and save it,
does it save to the same physical location the file was pulled from?

== am i on the right track?

I've scoured the web with little success. What i am working towards is
using the Master Record Table (MRT) of the NTFS and seeing if i can
find the sectors that aren't written to and write there for the primary
task and follow the pointers and write there for the secondary task.
Does anyone have a reference on how to programmatically access the MRT?

Ive also come across BIOS function INT 13H which is low level disk
operations. This seems to be of help, has anyone had success in using
it?

Thanks for any help.

Jesse

 

[bj7lewis]

Im working with windows xp professional, NTFS and programming with
Try first working with FAT32 for HDD if you can repartition(PartitionMagic)
or FAT12 using FDD) cause some here may know NTFS internals but like many
like me have only looked into FAT## FS and unix/linux FSs... Using FDD is
better cause you only have to start looking at the disk at a 1.14MB scale
and since your HDD partition isn't near 99.999% full this file
EatUpFreeSpace.dat will be too big(many MBs) to read(you as a human[using
HEX editor] at first before you can code the program to read it) the cluster
chain to see where all those EatUpFreeSpace.dat sectors are...

As a very gum-and-bandaid idea for an open space hard drive sweep,
could i simply create a file the size of the remaining data in the hard
drive save it then delete it? Would this in effect fill all available
space with user defined garbage? Although ideally Id like to pinpoint
empty sectors and write to them.

Also note clustors and sectors can be different depending to FS so
##K(FAT32/NTFS) size cluter can take up ## * 1024 sectors and only part the
first 1 sector may really be used. Example Windows for some reason needed to
allocate clustor to a file ##,###,### bytes and Windows went thur and found
many clustors for all but 349 bytes so Window allocates one more of size ##K
and and saves the 349 bytes and the rest of clustor is wasted space [(## *
1024) - 1 sectors never getting anymore delete to hold untill the file is
deleted and Windows later allocate clustors]...

Secondarily make a full delete, thus where a file was is now just
zeros. For this, if I were to open a file with any given language

Doing this does will only make write zeros where EatUpFreeSpace.dat sectors
actucally contained the files data any clustors(I think only the last one)
allocated to EatUpFreeSpace.dat may contain wasted space [(## * 1024) - 1
sectors] in my prev example holding 349 bytes. Filling EatUpFreeSpace.dat
with zeros fills all the other allocated clustor with zero and the last zero
the 349 bytes... So the wasted space now contains what was there before
either an ASCII format charatcer that some drive formatters write or random
data the used to be valid in a old file deleted before EatUpFreeSpace.dat
was created...

would the data be saved in a new sector and pointed to that leaving the
original data intact but unattainable?
IOW if you open a file from the hard disk make changes and save it,
does it save to the same physical location the file was pulled from?

I don't know this...

I've scoured the web with little success. What i am working towards is
using the Master Record Table (MRT) of the NTFS and seeing if i can
find the sectors that aren't written to and write there for the primary
task and follow the pointers and write there for the secondary task.
Does anyone have a reference on how to programmatically access the MRT?

I don't know this... Like I say FAT12 or FAT32 is more well known interally
than NTFS in the public. So look for Master Boot Record(MBR - HDD only) and
Boot Parameter Block(BPB in the bootsector of FDD and bootsector of each HDD
partition pointed to by the MBR)...

Ive also come across BIOS function INT 13H which is low level disk
operations. This seems to be of help, has anyone had success in using
it?

I don't know this...

 

[Phil Carmody]

I posted this on comp.lang.asm.x86, alt.os.development, comp.arch,
comp.lang.c++

Im working with windows xp professional, NTFS and programming with
MASM, c++ (free compiler) or visual basic 6.0

=== question 1

Primarily Im trying to design a program that has full control over a
hard disk. What it needs to do is find out what sectors haven't been
written to and be able to write anything there, but doesn't count
towards disk space, IOW the data is user defined garbage with no
consequense if overwritten.

So there's no consequence if it's not written at all.
Your solution is therefore to simply not do this.

Phil

 

[F.J.K.]

Oh my, so many nonos in one post

I posted this on comp.lang.asm.x86, alt.os.development, comp.arch,
comp.lang.c++

crossposting is great. Try posting in more than 50 groups for better
results though.

Im working with windows xp professional, NTFS and programming with
MASM, c++ (free compiler) or visual basic 6.0

wonderful. We would be especially interested in the VB parts of your
program.

Primarily Im trying to design a program that has full control over a
hard disk. What it needs to do is find out what sectors haven't been
written to and be able to write anything there, but doesn't count
towards disk space, IOW the data is user defined garbage with no
consequense if overwritten.

Sounds like an especially useful feature if you want to hide some kind
of malware. We are always interested in implementations like this, as

Secondarily make a full delete, thus where a file was is now just
zeros. For this, if I were to open a file with any given language
specific 'open' methood re-write the open data with the exact data
length of zeros then resave it, would that accomplish this task?

Perhaps. Of course this would depend largely on implementation specific
/ OS / fs details. As we always try to not focus too much on C++
specific issues, this is a perfect question to ask here.

Ive also come across BIOS function INT 13H which is low level disk
operations. This seems to be of help, has anyone had success in using
it?

Thanks for bringing the BIOS into play. This qualifies as being kind of
plattform independent, as both Linux and Windows PC's have a BIOS. Good
job.

Thanks for any help.

Im *really* sorry I couldn't help more.

Best luck, Felix

 

[ragtag99]

So there's no consequence if it's not written at all.
Your solution is therefore to simply not do this.

If that were the case I wouldn't have asked. Not to mention academic
pursuits, in many cases, have no practical application. But thanks for
taking the time to respond.

 

[ragtag99]

To bj7lewis

Thanks, that was a lot of good information, albeit I still have to look
a bit on NTFS beings that is the file system i prefer but thanks for
the help with all the internal workings

 

[=?ISO-8859-1?Q?Niels_J=F8rgen_Kruse?=]

To bj7lewis

Thanks, that was a lot of good information, albeit I still have to look
a bit on NTFS beings that is the file system i prefer but thanks for
the help with all the internal workings

You seem to want to prevent recovery of deleted files. Have you checked
that this is not a feature of Vista (or XP - I haven't checked)?

MacOS X has had secure deletion of the trash or unused part of the disk
with your choice of paranoia level for some time now.

There is still the problem of used blocks that may have ghosts of older
data, of course.

 

[red floyd]

To bj7lewis

Thanks, that was a lot of good information, albeit I still have to look
a bit on NTFS beings that is the file system i prefer but thanks for
the help with all the internal workings

Take if off of Comp.lang.c++

followups set.

 

[bj7lewis]

a bit on NTFS beings that is the file system i prefer but thanks for

the help with all the internal workings

By all means but NTFS and FAT is like apples and oranges they have
differences but are still fruits... NTFS is designed for security and FAT no
security... But at the "fruit" level they are basic the same NTFS is newer
than FAT32 and supports TB Drives but that is just a matter of ##K cluter
size... So if you start your design in FAT## and move your way to NTFS later
all you really need to learn is Master Record Table (MRT) thingy and NTFS
security but the cluster chains and the way files are allocated should be
the same...

a bit on NTFS beings that is the file system i prefer but thanks for
the help with all the internal workings

Also why is it prefer FAT32 is not dead is it?...

 

[robertwessel2]

By all means but NTFS and FAT is like apples and oranges they have
differences but are still fruits... NTFS is designed for security and FAT no
security... But at the "fruit" level they are basic the same NTFS is newer
than FAT32 and supports TB Drives but that is just a matter of ##K cluter
size... So if you start your design in FAT## and move your way to NTFS later
all you really need to learn is Master Record Table (MRT) thingy and NTFS
security but the cluster chains and the way files are allocated should be
the same...

The layout and structures on an NTFS volume are *nothing* like on a FAT
volume. For example, NTFS doesn't have chains of clusters to define a
file's data area. It has a tree structure indexing runs of clusters.
Directories are stored in B-trees. Free space is managed via a bitmap.
NTFS can store multiple datastreams for each file. About the only
commonality is that FAT and NTFS both store data in sectors on a disk...

 

[Phil Carmody]

If that were the case I wouldn't have asked. Not to mention academic
pursuits, in many cases, have no practical application. But thanks for
taking the time to respond.

Perhaps you shouldn't have asked.

What you actually asked, that is.

Perhaps you should have asked how to implement the high level
functionality that you want rather than how to implement the
misguided choice of low level functionality that you think you
need.

My logic above is sound given the premises you provide.

Phil

 

[ragtag99]

Perhaps you shouldn't have asked.

What you actually asked, that is.

Then Id be asking a question i didnt want to know the answer to. That
makes no sense.

Perhaps you should have asked how to implement the high level
functionality that you want

one of the reasons ive posted on comp.lang.c++ was hoping someone there
knew practical applications of the language.

rather than how to implement the
misguided choice of low level functionality that you think you
need.

....academic pursuits...
but if there is a high level way of doing it i would be interested in
getting the libraries to get the program running, then ill figure them
out after everything is all said and done.

My logic above is sound given the premises you provide.

Your logic is saying i shouldnt have asked a computer realted question
on computer related news groups. Thats pretty solid logic.

But perhaps all this bandwidth could have been saved if instead of
cryptic replies, "witty" one-liners and cyber tuffism the replies were
like: "youre looking at the problem wrong, this site deals with
something similar..." or "c++ has a library called 'sector write'
that..." or
"someone wrote a similar program and posted the source here..." or even
"a better newsgroup for this question would be..."

but i suppose some people gotta fill there free time some way or other.
But thanks to all those that helped though

 

[bj7lewis]

The layout and structures on an NTFS volume are *nothing* like on a FAT

volume. For example, NTFS doesn't have chains of clusters to define a
file's data area. It has a tree structure indexing runs of clusters.
Directories are stored in B-trees. Free space is managed via a bitmap.
NTFS can store multiple datastreams for each file. About the only
commonality is that FAT and NTFS both store data in sectors on a disk...

I said I didn't know anything about NTFS internals but I can still
theorize(NTFS is enough like FAT that you have complex list/tree that you
need to read/follow to find file sectors on the disk) coming with only NTFS
knowledge from and basic Windows networking class while not giving code
help...

 

[Bill Todd]

I said I didn't know anything about NTFS internals but I can still
theorize

Then you should explicitly label your uneducated guesses as 'theories'
rather then present them as fact.

You said, "NTFS is newer than FAT32 and supports TB Drives but that is
just a matter of ##K cluter size." That is incorrect:

1. NTFS is not, in fact, 'newer than FAT32': NTFS debuted with Windows
NT V3.1 in 1993, whereas FAT32 debuted with Windows 95 OSR2 in 1996.

2. NTFS as implemented (using 32-bit cluster IDs) supports drives up to
8 TB in size (possibly up to 16 TB - it's not clear whether it uses
signed or unsigned integers to count clusters) without deviating from
its 4 KB default cluster size at all, while its architecture (allowing
64-bit cluster IDs) supports drives up to 2^75 (or 2^76) in size using
the same standard 4 KB clusters (as well as both larger and smaller
clusters for situations where a different effective block size is
desirable for reasons unrelated to the total drive or file size).

In other words, NTFS's ability to support 'TB drives' (and files) has
nothing whatsoever to do with the cluster sizes that it supports.

By contrast, FAT32 depends heavily upon increasing its cluster size to
support large drives (and for that matter large files up to its 2 or 4
GB maximum file size, because scanning through close to a million linked
FAT32 entries to find a piece of data with a high file address can get
expensive - that's one reason why FAT32 cluster defaults reach their
maximum size of 32 KB quickly, on any drive larger than 32 GB). Its
architectural drive-size limit is 8 TB, and it requires use of 32 KB
clusters to get there.


You said, "the cluster chains and the way files are allocated should be
the same." That is also flat-out wrong:

1. NTFS does not 'chain' clusters at all, let alone use a single such
chain for the entire file as FAT32 does.

2. Unlike FAT32, NTFS uses bitmaps to track space allocation and embeds
small files in their MFT entries rather than allocating an entire
cluster as soon as a single byte exists.


You asked, "why is it prefer FAT32 is not dead is it?" I can help you
out with that as well: the larger a FAT32 file system gets, the longer
it takes to check it after an unclean shutdown (and there's *still* no
guarantee that any structural corruption resulting from that unclean
shutdown can be fixed), whereas being a journaled file system NTFS can
restart in a few seconds at most, regardless of the size of its file
system, and can recover from any normal unclean-shutdown corruption save
the incomplete write of a single, critical disk sector (which drives are
designed to protect against).

- bill