How to arrange many files of C source code

[James Harris]

Up to now I have usually written C modules as stand alone programs and
stored them in separate directories. This has been OK but is
inflexible, making interaction between them and having multiple
generations of a given piece of code awkward. Have you guys any
suggestions on how to organise C source code for use when developing
small to medium size projects?

This may be a generic query and maybe you use a certain tried and
tested scheme for every C program you write but if it helps to explain
the context it is as follows.

I would like to adopt a scheme that I can use for all C code from now
on. The model I have in mind is to have some apps (by which I mean
things such as user-facing programs) and some utilities. The utils
should be usable by multiple apps. At some point it may even be useful
for the apps to call each other. That is not a current requirement,
only a query in my mind as to whether there is some reason that apps
and utilities need to be distinguished.

Different generations of a given app or utility may need to be
available at the same time. In other words, I may need to keep a
certain piece of code around even though there is a newer version of
the same code.

At some point - might as well be now - I think I will need to use some
source code management application: GIT, Bazaar, Subversion or
similar. I might use it initially for local working but later to allow
collaboration. I guess that the choice of SCM affects how source code
is laid out. For example, it would be simpler to manage all files in a
given folder rather than selected ones.

I would like to be able to treat each app or util as an entity and
move or copy it around without having to also copy anything other than
the utils it depends on.

Ideally, compilation would treat the source code directories as read-
only and generate all of its output in a place relevant to the target
architecture; I would like to be able to build the same code for
different operating systems.

The above leads to questions over how best to include headers from
other directories and how to specify what to link together. I would
like to avoid use of environment variables as far as possible. Builds
should take the latest versions of a utility by default and only
choose older ones if told explicitly to do so. Makefiles are great if
building under Unix and I would prefer to use them to specify where
referenced files sit. Not sure what I would use if building under
Windows which I might have to do in some cases.

Sorry that's a long list. Hopefully it is a fairly standard set of
requirements!

James

 

[Shao Miller]

Sorry that's a long list. Hopefully it is a fairly standard set of
requirements!

If you wish to target, let's say, Linux, Mac OS X, Windows, well
Makefiles can work for all three. You can cross-compile for Windows
using MinGW on Linux, though the resulting programs cannot be debugged
with WinDbg (but can be debugged with a Windows-aware gdb). If you
compile for Windows using, let's say, the Windows driver kit (which is
capable of compiling user-mode programs, too), then you'll probably use
a different build system (the WDK's) than for the other targets.

Why do you wish to avoid environment variables? All three of these
operating systems have them, all three have a "path" where programs are
searched-for... Perhaps you wish for a 'make install' process to
install into a common path for each OS?

If you already have things organized a certain way and are running into
complications with using 'make', there might be a better forum than
comp.lang.c for finding assistance. If you're simply wondering about a
suggested organizational structure, it's pretty subjective... Here's
something I would find pretty and useful:

./
./.git/
./ReadMe.txt
./Makefile
./License.txt
./other_project_stuff
./inc/
./inc/portable.h
./inc/common.h
./inc/app1/
./inc/app1/foo.h
./inc/util1/
./inc/util1/bar.h
./src/
./src/app1/
./src/app1/foo.c
./src/app1/main.c
./src/utils1/
./src/utils1/bar.c
./src/utils1/main.c
./bin/
./bin/app1
./bin/util1

Straying from the forum's topic and with regards to MinGW, here's a
Makefile for building Windows programs with MinGW on Linux:


http://git.zytor.com/?p=users/sha0/...e6fc45c20a6e7e99a8;hb=refs/heads/mdiskchk_win

And here is a handy web-page for building programs with the WDK, in case
you choose that route:

http://www.wd-3.com/archive/WinBuild.htm

 

[Malcolm McLean]

Have you guys any
suggestions on how to organise C source code for use when developing
small to medium size projects?

Separate files out into four groups: this program only: this platform only,this program only: any platform, this platform only: many programs, and any platform, many program.

Obviously you want as much code as possible in the files in the last group,and as little as possible in the first group.

Now give the files clean interfaces. Use static heavily to make as many files as possible idempotent. (This doesn't apply on a small embedded system).Don't make a file dependent on another just because it's calling a few trivial roll your own string functions.

Now give each file a unit test, labelled mysourcemain / main.

What you should end up with is programs with trivial driver files consisting of just a short main() that holds everything together.

 

[James Harris]

....

Why do you wish to avoid environment variables?  All three of these
operating systems have them, all three have a "path" where programs are
searched-for...

As a rule, use of environment variables introduces a dependency on
something which is outside the makefile. If they are set incorrectly
the build will fail or, worse, will silently build incorrectly.

I wouldn't rule them out completely but I think they are overused.

Perhaps you wish for a 'make install' process to
install into a common path for each OS?

If you already have things organized a certain way and are running into
complications with using 'make', there might be a better forum than
comp.lang.c for finding assistance.  If you're simply wondering about a
suggested organizational structure, it's pretty subjective...  Here's
something I would find pretty and useful:

   ./
   ./.git/
   ./ReadMe.txt
   ./Makefile
   ./License.txt
   ./other_project_stuff
   ./inc/
   ./inc/portable.h
   ./inc/common.h
   ./inc/app1/
   ./inc/app1/foo.h
   ./inc/util1/
   ./inc/util1/bar.h
   ./src/
   ./src/app1/
   ./src/app1/foo.c
   ./src/app1/main.c
   ./src/utils1/
   ./src/utils1/bar.c
   ./src/utils1/main.c
   ./bin/
   ./bin/app1
   ./bin/util1

Thanks. That's the kind of suggestion I wanted but it leads to some
questions:
* What would be the process to build app1 - change to app1's directory
and run 'make'?
* How could app1's makefile refer to utils1 components?
* How would app1's makefile ensure utils1's components were up to
date?
* What if there were multiple generations of utils1?
* Why put the headers in separate inc folders?
* Does the structure mean there would be one GIT directory for all
those subdirectories? I was thinking that app1 (and utils1 for that
matter) would be a separate project.

James

 

[Shao Miller]

...


As a rule, use of environment variables introduces a dependency on
something which is outside the makefile. If they are set incorrectly
the build will fail or, worse, will silently build incorrectly.

I wouldn't rule them out completely but I think they are overused.

I thought you were referring to using environment variables within your
programs, rather than for building. I think you'd mentioned that you
wanted the programs to be able to find one another; environment
variables seem like a fair way to accomplish that.

Thanks. That's the kind of suggestion I wanted but it leads to some
questions:
* What would be the process to build app1 - change to app1's directory
and run 'make'?

A top-level Makefile can invoke a Makefile in each subdir of src/

* How could app1's makefile refer to utils1 components?
* How would app1's makefile ensure utils1's components were up to
date?
* What if there were multiple generations of utils1?

These questions might meet with good answers in another forum. Perhaps
someone here might have a suggestion for a 'make'-related forum.

* Why put the headers in separate inc folders?

To avoid filename collision in inc/

* Does the structure mean there would be one GIT directory for all
those subdirectories? I was thinking that app1 (and utils1 for that
matter) would be a separate project.

You can track all projects together as a single Git repository, or Git
also allows you to have a top-level repository with sub-repositories,
such that each src/subproj/ could have its own src/subproj/.git/

 

[James Harris]

I thought you were referring to using environment variables within your
programs, rather than for building.  I think you'd mentioned that you
wanted the programs to be able to find one another; environment
variables seem like a fair way to accomplish that.

Sorry - I tried to keep the description short. Let me go the other way
and illustrate by using something specific in the form of the
directory tree you suggested.

Say you wanted to include a header from another directory would you
write something along the lines or

#include "../../src/utils1/header.h"

or would it be better to write

#include "header.h"

and to add a compile switch along the lines of

--header-source ../../src/utils1

In either case how is it best to specify the referred-to directories?
Is the idea of relative subdirectories good or should they be absolute
or should they be relative to a site-chosen base directory? Is there a
better way to specify where the desired header should be looked for
bearing in mind that there may be multiple generations of a given
utility and by default the most recent one is wanted? So if the
directory tree includes

./src/utilA.1
./src/utilA.2

these would be two versions of utilA. I know that if using Unix I
could link ./src/utilA to whatever was the most recent version but
that wouldn't work on Windows and even on Unix I wondered if having
two names for the one directory would confuse a source code management
package.

I know that's quite a long description. I may be better to post
individual questions separately but even an overview of how others
organise their source code is a start so thanks for what you explained
earlier.

Of course, the above is just for compilation includes. There are the
same issues with telling the linker where to look for object modules.

A top-level Makefile can invoke a Makefile in each subdir of src/

If using make I'm not sure I would want to build all packages at once.
For example, if I was working on application A I would only want to
make A and any utilities A depends on (which utilities might have
changed).

These questions might meet with good answers in another forum.  Perhaps
someone here might have a suggestion for a 'make'-related forum.

Thanks. Yes, I'll be able to check up on make specifics that I haven't
used before. I was more interested in how C programmers organise their
source code directories etc so that one module can refer to modules in
other directories.

To avoid filename collision in inc/

Wouldn't the headers for, say, util2 sit in the ./src/util2 directory?

James

 

[Paul N]

A top-level Makefile can invoke a Makefile in each subdir of src/

Isn't that what is warned about in http://miller.emu.id.au/pmiller/books/rmch/
"Recursive Make Considered Harmful" ?

 

[Shao Miller]

Isn't that what is warned about in http://miller.emu.id.au/pmiller/books/rmch/
"Recursive Make Considered Harmful" ?

It seems to be. That web-page says that its content is copyright 1997,
so I'm not sure if it's still relevant. As far as I know, one can do
dependency magic with a 'make' process, these days. I don't recall
suffering from recursive 'make', recently... But maybe other people do.

 

[Shao Miller]

Sorry - I tried to keep the description short. Let me go the other way
and illustrate by using something specific in the form of the
directory tree you suggested.

Say you wanted to include a header from another directory would you
write something along the lines or

#include "../../src/utils1/header.h"

or would it be better to write

#include "header.h"

and to add a compile switch along the lines of

--header-source ../../src/utils1

The latter. For each project, I'd tell the implementation to look for
#include "file.h" in inc/ and inc/thisproj/ If thisproj needed
knowledge of otherproj, thisproj's .c files would do #include
"otherproj/other.h" (which is under inc/).

In either case how is it best to specify the referred-to directories?
Is the idea of relative subdirectories good or should they be absolute
or should they be relative to a site-chosen base directory? Is there a
better way to specify where the desired header should be looked for
bearing in mind that there may be multiple generations of a given
utility and by default the most recent one is wanted? So if the
directory tree includes

./src/utilA.1
./src/utilA.2

these would be two versions of utilA. I know that if using Unix I
could link ./src/utilA to whatever was the most recent version but
that wouldn't work on Windows and even on Unix I wondered if having
two names for the one directory would confuse a source code management
package.

If those are directories, you could build for both. I'm not sure if
you're talking about directories or binaries, here.

I know that's quite a long description. I may be better to post
individual questions separately but even an overview of how others
organise their source code is a start so thanks for what you explained
earlier.

I hope it helps in some small way. It seems a bit "fringe" for
comp.lang.c.

Of course, the above is just for compilation includes. There are the
same issues with telling the linker where to look for object modules.

Well what you might do is have, somewhere on your build system:

./linux_x86/

and invoke 'make' from there, pointing it to the top-level Makefile in
your repository, then it can build:

./linux_x86/obj/app1/main.o
./linux_x86/obj/util1/main.o
./linux_x86/bin/app1
./linux_x86/bin/util1

That would work, for me.

Or, you could peek at what Apple's Bonjour does, which is different.
(To name a random example.)

If using make I'm not sure I would want to build all packages at once.
For example, if I was working on application A I would only want to
make A and any utilities A depends on (which utilities might have
changed).

That is why it might be handy to step inside a src/subdir/ and invoke
'make' from there, using src/subdir/Makefile I'd say the top-level
Makefile should be pretty "dumb".

Thanks. Yes, I'll be able to check up on make specifics that I haven't
used before. I was more interested in how C programmers organise their
source code directories etc so that one module can refer to modules in
other directories.

I think it's fairly subjective. There haven't been too many respondents
to the thread, so it might even be too far from C to interest others.

Wouldn't the headers for, say, util2 sit in the ./src/util2 directory?

I'd rather have them in inc/util2/ and every project looks in inc/ and
inc/thisproj/ for headers. However, I have seen projects that match
what you describe, too.

 

[James Harris]

....

I think it's fairly subjective.  There haven't been too many respondents
to the thread, so it might even be too far from C to interest others.

It did seem odd at first. I thought there would have been a number of
suggestions over how best to organise C source code for multiple
projects. After all, we tend to build up loads of code after
programming for a few years, some of it utility code that is used in
different projects. The code has to be stored somewhere! Possibly
there were few other replies because people agreed with or had little
to add to the first suggestion made. Or perhaps there are no common
approaches that C programmers use. Or maybe folks don't address the
specific things I was asking about. At any rate, I appreciate the help
with this from you and Malcolm. It has given me some ideas for ways to
get started.

James

 

[Ian Collins]

Sorry - I tried to keep the description short. Let me go the other way
and illustrate by using something specific in the form of the
directory tree you suggested.

Say you wanted to include a header from another directory would you
write something along the lines or

#include "../../src/utils1/header.h"

or would it be better to write

#include "header.h"

and to add a compile switch along the lines of

--header-source ../../src/utils1

In either case how is it best to specify the referred-to directories?

It's easier to simply write

#include <utils1/header.h>

Is the idea of relative subdirectories good or should they be absolute
or should they be relative to a site-chosen base directory?

I prefer the latter.

Is there a
better way to specify where the desired header should be looked for
bearing in mind that there may be multiple generations of a given
utility and by default the most recent one is wanted?

If you must (why would you?) do that use your source control tool to
manage the branches.

 

[Lew Pitcher]

It's easier to simply write

#include <utils1/header.h>

and have -I <path to src> as one of your compile options.

which, with GCC, can be dangerous.

Consider, say you have a source-supplied header called
time.h
in your source directory.

When you
#include <time.h>
in your source, you'll get the source-supplied time.h header from your
source directory rather than the system-supplied time.h header from the
standard include directories.

This happens because GCC searches the directories named by the -I
option /before/ it searches the standard system include directories.
(See the GCC instructions on -I for details).

And, yes, I've seen this happen in at least one source package, where the
authors figured that it was simpler to -I their include library (which
included some non-standard headers named with the same names as standard
headers) than it was to
#include "./path/to/application/include/files/header.h"

 

[Ian Collins]

which, with GCC, can be dangerous.

Anything can be dangerous with any compiler if you break simple common
sense rules.

Consider, say you have a source-supplied header called
time.h
in your source directory.

When you
#include <time.h>

Which isn't what I wrote.

I never have headers in the top level directory, so the includes are

in your source, you'll get the source-supplied time.h header from your
source directory rather than the system-supplied time.h header from the
standard include directories.

Which would get fixed either when the code was reviewed, or the first
time it was built.

This happens because GCC searches the directories named by the -I
option /before/ it searches the standard system include directories.
(See the GCC instructions on -I for details).

I've yet to find a compiler that doesn't.

And, yes, I've seen this happen in at least one source package, where the
authors figured that it was simpler to -I their include library (which
included some non-standard headers named with the same names as standard
headers) than it was to
#include "./path/to/application/include/files/header.h"

...if you break simple common sense rules.

 

[James Harris]

....

When you
  #include <time.h>
in your source, you'll get the source-supplied time.h header from your
source directory rather than the system-supplied time.h header from the
standard include directories.

This happens because GCC searches the directories named by the -I
option /before/ it searches the standard system include directories.
(See the GCC instructions on -I for details).

I would have thought a compiler would/should allow a separate set of
directories for quoted includes as opposed to angle bracket includes
because the programmer's intent is different. In fact GCC seems to
support this differentiation with the -I- option and the -iquote
option.

James

 

[Bart van Ingen Schenau]

Isn't that what is warned about in
http://miller.emu.id.au/pmiller/books/rmch/ "Recursive Make Considered
Harmful" ?

Yes, but s/invoke a Makefile in/include a Makefile fragment from/ in the
quote from Shao Miller and the objections from that article are also
covered.

Bart v Ingen Schenau

 

[Bart van Ingen Schenau]

...


It did seem odd at first. I thought there would have been a number of
suggestions over how best to organise C source code for multiple
projects. After all, we tend to build up loads of code after programming
for a few years, some of it utility code that is used in different
projects. The code has to be stored somewhere! Possibly there were few
other replies because people agreed with or had little to add to the
first suggestion made. Or perhaps there are no common approaches that C
programmers use. Or maybe folks don't address the specific things I was
asking about. At any rate, I appreciate the help with this from you and
Malcolm. It has given me some ideas for ways to get started.

Another possibility is that it is actually quite rare to have the code
for multiple projects in a single tree.
Most often, a single source-tree only contains the code for a single
project, and a project uses only a single version of each module (which
can change over time as new versions become available, but at each point
in time there is only one version in use by a particular project).

As for where code gets stored, that is in a repository of a version
control system (VCS), like Subversion or Git. To some extent the
repository layout must mirror the directory structure of the sources on
disk, but it is certainly not a one-to-one correspondence.

James

Bart v Ingen Schenau

 

[Jorgen Grahn]

.

At some point - might as well be now - I think I will need to use some
source code management application: GIT, Bazaar, Subversion or
similar.

It should be now. Good free tools have existed for over thirty years;
not using them is just a way to create more problems for yourself.

/Jorgen

 

[Jorgen Grahn]

Up to now I have usually written C modules as stand alone programs and
stored them in separate directories. This has been OK but is
inflexible, making interaction between them and having multiple
generations of a given piece of code awkward. Have you guys any
suggestions on how to organise C source code for use when developing
small to medium size projects?

This may be a generic query and maybe you use a certain tried and
tested scheme for every C program you write but if it helps to explain
the context it is as follows.

I would like to adopt a scheme that I can use for all C code from now
on. The model I have in mind is to have some apps (by which I mean
things such as user-facing programs) and some utilities. ....

Sorry that's a long list. Hopefully it is a fairly standard set of
requirements!

I remember wondering about all this (natural, since books and
courses rarely cover it).

But I think you're thinking too hard, and trying to find a general
solution to something which is much easier to solve on a case-by-case
basis. You're also inventing terminology (apps, utilities) which I
don't understand.

I have a humble set of hobby projects. They're in version control
(CVS, Git) so there's a structure enforced there already: anything I
want to say "this is version 3.14 of FooBar" about has to be separated
from "Barney 2.18".

Almost all of them are just a flat directory, plus a test/
subdirectory. I have considered creating utility libraries to reuse,
but in the end I decided against it[1]. If there's something I really
need in two projects, I copy it, but remember that there are now two
copies. If the need continues, I can always split it out later. This
hasn't happened so far, and it hasn't been a major problem[2].

/Jorgen

[1] Started to design a socket library, but realized that without an
application, it was just masturbation, and would end up with me
designing a lot of stuff with no practical purpose.

[2] I should add here that I mostly use C++, so there are fewer basic
building blocks missing which I have to provide myself.

 

[Nick Keighley]

Another possibility is that it is actually quite rare to have the code
for multiple projects in a single tree.
Most often, a single source-tree only contains the code for a single
project, and a project uses only a single version of each module (which
can change over time as new versions become available, but at each point
in time there is only one version in use by a particular project).

except we usually end up supporting multiple versions

 

[Nick Keighley]

Up to now I have usually written C modules as stand alone programs and
stored them in separate directories. This has been OK but is
inflexible, making interaction between them and having multiple
generations of a given piece of code awkward. Have you guys any
suggestions on how to organise C source code for use when developing
small to medium size projects?

This may be a generic query and maybe you use a certain tried and
tested scheme for every C program you write but if it helps to explain
the context it is as follows.

I would like to adopt a scheme that I can use for all C code from now
on. The model I have in mind is to have some apps (by which I mean
things such as user-facing programs) and some utilities.
...

Sorry that's a long list. Hopefully it is a fairly standard set of
requirements!

I remember wondering about all this (natural, since books and
courses rarely cover it).

But I think you're thinking too hard, and trying to find a general
solution to something which is much easier to solve on a case-by-case
basis.  You're also inventing terminology (apps, utilities) which I
don't understand.

I have a humble set of hobby projects.  They're in version control
(CVS, Git) so there's a structure enforced there already: anything I
want to say "this is version 3.14 of FooBar" about has to be separated
from "Barney 2.18".

Almost all of them are just a flat directory, plus a test/
subdirectory. I have considered creating utility libraries to reuse,
but in the end I decided against it[1].  If there's something I really
need in two projects, I copy it, but remember that there are now two
copies.  If the need continues, I can always split it out later. This
hasn't happened so far, and it hasn't been a major problem[2].

/Jorgen

[1] Started to design a socket library, but realized that without an
    application, it was just masturbation, and would end up with me
    designing a lot of stuff with no practical purpose.

[2] I should add here that I mostly use C++, so there are fewer basic
    building blocks missing which I have to provide myself.

well maybe fewer but still a lot of them! I have shared libraries for
such things as logging, printing, reading configuration files. If
projects do similar things they share similar facilities (graphics or
maths)