Version Control Software

[Chris Angelico]

If you're bringing in the *entire* CPython code base, as shown here:

http://hg.python.org/

keep in mind that it includes the equivalent of four independent
implementations:

- CPython 2.x
- CPython 3.x
- Stackless
- Jython

Hrm. Why are there other Pythons in the cpython repository? Yes,
CPython 2.x and 3.x, but why the other two?

Plus, no offence intended at Pike which I'm sure is an awesome language,
but it may not be quite as much active development as Python... as you
point out yourself, there are nearly three times as many commits to
CPython as to Pike, which coincidentally (or not) corresponds to the
CPython repo being nearly three times as large as the Pike repo.

Yeah. Actually, I suspect that what's going on here, and what led to
my confusion, is that Pike wasn't always done using git, so quite a
few of the earlier versions simply aren't here. So it's an error in my
perceptions rather than any real difference.

However, comparisons aside, 200MB is still a fair bit to fetch before
doing anything with Python. Does Mercurial have any equivalent of
git's shallow clone feature?

ChrisA

 

[Terry Reedy]

This is the python.org collection of repositories, not just cpython.

Hrm. Why are there other Pythons in the cpython repository?

There are not. The cpython repository
http://hg.python.org/cpython/
only contains cpython. As I write, the last revision is 84110. Windows
says that my cpython clone has about 1400 folders, 15000 files, and 500
million bytes

 

[Chris Angelico]

There are not. The cpython repository
http://hg.python.org/cpython/
only contains cpython. As I write, the last revision is 84110. Windows says
that my cpython clone has about 1400 folders, 15000 files, and 500 million
bytes

Ah, well it's this one that I have. So it should have only CPython in it.

ChrisA

 

[Chris “Kwpolska†Warrick]

Hmm. ~/cpython/.hg is 200MB+, but ~/pike/.git is only 86MB. Does
Mercurial compress its content? A tar.gz of each comes down, but only
to ~170MB and ~75MB respectively, so I'm guessing the bulk of it is
already compressed. But 200MB for cpython seems like a lot.

Next time, do a more fair comparison.

I created an empty git and hg repository, and created a file promptly
named “file†with DIGIT ONE (0x31; UTF-8/ASCII–encoded) and commited
it with “c1†as the message, then I turned it into “12†and commited
as “c2†and did this one more time, making the file “123†at commit
named “c3â€.

[kwpolska@kwpolska-lin .hg@default]% cat * */* */*/* 2>/dev/null | wc -c
1481
[kwpolska@kwpolska-lin .git@master]% cat * */* */*/* */*/*/* 2>/dev/null | wc -c
16860 ↠WRONG!

There is just one problem with this: an empty git repository starts at
15216 bytes, due to some sample hooks. Let’s remove them and try
again:

[kwpolska@kwpolska-lin .git@master]% rm hooks/*
[kwpolska@kwpolska-lin .git@master]% cat * */* */*/* */*/*/* */*/*/*
2>/dev/null | wc -c
2499

which is a much more sane number. This includes a config file (in the
ini/configparser format) and such. According to my maths skils (or
rather zsh’s skills), new commits are responsible for 1644 bytes in
the git repo and 1391 bytes in the hg repo.

(I’m using wc -c to count the bytes in all files there are. du is
unaccurate with files smaller than 4096 bytes.)

 

[Roy Smith]

(I’m using wc -c to count the bytes in all files there are. du is
unaccurate with files smaller than 4096 bytes.)

It's not that du is not accurate, it's that it's measuring something
different. It's measuring how much disk space the file is using. For
most files, that's the number of characters in the file rounded up to a
full block. For large files, I believe it also includes the overhead of
indirect blocks or extent trees. And, finally, for sparse files, it
takes into account that some logical blocks in the file may not be
mapped to any physical storage.

So, whether you want to use "du" or "wc -c" depends on what you're
trying to measure. If you want to know how much disk space you're
using, du is the right tool. If you want to know how much data will be
transmitted if the file is serialized (i.e. packed in a tarball or sent
via a "{hg,git} clone" operation), then "wc-c" is what you want.

All that being said, for the vast majority of cases (and I would be
astonished if this was not true for any real-life vcs repo), the
difference between what wc and du tell you is not worth worrying about.
And du is going to be a heck of a lot faster.

 

[Lele Gaifax]

It's not that du is not accurate, it's that it's measuring something
different. It's measuring how much disk space the file is using. For
most files, that's the number of characters in the file rounded up to a
full block.

I think “du -c†emits a number very close to “wc -câ€.


ciao, lele.

 

[Jason Swails]

On Sun, Jun 16, 2013 at 9:30 AM, Chris “Kwpolska†Warrick <

Hmm. ~/cpython/.hg is 200MB+, but ~/pike/.git is only 86MB. Does
Mercurial compress its content? A tar.gz of each comes down, but only
to ~170MB and ~75MB respectively, so I'm guessing the bulk of it is
already compressed. But 200MB for cpython seems like a lot.

Next time, do a more fair comparison.

I created an empty git and hg repository, and created a file promptly
named “file†with DIGIT ONE (0x31; UTF-8/ASCII–encoded) and commited
it with “c1†as the message, then I turned it into “12†and commited
as “c2†and did this one more time, making the file “123†at commit
named “c3â€.

[kwpolska@kwpolska-lin .hg@default]% cat * */* */*/* 2>/dev/null | wc -c
1481
[kwpolska@kwpolska-lin .git@master]% cat * */* */*/* */*/*/* 2>/dev/null
| wc -c
16860 ↠WRONG!

There is just one problem with this: an empty git repository starts at
15216 bytes, due to some sample hooks. Let’s remove them and try
again:

[kwpolska@kwpolska-lin .git@master]% rm hooks/*
[kwpolska@kwpolska-lin .git@master]% cat * */* */*/* */*/*/* */*/*/*
2>/dev/null | wc -c
2499

which is a much more sane number. This includes a config file (in the
ini/configparser format) and such. According to my maths skils (or
rather zsh’s skills), new commits are responsible for 1644 bytes in
the git repo and 1391 bytes in the hg repo.

This is not a fair comparison, either. If we want to do a fair comparison
pertinent to this discussion, let's convert the cpython mercurial
repository into a git repository and allow the git repo to repack the diffs
the way it deems fit.

I'm using the git-remote-hg.py script [
https://github.com/felipec/git/blob/fc/master/contrib/remote-helpers/git-remote-hg.py]
to clone a mercurial repo into a native git repo. Then, in one of the rare
cases, using git gc --aggressive. [1]

The result:

Git:
cpython_git/.git $ du -h --max-depth=1
40K ./hooks
145M ./objects
20K ./logs
24K ./refs
24K ./info
146M .

Mercurial:
cpython/.hg $ du -h --max-depth=1
209M ./store
20K ./cache
209M .


And to help illustrate the equivalence of the two repositories:

Git:

cpython_git $ git log | head; git log | tail

commit 78f82bde04f8b3832f3cb6725c4bd9c8d705d13b
Author: Brett Cannon <[email protected]>
Date: Sat Jun 15 23:24:11 2013 -0400

Make test_builtin work when executed directly

commit a7b16f8188a16905bbc1d49fe6fd940078dd1f3d
Merge: 346494a af14b7c
Author: Gregory P. Smith <[email protected]>
Date: Sat Jun 15 18:14:56 2013 -0700
Author: Guido van Rossum <[email protected]>
Date: Mon Sep 10 11:15:23 1990 +0000

Warning about incompleteness.

commit b5e5004ae8f54d7d5ddfa0688fc8385cafde0e63
Author: Guido van Rossum <[email protected]>
Date: Thu Aug 9 14:25:15 1990 +0000

Initial revision

Mercurial:

cpython $ hg log | head; hg log | tail

changeset: 84163:5b90da280515
bookmark: master
tag: tip
user: Brett Cannon <[email protected]>
date: Sat Jun 15 23:24:11 2013 -0400
summary: Make test_builtin work when executed directly

changeset: 84162:7dee56b6ff34
parent: 84159:5e8b377942f7
parent: 84161:7e06a99bb821
user: Guido van Rossum <[email protected]>
date: Mon Sep 10 11:15:23 1990 +0000
summary: Warning about incompleteness.

changeset: 0:3cd033e6b530
branch: legacy-trunk
user: Guido van Rossum <[email protected]>
date: Thu Aug 09 14:25:15 1990 +0000
summary: Initial revision

They both appear to have the same history. In this particular case, it
seems that git does a better job in terms of space management, probably due
to the fact that it doesn't store duplicate copies of identical source code
that appears in different files (it tracks content, not files).

That being said, from what I've read both git and mercurial have their
advantages, both in the performance arena and the features/usability arena
(I only know how to really use git). I'd certainly take a DVCS over a
centralized model any day.

All the best,
Jason

[1] I know I just posted in this thread about --aggressive being bad, but
the packing from the translation was horrible --> the translated git repo
was ~2 GB in size. An `aggressive' repacking was necessary to allow git to
decide how to pack the diffs.

 

[Terry Reedy]

I think “du -c†emits a number very close to “wc -câ€.

In Windows Explorer, the Properties box displays both the Size and 'Size
on disk', in both (KB or MB) and bytes. The block size for the disk I am
looking at is 4KB, so the Size on disk in KB is a multiple of that.