Since I had no interesting books to read today, nor interesting films to watch, I decided to scavenge for the most intriguing content one can find online. I ended up reading the Linux kernel mailing lists, but those discussions seemed to be 18+, so I settled for the comparatively civil emacs-devel.
For those unfamiliar, emacs-devel is the primary development discussion list for GNU Emacs – where design decisions get made, patches get reviewed, and occasionally where people spend 200 messages arguing about version control software. This is the story of that last one.
2008: “This question is over and decided”
In March 2008, Emacs was migrating from CVS (yes, CVS) to something more “modern”. The two contenders were Git and Bazaar.
- Git, created by Linus Torvalds for the Linux kernel.
- Bazaar was a GNU project, maintained by Canonical
A 236-message thread erupted on emacs-devel. People benchmarked both tools. The results were not subtle. Andreas Schwab, one of the core developers, reported his first impression:
My first impression is that bzr is slow, so slow that it is completely unusable. How can it come that a simple bzr log takes more than a minute to even start? Even cvs log is instantaneous in comparison, although it has to request the log from the server.
David Kastrup found it equally puzzling:
I find this surprising: “git log” is pretty much instantaneous, and git recalculates a code piece’s history in the process. In contrast, one has to tell Bazaar when one copies or moves or renames files, so it should have the information available right away.
The actual numbers:
git log | head -10.012 seconds. The same command with Bazaar took 21.5 seconds.- Committing a single-file change: 0.08 seconds with Git, 17 seconds with Bazaar.
The benchmarks kept coming. Stefan Monnier, the head maintainer, set the bar low:
I don’t care if Bzr is slower or faster than Git, but in order to switch to Bzr, we need it to be ‘fast enough’. Currently it is not. At the very least the ‘bzr diff’ should not take more than a couple seconds.
Meanwhile, Jonathan Lange, an actual Bazaar developer from Canonical, was in the thread doing tech support.
His recommended workflow for the initial checkout:
An even better way to do the initial download is this:
$ wget http://bzr.notengoamigos.org/emacs.tar.gz
$ tar xzf emacs.tar.gz
$ bzr init-repo emacs-bzr
$ cd emacs-bzr
$ bzr branch ../emacs trunk
$ cd trunk
$ bzr pull –remember http://bzr.notengoamigos.org/emacs/trunk/
Compare that to git clone!
Someone in the thread finally asked the obvious question:
To the emacs maintainers and decision makers: What more information is required to convince bzr is not the right tool at the present moment?
Richard Stallman’s reply:
This decision is not a decision for the present moment. It is a long term decision. So it would be better to wait a few months while Bzr developers improve it, than to make some other “temporary” decision that would probably be hard to reverse.
And in case anyone missed the point, in a separate message:
This question is over and decided. We will use GNU Bzr, because it is a GNU package.
When someone pointed out that this political decision was “wiping away all technical arguments,” RMS replied:
The rule that GNU packages should support each other helps make the GNU system as a whole work better.
Someone asked the obvious follow-up: “Why can’t we just make Git part of the GNU system?”
RMS:
We could include it in the GNU system, but its developers are not likely to want to make it a GNU package.
To be fair to RMS, there’s a real principle here. If the GNU project doesn’t use its own tools, it sends a message that those tools aren’t good enough, which undermines the whole idea of a self-sufficient free software ecosystem. He’d been making this argument for decades, and it had served the project well in many other cases. The problem wasn’t the principle. The problem was that Bazaar couldn’t live up to it.
The 236-message thread, the benchmarks, the Canonical employee’s workarounds; none of it changed the outcome. The decision was political, and the politics were settled.
2008-2012: The long tail
The rest of the world moved to Git. GitHub launched in 2008 and exploded. Emacs contributors, meanwhile, had to learn Bazaar, a tool they used nowhere else, just to submit patches.
Threads like “Help me unstick my bzr, please” and “Can NOT bzr the emacs repos (may be bzr has a memory leak)” became regular occurrences.
Then in 2012, Canonical laid off the Bazaar development team.
2013
In March 2013, a year after Bazaar’s development ceased, John Wiegley posted what everyone had been wanting to say:
We have often debated the merits of Git vs. Bazaar, and which one the GNU project should use for Emacs development. I think now is an appropriate time to revisit this decision.
My main reason for bringing this up again is that Bazaar development has effectively stalled. There are major bugs which have been in their bug-tracker for years now – bugs affecting Emacs development, such as the ELPA repository – which have been ignored all this time.
So, to Richard as the undisputed Czar of all things Emacs: can we now, pretty please, switch to Git?
200 messages followed.
RMS’s first response:
The maintainer says he is fixing some bugs, and I asked him just yesterday to fix the ELPA branch bug. I’d like to give him a reasonable time to do this.
The bug was 1.5 years old. Dmitry Gutov called it out:
Isn’t this a bit late? The bug is 1.5 years old. Was he not aware of it before?
Then RMS revealed his hand:
I am trying to determine whether Bzr is effectively maintained or not. I’d rather get a Yes answer than a No answer.
That’s a remarkably honest thing to say publicly. He wasn’t hiding his preference. He genuinely believed in the principle of GNU projects supporting each other, and he was hoping reality would cooperate.
Joakim Verona, a longtime Bazaar user and Emacs contributor, described the reality on the ground:
I have done my best to be a constructive user of the tool, and I have had many technical difficulties. When I try to find solutions to the issues I notice the following:
- The bzr community is very helpful. This is good.
- There are many well known bugs. There are also many well known patches for these, some of them provided by Emacs developers. They never enter upstream. By “never” I mean years. This is bad.
I don’t have time to read the Bzr mailing list. Or any development mailing list. The only such list I am on is this one. You might as well tell me to fly to the moon as tell me to read something on the Bzr list.
And when asked whether the users of Bazaar should have a say in whether Bazaar is sufficiently maintained:
When I have to decide whether a maintainer is doing an adequate job or needs to be replaced, I pay attention to whatever relevant information I get. However, to give users “a say” in the decision seems improper, so I don’t do that.
Karl Fogel, a veteran open source developer (author of Producing Open Source Software and one of the original Subversion developers), delivered the sharpest critique in the thread:
Well, really, you don’t have time to pay close enough attention to Bzr development to competently decide whether it’s still a good choice for Emacs. That’s fine – no one has time to do every important thing, and you do many other important things.
But then why do you think you still have the time & mental bandwidth to make this decision well? Why not delegate it to the Emacs maintainers on the grounds that you no longer have time to do a good job of this evaluation?
He pointed out that asking one person about one bug is not a proxy for project health, and that others in the thread had already done more thorough research than RMS could, given his time constraints.
RMS’s reply:
Because more than Emacs is at stake here.
One line, but it’s the core of his worldview. If the flagship GNU project abandons a GNU tool, what signal does that send to every other GNU package? He wasn’t wrong about the stakes. He was wrong about the tool.
Karl pushed once more:
You should either devote enough time to evaluating Bzr’s maintenance state to get a reliable answer, or delegate to someone who can do so. Instead, you’re asking the maintainers to rely on your investigation… yet you clearly don’t have time to do a good job. This is a poor use of everyone’s time.
RMS:
I already have a plan for how to proceed on this, and I am doing it.
No details. No timeline. No delegation. Trust me.
Meanwhile, Stefan Monnier, posted exactly one substantive message in the entire 200-message thread:
Just like I didn’t fight Richard’s choice of Bazaar, I don’t care very much whether we keep using Bazaar or we change to Git, Monotone, Darcs, Mercurial, OpenCM, Fossil, younameit.
The only thing I care for now is to move away from Bazaar for the ’elpa’ branch because Bazaar can’t handle it properly.
Leo Liu summarized what everyone was thinking:
Most GNU projects aren’t using BZR as you might be aware.
While helping BZR fixing bugs might be a gain for BZR, it is a loss as a whole for GNU. Volunteers spend their spare time on GNU projects and if 20% of that time is taken up by wrestling with BZR, it becomes costly to the point discouraging people from joining.
For the greater good of GNU, move off BZR seems like the only sound choice.
The thread ended without a clear resolution. RMS had a plan. He was working on it. The 200 messages didn’t produce a decision, but they did make the community’s position unmistakable.
2013: The ELPA branch breaks
Later that year, Stefan made the practical move that set the stage for everything that followed. The ELPA branch was broken on Bazaar, a bug that crashed on checkout, with no one left to fix it. Stefan moved it to Git, and his announcement showed exactly the kind of careful leadership that had kept Emacs development running through all of this:
I’m not terribly happy about this change, since it means we’ll be using two different tools (Git for ’elpa’ and Bzr for ’trunk’), but I really see no other way out.
I don’t want this to be a discussion about the merits/pitfalls of Git vs Bzr, and this is not an occasion to discuss the use of Git for the ’trunk’ either.
He knew exactly what everyone was thinking “if Git is good enough for ELPA, why not for trunk?” and he headed it off. One problem at a time.
2014: ESR pushes the button
By August 2014, Eric S. Raymond had the conversion scripts ready.
He’d been working on it quietly:
You haven’t heard much about it because the hard work is all done. I have the scripts ready to go and need only about eight hours’ notice before pushing the button.
The actual migration happened in November 2014. On November 13th, ESR posted a six-word message:
Commits are open. Have at it.
Which some even described as heroic.
Six years of debate, 236 messages in the 2008 thread, 200 messages in the 2013 thread, Stefan’s years of quiet maintenance, countless “help me unstick my bzr” pleas, one dead version control system, and it ended with six words.
The aftermath
The days following the migration were educational. Half the core contributors had never used Git:
- “This Is The Git Help Mailing List”
- “git pull fails with merge conflicts. How can this possibly happen?”
- “A simple git workflow for the rest of us”
- “need help adjusting workflow to git”
- “Good book on Git”
- “Obscure error/warning/information message from git pull” (124 messages)
These were people who’d been developing one of the most important text editors in the world for years, asking basic Git questions, because they’d been stuck on Bazaar while the rest of the world moved on.
What a bzr saga! Anyway, better than any film I could have watched tonight.