Jelmer Vernooij - ubuntuhttps://www.jelmer.uk/2021-09-06T22:00:00+02:00Web Hooks for the Janitor2021-09-06T22:00:00+02:002021-09-06T22:00:00+02:00Jelmer Vernooijtag:www.jelmer.uk,2021-09-06:janitor-webhooks.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>As covered in <a class="reference external" href="https://www.jelmer.uk/fresh-builds.html">my post from last week</a>, the Janitor now regularly tries to
import new upstream git snapshots or upstream releases into packages in <a class="reference external" href="https://wiki.debian.org/DebianUnstable">Sid</a>.</p>
<div class="section" id="moving-parts">
<h2>Moving parts</h2>
<p>There are about 30,000 packages in sid, and it usually takes a couple of weeks
for the janitor to cycle through all of them. Generally speaking, there are up
to three moving targets for each package:</p>
<ul class="simple">
<li>The packaging repository; <a class="reference external" href="https://qa.debian.org/cgi-bin/vcswatch">vcswatch</a> regularly scans this for changes,
and notifies the janitor when a repository has changed. For <a class="reference external" href="https://salsa.debian.org">salsa</a>
repositories it is instantly notified through a web hook</li>
<li>The upstream release tarballs; the <span class="caps">QA</span> watch service regularly polls these,
and the janitor scans for changes in the <span class="caps">UDD</span> tables with watch data (used for
<a class="reference external" href="https://janitor.debian.net/fresh-releases/">fresh-releases</a>)</li>
<li>The upstream repository; there is no service in Debian that watches this at
the moment (used for
<a class="reference external" href="https://janitor.debian.net/fresh-snapshots/">fresh-snapshots</a>)</li>
</ul>
<p>When the janitor notices that one of these three targets has changed, it
prioritizes processing of a package - this means that a push to a packaging
repository on salsa usually leads to a build being kicked off within 10
minutes. New upstream releases are usually noticed by <span class="caps">QA</span> watch within a day or
so and then lead to a build. Now commits in upstream repositories don’t get
noticed today.</p>
<p>Note that there are no guarantees; the scheduler tries to be clever and not
e.g. rebuild the same package over and over again if it’s constantly changing
and takes a long time to build.</p>
<p>Packages without priority are processed with a scoring system that takes into
account perceived value (based on e.g. popcon), cost (based on wall-time
duration of previous builds) and likelihood of success (whether recent builds
were successful, and how frequently the repositories involved change).</p>
</div>
<div class="section" id="webhooks-for-upstream-repositories">
<h2>webhooks for upstream repositories</h2>
<p>At the moment there is no service in Debian (yet - perhaps this is something
that vcswatch or a sibling service could also do?) that scans upstream
repositories for changes.</p>
<p>However, if you maintain an upstream package, you can use a webhook to notify
the janitor that commits have been made to your repository, and it will create
a new package in fresh-snapshots. Webhooks from the
following hosting site software are currently supported:</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/">GitHub</a></li>
<li><a class="reference external" href="https://gitlab.com/">Gitlab</a></li>
<li><a class="reference external" href="https://launchpad.net/">Launchpad</a></li>
<li><a class="reference external" href="https://gitea.io/">Gitea</a></li>
<li><a class="reference external" href="https://gogs.io/">Gogs</a></li>
</ul>
<p>You can simply use the <span class="caps">URL</span> <a class="reference external" href="https://janitor.debian.net/">https://janitor.debian.net/</a> as the target for hooks. There is no need to specify a secret, and the hook can either use a <span class="caps">JSON</span> or form encoding payload.</p>
<p>The endpoint should tell you whether it understood a webhook request, and
whether it took any action. It’s fine to submit webhooks for repositories that
the janitor does not (yet) know about.</p>
<div class="section" id="github-1">
<h3>GitHub</h3>
<p>For GitHub, you can do so in the <tt class="docutils literal">Webhooks</tt> section of the <tt class="docutils literal">Settings</tt> tab. Fill the form as shown below and click on <tt class="docutils literal">Add webhook</tt>:</p>
<img alt="" src="/images/github-webhook.png" />
</div>
<div class="section" id="gitlab-1">
<h3>GitLab</h3>
<p>On GitLab instances, you can find the <tt class="docutils literal">Webhooks</tt> tab under the <tt class="docutils literal">Settings</tt> menu for each repository (under the gear symbol). Fill the form in as shown below and click <tt class="docutils literal">Add Webhook</tt>:</p>
<img alt="" src="/images/gitlab-webhook.png" />
</div>
<div class="section" id="launchpad-1">
<h3>Launchpad</h3>
<p>For Launchpad, go to the repository (for Git) web view and click <tt class="docutils literal">Manage Webhooks</tt>. From there, you can add a new webhook; fill the form in as shown below and click <tt class="docutils literal">Add Webhook</tt>:</p>
<img alt="" src="/images/launchpad-webhook-git.png" />
</div>
</div>
Thousands of Debian packages updated from their upstream Git repository2021-08-25T20:00:00+02:002021-08-25T20:00:00+02:00Jelmer Vernooijtag:www.jelmer.uk,2021-08-25:fresh-builds.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>Linux distributions like Debian fulfill an important function in the <span class="caps">FOSS</span> ecosystem - they are system integrators that take existing free and open source software projects and adapt them where necessary to work well together. They also make it possible for users to install more software in an easy and consistent way and with some degree of quality control and review.</p>
<p>One of the consequences of this model is that the distribution package often lags behind upstream releases. This is especially true for distributions that have tighter integration and standardization (such as Debian), and often new upstream code is only imported irregularly because it is a manual process - both updating the package, but also making sure that it still works together well with the rest of the system.</p>
<p>The process of importing a new upstream used to be (well, back when I started working on
Debian packages) fairly manual and something like this:</p>
<ul class="simple">
<li>Go to the upstream’s homepage, find the tarball and signature and verify the tarball</li>
<li>Make modifications so the tarball matches Debian’s format</li>
<li>Diff the original and new upstream tarballs and figure out whether changes
are reasonable and which require packaging changes</li>
<li>Update the packaging, changelog, build and manually test the package</li>
<li>Upload</li>
</ul>
<div class="section" id="ecosystem-improvements">
<h2>Ecosystem Improvements</h2>
<p>However, there have been developments over the last decade that make it easier to import new upstream releases into Debian packages.</p>
<div class="section" id="uscan-and-debian-qa-watch">
<h3>Uscan and debian <span class="caps">QA</span> watch</h3>
<p><a class="reference external" href="https://manpages.debian.org/buster/devscripts/uscan.1.en.html">Uscan</a> and
<a class="reference external" href="https://wiki.debian.org/debian/watch">debian/watch</a> have been around for a
while and make it possible to find upstream tarballs.</p>
<p>A debian watch file usually looks something like this:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="go">version=4</span>
<span class="go">http://somesite.com/dir/filenamewithversion.tar.gz</span>
</pre></div></td></tr></table></div>
<p>The <a class="reference external" href="https://qa.debian.org/cgi-bin/watch"><span class="caps">QA</span> watch service</a> regularly polls
all watch locations in the archive and makes the information available, so it’s
possible to know which packages have changed without downloading each one of them.</p>
</div>
<div class="section" id="git">
<h3>Git</h3>
<p>Git is fairly ubiquitous nowadays, and most upstream projects and packages in Debian use it. There are still exceptions that do not use any version control system or that use a different control system, but they are becoming increasingly rare. <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a></p>
<a class="reference external image-reference" href="https://trends.debian.net/#version-control-system"><img alt="" src="https://trends.debian.net/vcs_testing-stacked.png" /></a>
</div>
<div class="section" id="debian-upstream-metadata">
<h3>debian/upstream/metadata</h3>
<p><a class="reference external" href="https://dep-team.pages.debian.net/deps/dep12/"><span class="caps">DEP</span>-12</a> specifies a file format with metadata about the upstream project that a package was based on. In particular relevant for our case is the fact it has fields for the location of the upstream version control location.</p>
<p>debian/upstream/metadata files look something like this:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span></pre></div></td><td class="code"><div><pre><span></span><span class="nn">---</span>
<span class="nt">Repository</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://www.dulwich.io/code/dulwich/</span>
<span class="nt">Repository-Browse</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://www.dulwich.io/code/dulwich/</span>
</pre></div></td></tr></table></div>
<p>While <span class="caps">DEP</span>-12 is still a draft, it has already been widely adopted - there are <a class="reference external" href="https://codesearch.debian.net/search?q=path%3Adebian%2Fupstream%2Fmetadata+Repository%3A&literal=1">about 10000 packages</a> in Debian that ship a debian/upstream/metadata file with Repository information.</p>
</div>
<div class="section" id="autopkgtest">
<h3>Autopkgtest</h3>
<p>The <a class="reference external" href="https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst">Autopkgtest</a>
standard and associated tooling provide a way to run a defined set of tests
against an installed package. This makes it possible to verify that a package
is working correctly as part of the system as a whole. <a class="reference external" href="https://ci.debian.net">ci.debian.net</a> regularly runs these tests against Debian packages to
detect regressions.</p>
</div>
<div class="section" id="vcs-git-headers">
<h3>Vcs-Git headers</h3>
<p>The Vcs-Git headers in debian/control are the equivalent of the Repository field in debian/upstream/metadata, but for the packaging repositories (as opposed to the upstream ones).</p>
<p>They’ve been around for a while and are widely adopted, as can be seen from <a class="reference external" href="https://upsilon.cc/~zack/stuff/vcs-usage/">zack’s stats</a>:</p>
<img alt="" src="/images/fresh-builds-vcs.png" />
<p>The <a class="reference external" href="https://qa.debian.org/cgi-bin/vcswatch">vcswatch service</a> that regularly
polls packaging repositories to see whether they have changed makes it a lot
easier to consume this information in usable way.</p>
</div>
<div class="section" id="debhelper-adoption">
<h3>Debhelper adoption</h3>
<p>Over the last couple of years, Debian has slowly been converging on a single
build tool - debhelper’s dh interface.</p>
<p>Being able to rely on a single build tool makes it easier to write code to
update packaging when upstream changes require it.</p>
<a class="reference external image-reference" href="https://trends.debian.net/#build-systems"><img alt="" src="https://trends.debian.net/build-system_testing-stacked.png" /></a>
</div>
<div class="section" id="debhelper-dwim">
<h3>Debhelper <span class="caps">DWIM</span></h3>
<p>Debhelper (and its helpers) increasingly can figure out how to do the Right
Thing in many cases without being explicitly configured. This makes packaging
less effort, but also means that it’s less likely that importing a new upstream
version will require updates to the packaging.</p>
<p>With all of these improvements in place, it actually becomes feasible in a lot
of situations to update a Debian package to a new upstream version
automatically. Of course, this requires that all of this information is
available, so it won’t work for all packages. In some cases, the packaging for
the older upstream version might not apply to the newer upstream version.</p>
<p>The Janitor has attempted to import a new upstream Git snapshot and a new
upstream release for every package in the archive where a debian/watch file or
debian/upstream/metadata file are present.</p>
<p>These are the steps it uses:</p>
<ul class="simple">
<li><dl class="first docutils">
<dt>Find new upstream version</dt>
<dd><ul class="first last">
<li>If release, use debian/watch - or maybe tagged in upstream repository</li>
<li>If snapshot, use debian/upstream/metadata’s Repository field</li>
<li><em>If neither is available, use guess-upstream-metadata from upstream-ontologist to guess the upstream Repository</em></li>
</ul>
</dd>
</dl>
</li>
<li>Merge upstream version into packaging repository, possibly importing tarballs using pristine-tar</li>
<li>Update the changelog file to mention the new upstream version</li>
<li><dl class="first docutils">
<dt>Run some checks to ensure there are no unintentional changes, e.g.:</dt>
<dd><ul class="first last">
<li><dl class="first docutils">
<dt>Scan diff between old and new for surprising license changes</dt>
<dd><ul class="first last">
<li>Today, abort if there are any - in the future, maybe update debian/copyright</li>
</ul>
</dd>
</dl>
</li>
<li>Check for obvious compatibility breaks - e.g. sonames changing</li>
</ul>
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt>Attempt to update the packaging to reflect upstream changes</dt>
<dd><ul class="first last">
<li>Refresh patches</li>
</ul>
</dd>
</dl>
</li>
<li>Attempt to build the package with deb-fix-build, to deal with any missing dependencies</li>
<li>Run the autopkgtests with deb-fix-build to deal with missing dependencies, and abort if any tests fail</li>
</ul>
</div>
</div>
<div class="section" id="results">
<h2>Results</h2>
<p>When run over all packages in unstable (<a class="reference external" href="https://packages.debian.org/unstable">sid</a>), this process works for a surprising number of them.</p>
<div class="section" id="fresh-releases">
<h3>Fresh Releases</h3>
<p>For fresh-releases (aka imports of upstream releases), processing all packages maintained in Git for which <span class="caps">QA</span> watch reports new releases (about 11,000):</p>
<img alt="" src="/images/fresh-releases.png" />
<p>That means about 2300 packages updated, and about 4000 unchanged.</p>
</div>
<div class="section" id="fresh-snapshots">
<h3>Fresh Snapshots</h3>
<p>For fresh-snapshots (aka imports of latest Git commit from upstream), processing all packages maintained in Git (about 26,000):</p>
<img alt="" src="/images/fresh-snapshots.png" />
<p>Or 5100 packages updated and 2100 for which there was nothing to do, i.e. no upstream commits since the last Debian upload.</p>
<p>As can be seen, this works for a surprising fraction of packages. It’s possible to get the numbers up even higher, by both improving the tooling, the autopkgtests and the metadata that is provided by packages.</p>
</div>
</div>
<div class="section" id="using-these-packages">
<h2>Using these packages</h2>
<p>All the packages that have been built can be accessed from the Janitor <span class="caps">APT</span> repository. More information can be found at <a class="reference external" href="https://janitor.debian.net/fresh">https://janitor.debian.net/fresh</a>, but in short - run:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="go">echo deb "[arch=amd64 signed-by=/usr/share/keyrings/debian-janitor-archive-keyring.gpg]" \</span>
<span class="go"> https://janitor.debian.net/ fresh-snapshots main | sudo tee /etc/apt/sources.list.d/fresh-snapshots.list</span>
<span class="go">echo deb "[arch=amd64 signed-by=/usr/share/keyrings/debian-janitor-archive-keyring.gpg]" \</span>
<span class="go"> https://janitor.debian.net/ fresh-releases main | sudo tee /etc/apt/sources.list.d/fresh-releases.list</span>
<span class="go">sudo curl -o /usr/share/keyrings/debian-janitor-archive-keyring.gpg https://janitor.debian.net/pgp_keys</span>
<span class="go">apt update</span>
</pre></div></td></tr></table></div>
<p>And then you can install packages from the fresh-snapshots (upstream git snapshots) or fresh-releases suites on a case-by-case basis by running something like:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="go">apt install -t fresh-snapshots r-cran-roxygen2</span>
</pre></div></td></tr></table></div>
<p>Most packages are updated based on information provided by vcswatch and qa watch, but it’s also possible for upstream repositories to call a web hook to trigger <a class="reference external" href="https://janitor.debian.net/fresh-builds#requesting-new-packages">a refresh of a package</a>.</p>
<p>These packages were built against unstable, but should in almost all cases also work for testing.</p>
</div>
<div class="section" id="caveats">
<h2>Caveats</h2>
<p>Of course, since these packages are built automatically without human supervision it’s likely that some of them will have bugs in them that would otherwise have been caught by the maintainer.</p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>I’m not saying that a monoculture is great here, but it does help distributions.</td></tr>
</tbody>
</table>
</div>
Migrating packaging from Bazaar to Git2013-06-02T02:01:00+02:002013-06-02T02:01:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2013-06-02:pristine-tar-migrate-bzr-to-git.html<p>A while ago I migrated most of my packages from Bazaar to Git. The rest of the
world has decided to use Git for version control, and I don’t have enough
reason to stubbornly stick with Bazaar and make it harder for myself to
collaborate with others.</p>
<p>So I’m moving away from a workflow I know and have polished over the last few
years - including the various bzr plugins and other tools involved. Trying to
do the same thing using git is frustrating and time-consuming, but I’m sure
that will improve with time. In particular, I haven’t found a good way to merge
in a new upstream release (from a tarball) while referencing the relevant
upstream commits, like <tt class="docutils literal">bzr <span class="pre">merge-upstream</span></tt> can. Is there a good way to do this?
What helper tools can you recommend for maintaining a Debian package in git?</p>
<p>Having been upstream for <tt class="docutils literal"><span class="pre">bzr-git</span></tt> earlier, I used its <tt class="docutils literal"><span class="pre">git-remote-bzr</span></tt>
implementation to do the conversions of the commits and tags:</p>
<pre class="literal-block">
% git clone bzr::/path/to/bzr/foo.bzr /path/to/git/foo.git
</pre>
<p>One of my last contributions to bzr-git was a <tt class="docutils literal">bzr <span class="pre">git-push-pristine-tar-deltas</span></tt>
subcommand, which will export all bzr-builddeb-style pristine-tar metadata
to a pristine-tar branch in a Git repository that can be used by
<em>pristine-tar</em> directly or through something like <em>git-buildpackage</em>.</p>
<p>Once you have created a git clone of your bzr branch, it should be a matter of
running <tt class="docutils literal">bzr <span class="pre">git-push-pristine-tar-deltas</span></tt> with the target git repository
and the Debian package name:</p>
<pre class="literal-block">
% cd /path/to/bzr/foo.bzr
% bzr git-push-pristine-tar-deltas /path/to/git/foo.git foo
% cd /path/to/git/foo.git foo
% git branch
* master
pristine-tar
</pre>
Bazaar: A retrospective2012-12-19T22:36:43+01:002012-12-19T22:36:43+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2012-12-19:bzr-retrospective.html<p>For the last 7 years I’ve been involved in the <a class="reference external" href="http://bazaar-vcs.org/">Bazaar</a> project. Since
I am slowly stepping down, I recently wrote <a class="reference external" href="/pages/bzr-a-retrospective.html">a retrospective</a> on the
project as I experienced it for the last 7 years.</p>
<p>Thanks to a few kind people for proofreading earlier drafts; if you spot any
errors, please let me know in the comments.</p>
Samba 4.0.0, finally2012-12-11T18:00:00+01:002012-12-11T18:00:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2012-12-11:samba-4.0.0.html<p>This afternoon <a class="reference external" href="https://www.samba.org/samba/news/releases/4.0.0.html">we released version 4.0.0</a> of Samba. This is a significant
milestone, and I’m very proud of the result. Samba 4 is the first version
that can be a domain controller in an Active Directory domain.</p>
<p>We embarked on this journey almost a decade ago - <a class="reference external" href="https://git.samba.org/?p=samba.git;a=commit;h=b0510b5428b3461aeb9bbe3cc95f62fc73e2b97f">the first commit</a> is from
August 2003. It’s been a long and bumpy ride. I hardly recognize
the people in this <a class="reference external" href="https://www.samba.org/samba/images/team_l2003.jpg">team photo from 2003</a> (I’m second from the left).</p>
<p>A lot has happened in that time. We wrote <a class="reference external" href="https://www.ohloh.net/p/samba/contributors/summary">a few million lines of code</a>. We migrated from <span class="caps">CVS</span> to Subversion to Git. We’ve drifted apart and grown back together as a <a class="reference external" href="https://www.samba.org/samba/team/">team</a>.</p>
<p>In my youthful naivity I predicted a release “within 1 or 2
years” during <a class="reference external" href="http://www.samba.org/~jelmer/jelmer-nluug-vj04.pdf">a talk at the <span class="caps">NLUUG</span></a> in 2004. But Active Directory was a lot
harder than we thought, and there were quite a few other distractions as well.
I’m glad this release, which is by far the biggest and longest running software
project I have ever worked on, has finally happened.</p>
<p>Some older RCs of Samba 4 have already been packaged for Debian and Ubuntu,
in the <tt class="docutils literal">samba4</tt> source package. For Debian jessie, these will be integrated
into the main <tt class="docutils literal">samba</tt> source package. Please use <tt class="docutils literal">experimental</tt> if you do
want to try the existing packages, as it is most up to date.</p>
Last day at Canonical2012-10-17T00:00:00+02:002012-10-17T00:00:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2012-10-17:last-day-at-canonical.html<p>This Friday will be my last day at <a class="reference external" href="http://www.canonical.com/">Canonical</a>.</p>
<p>It has been a lot of fun working here. There is so much I have learned in
the last three years. I’m going to miss my colleagues.</p>
<p>Over the last couple of months I have slowly stepped down from my involvement
in Bazaar and the Bazaar packaging in Debian and Ubuntu.
I would like to stay involved in Ubuntu, but we will see how that goes.</p>
<p>I’m taking some time off until the end of the year to see the world and
hack, before starting something new in February.</p>
Summer of Code 20112011-07-18T08:53:05+02:002011-07-18T08:53:05+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2011-07-18:272-Summer-of-Code-2011.html<p>The Samba team is once again participating in the Summer of Code this year.
This year we have 4 students working on various projects related to Samba.</p>
<p>This year I am mentoring Dhananjay Sathe, who is improving the
<a class="reference external" href="http://wiki.samba.org/index.php/SambaGtk"><span class="caps">GTK</span>+ frontends for Samba</a>. In
particular, he is making it possible to manage shares and users of a remote
Samba or Windows machine.</p>
<p>Dhananjay is also <a class="reference external" href="http://dsathe.blogspot.com/2011/07/samba-with-soc-google-summer-of-code.html">blogging about his progress</a>.</p>
libapache2-mod-bzr2011-01-02T19:07:53+01:002011-01-02T19:07:53+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2011-01-02:269-libapache2-mod-bzr.html<p>During the last two days I hacked together a <a class="reference external" href="http://bazaar.canonical.com/">Bazaar</a> module for <a class="reference external" href="http://www.apache.org/">Apache</a>.
This module makes it possible to easily enable the Bazaar smart server for
Bazaar branches. It also can display a simple placeholder page for Bazaar
branches without working tree. It’s surprisingly easy to write Apache modules.</p>
<p>The main advantage this has over a mod_wsgi / mod_python / mod_fcgi setup is that it doesn’t require any additional Python hacking on the users side or other configuration outside of Apache, and it doesn’t require configuration for each single branch in the Apache configuration. In the future I’d also like to support the settings “BazaarFrontend <a class="reference external" href="https://launchpad.net/wikkid">Wikkid</a>” and “BazaarFrontend <a class="reference external" href="https://launchpad.net/loggerhead">Loggerhead</a>“.</p>
<p>The configuration is currently as simple as:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span></pre></div></td><td class="code"><div><pre><span></span><span class="nb">LoadModule</span><span class="w"> </span>bzr_module<span class="w"> </span><span class="sx">/usr/lib/apache2/modules/mod_bzr.so</span>
<span class="nb">BazaarSmart</span><span class="w"> </span><span class="k">on</span>
<span class="nb">BazaarFrontend</span><span class="w"> </span>Basic
</pre></div></td></tr></table></div>
<p>in your <strong>apache2.conf</strong>. The <strong>BazaarSmart</strong> and <strong>BazaarFrontend</strong> directives can appear in <Directory> or <Location> clauses as well, if you’d like to have different behaviour for different directories.</p>
<p>At the moment this project is a proof of concept, and probably not something
you would want to run in production. For example, there is no way to limit the
access to a branch to read only. I need to double-check there are no threading issues.</p>
<p>Testing and patches are welcome. The project is hosted here:</p>
<ul class="simple">
<li><a class="reference external" href="https://launchpad.net/apache-bzr">https://launchpad.net/apache-bzr</a></li>
</ul>
<p><em>Currently Playing: Stream of Passion - Calliopeia</em></p>
On the way to Samba 4: Part 22011-01-02T16:30:13+01:002011-01-02T16:30:13+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2011-01-02:268-On-the-way-to-Samba-4-Part-2.html<p>It’s been more than a month since the last status update on my Samba 4 work -
much more than the two weeks I promised.</p>
<p>During the holidays I finally managed to release the <a class="reference external" href="http://wiki.samba.org/index.php/Samba4/Releases/4.0.0alpha14">new alpha of Samba 4</a>,
as well as releases of some of our companion libraries (<a class="reference external" href="http://tdb.samba.org/">tdb</a>, <a class="reference external" href="http://talloc.samba.org/">talloc</a>,
tevent and <a class="reference external" href="http://ldb.samba.org/">ldb</a>). The release includes a significant amount of bug fixes and
a lot of work towards a properly functioning Active Directory <span class="caps">DC</span>, too much to
list here.</p>
<p>This release I’ve mainly been involved in improving our Python bindings and our
handling of internal and external libraries. We now use symbol versioning for
our copy of Heimdal Kerberos as well as some of our other libraries. Hopefully
this will fix some of the issues users of the evolution-mapi plugin have been
seeing where they end up with both <span class="caps">MIT</span> Kerberos and Heimdal Kerberos loaded
into the same process (with all the consequences of overlapping symbol names).
Samba 4 now also has the ability to work with the system Heimdal rather than
using the bundled copy. I have packaged alpha14 for Debian and Ubuntu (fixing
most of the open bugs against the Samba 4 package in the <span class="caps">BTS</span>), but am currently
waiting for the new release of ldb to pass through <span class="caps">NEW</span> before I can upload.</p>
<p>The next release is scheduled for the first week of February.</p>
<p><em>Currently Playing: Stream of Passion - Haunted</em></p>
On the way to Samba 4: Part 12010-11-24T20:44:00+01:002010-11-24T20:44:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-11-24:267-On-the-way-to-Samba-4.html<p>After <a class="reference external" href="http://www.sambaxp.org/">Samba <span class="caps">XP</span> 2008</a> <a class="reference external" href="http://samba.org/~abartlet/">Andrew</a> and I started keeping a wiki page with our bi-weekly goals and achievements for Samba 4. Because planning in a Free Software project is hard (time availability and priorities change over time, and other volunteers are equally unpredictable) we called this our <a class="reference external" href="http://wiki.samba.org/index.php/Samba4/Andrew_and_Jelmers_Fantasy_Page">“Fantasy Page”</a>; it listed things we wanted to work on next (“fantasies”), but reality being what it is we would usually actually end up working on something entirely different. We discussed our progress and new plans in - what I would now call - a bi-weekly standup call.</p>
<p>There were several reasons for doing this. It gave us some sense of direction as well as a sense of accomplishment; a way to look back at the end of the year and realize how much we had actually achieved. Because Samba 4 is such a long term project (it is 7 years old at this point) it is easy to become disillusioned, to look back at a year of commits and to not see the gradual improvement, just the fact that there is no release yet.</p>
<p>We managed to keep this up for <a class="reference external" href="http://wiki.samba.org/index.php/Samba4/Andrew_and_Jelmers_Fantasy_Page/2008">two</a> <a class="reference external" href="http://wiki.samba.org/index.php/Samba4/Andrew_and_Jelmers_Fantasy_Page/2009">years</a>, much longer than I had anticipated, and eventually started to slip last year.</p>
<p>More recently <a class="reference external" href="http://www.kblin.org">Kai</a> and <a class="reference external" href="http://blog.tridgell.net/">Tridge</a> have started to blog weekly about their efforts to make Samba 4.0 a reality and I’m going to join them by trying to blog regularly - every two weeks - about my contributions, even if there were none.</p>
<p>In the next two weeks I plan to work on finally getting alpha 14 of Samba 4 out and on fixing the daily builds of Samba 4 and OpenChange for Ubuntu on Launchpad after we did a massive reorganization of the private libraries in Samba 4.</p>
<p><em>Current Playing: Zero 7 - Somersault</em></p>
Mumble and bluetooth2010-11-07T21:38:35+01:002010-11-07T21:38:35+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-11-07:266-Mumble-and-bluetooth.html<p><a class="reference external" href="http://mumble.sourceforge.net/">Mumble</a> is an open source, low-latency, high quality voice chat application that we’re using at Canonical, and which Samba has recently also adopted.</p>
<p>After I busted the cable of my cabled headset and inspired by <a class="reference external" href="http://mdzlog.alcor.net/2010/05/26/using-mumble-with-a-bluetooth-headset/">Matt’s post about Mumble and Bluetooth</a> I bought a cheap Nokia Bluetooth headset that works with my laptop as well as my phone. By using <a class="reference external" href="http://blueman-project.org/">BlueMan</a> I even found that the headset worked out of the box on Maverick.</p>
<p>A nice feature in Mumble is that it is controllable using D-Bus. Blueman supports running an arbitrary command when the answer button is pressed. Combining these two features, it is possible to automatically toggle mute when the answer button is pressed:</p>
<pre class="code python literal-block">
<span class="ch">#!/usr/bin/python</span><span class="w">
</span><span class="kn">import</span> <span class="nn">dbus</span><span class="w">
</span><span class="n">bus</span> <span class="o">=</span> <span class="n">dbus</span><span class="o">.</span><span class="n">SessionBus</span><span class="p">()</span><span class="w">
</span><span class="n">mumble</span> <span class="o">=</span> <span class="n">bus</span><span class="o">.</span><span class="n">get_object</span><span class="p">(</span><span class="s2">"net.sourceforge.mumble.mumble"</span><span class="p">,</span> <span class="s2">"/"</span><span class="p">)</span><span class="w">
</span><span class="n">is_muted</span> <span class="o">=</span> <span class="n">mumble</span><span class="o">.</span><span class="n">isSelfMuted</span><span class="p">()</span><span class="w">
</span><span class="n">mumble</span><span class="o">.</span><span class="n">setSelfMuted</span><span class="p">(</span><span class="ow">not</span> <span class="n">is_muted</span><span class="p">)</span>
</pre>
<p>To use this script, set its path in the configuration tab for the “Headset” plugin in blueman.</p>
<p>The only remaining problem with this setup is that I can’t keep the headset on during my work day, as I don’t have a way to put it in standby mode automatically. This means that my battery runs out pretty quickly, even when nothing is happening on Mumble.</p>
<p><em>Currently Playing: Red Sparowes - Finally As That Blazing Sun Shone</em></p>
OpenChange server and SOGo2010-10-26T17:47:16+02:002010-10-26T17:47:16+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-10-26:264-OpenChange-server-and-SOGo.html<p>There’s more good news on the OpenChange front. Julien has been working
together with Wolfgang and Ludovic from <a class="reference external" href="http://www.inverse.ca/english.html">Inverse</a> to leverage the server-side
support in OpenChange to provide native Exchange server support in <a class="reference external" href="http://ww.sogo.nu/">SOGo</a>.</p>
<p>A couple of days ago we <a class="reference external" href="http://www.openchange.org/">announced</a> that there now is an initial version that
allows the use of Outlook against a SOGo server through OpenChange.</p>
<p>There is a screencast <a class="reference external" href="http://www.youtube.com/v/oSZJ95YeXYE">up on youtube</a> (there is also a <a class="reference external" href="http://www.openchange.org/openchange-2010-10-15.mov">.mov version of the
screencast</a>).</p>
<p>As far as I know, this is the first time it’s possible to actually use Outlook
clients with a non-Microsoft Exchange-compatible server, without the need for
plugins on the Outlook side. And it’s all Free Software. Of course, this is
just a preview, and not something we’d recommend everybody to run in production
just yet. But it’s exciting to finally see this come together.</p>
<p>We already have OpenChange packages in <a class="reference external" href="http://packages.debian.org/openchangeserver">Debian</a> and <a class="reference external" href="http://packages.ubuntu.com/openchangeserver">Ubuntu</a> but I hope I can
help get SOGo packaged for both distributions as well.</p>
Samba 4 and OpenChange daily Ubuntu packages2010-09-28T10:39:50+02:002010-09-28T10:39:50+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-09-28:263-Samba-4-and-OpenChange-daily-Ubuntu-packages.html<div class="section" id="daily-builds">
<h2>Daily builds</h2>
<p>As of a month ago there are Ubuntu archives with fresh packages of <a class="reference external" href="http://wiki.samba.org/index.php/Samba4">Samba 4</a> and <a class="reference external" href="http://www.openchange.org/">OpenChange</a>, built on a daily basis day from the latest upstream revision.</p>
<p>This means that it is now possible to run a version of Samba 4 that is less than 24 hours old, without having to know how to extract source code from the version control system that upstream is using, without having to know how to build and install an application from source, but perhaps most importantly: without having to go through the tedious process of manually updating the source code and rebuilding.</p>
<p><a class="reference external" href="http://www.openchange.org/">OpenChange</a> is tightly coupled to Samba 4, so installing a new version of OpenChange usually involves installing a new version of Samba 4 as well. To make matters more confusing, the two projects use different version control systems (Samba 4 is in <a class="reference external" href="http://git-scm.com/">Git</a>, while OpenChange is in <a class="reference external" href="http://subversion.tigris.org/">Subversion</a>) and different build systems (Samba 4 uses <a class="reference external" href="http://code.google.com/p/waf/">waf</a>, OpenChange uses <a class="reference external" href="http://www.gnu.org/software/autoconf/">autoconf</a> and make).</p>
<p>I have been involved in Samba 4 and OpenChange as an upstream developer and more recently also as a packager for both <a class="reference external" href="http://www.debian.org/">Debian</a> and <a class="reference external" href="http://www.ubuntu.com/">Ubuntu</a>.</p>
<p>As an upstream developer for both these projects it is important for me that users can easily run the development versions. It makes it possible for interested users to confirm the fixes for issues they have reported and to test new features. The more users run the development version, the more confident I can be as a developer that <a class="reference external" href="http://lists.samba.org/archive/samba-technical/2010-September/073481.html">doing a release will not cause any unexpected surprises</a>.</p>
<p>As a packager it is useful to know when there are upstream changes that are going to break my package with the next release.</p>
</div>
<div class="section" id="recipes">
<h2>Recipes</h2>
<p>The daily builds work using so-called recipes which describe how to build a Debian source package from a set of Bazaar branches. For example, the <a class="reference external" href="https://code.edge.launchpad.net/~samba-team/+recipe/4.0-ppa">Samba 4 recipe</a> looks like this:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span># bzr-builder format 0.2 deb-version 4.0.0~alpha14~bzr{revno}~ppa{revno:packaging}+{revno:debian}
lp:samba
merge debian lp:~samba-team/samba/unstable
merge packaging lp:~samba-team/samba/4.0-ppa-maverick
</pre></div></td></tr></table></div>
<p>This dictates that a source package should be built by taking the <a class="reference external" href="https://code.launchpad.net/~samba-team/samba/trunk">upstream Samba branch</a> and merging the <a class="reference external" href="https://code.launchpad.net/~samba-team/samba/unstable">Debian packaging</a> and <a class="reference external" href="https://code.launchpad.net/~samba-team/samba/4.0-ppa-maverick">some recipe-specific tweaking</a>. The last bit on the first line indicates the version string to be used when generating a changelog entry for the daily build.</p>
<p>Every night Launchpad (through <a class="reference external" href="https://launchpad.net/bzr-builder">bzr-builder</a>) merges these branches and attempts to build the resulting source package, e-mailing me in case of build problems. Generally I fix issues that come up by committing directly to upstream <span class="caps">VCS</span> or to the Debian packaging branch. There is no overhead in maintaining the daily build after I’ve set it up.</p>
<p>For more information on creating source package recipes, see <a class="reference external" href="https://help.launchpad.net/Packaging/SourceBuilds/GettingStarted">getting started</a>.</p>
<div class="section" id="toolchain">
<h3>Toolchain</h3>
<p>The entire toolchain that does the daily package builds for Ubuntu is Free Software, and I have contributed to various bits of that toolchain over the years. It’s exciting to see everything come together.</p>
<div class="section" id="soyuz">
<h4>Soyuz</h4>
<p>Launchpad consists of multiple pillars - one of those pillars is Soyuz, which I hack on as part of my day job at Canonical. Soyuz is responsible for the archive management and package building. Debian source packages (a combination of upstream source code and packaging metadata) get uploaded by users and then built for various architectures on our <a class="reference external" href="http://launchpad.net/builders">buildfarm</a> and published to the Ubuntu archive or to users personal package archives.</p>
</div>
<div class="section" id="launchpad-code">
<h4>Launchpad-code</h4>
<p>Another pillar of Launchpad is Launchpad-code, which is responsible for the hosting and management of version control branches. Launchpad users can either host their branches on Launchpad directly or mirror branches (either native Bazaar branches or branches in a foreign format such as Subversion, Git or Mercurial). The mirrorring of native and foreign branches happens using standard Bazaar <span class="caps">API</span>’s. In the case of Samba and OpenChange we import the branches of the upstream projects (Samba is in Git, OpenChange is in Subversion) and the packaging for both projects is in Bazaar.</p>
<p>Launchad-code calls out to Bazaar to do the actual mirrorring. Over the last few years I have done a lot of work to improve Bazaars support for foreign branches, in particular on supporting Subversion, Git and Mercurial. As the code mirrorring in Launchpad is one of the biggest users of bzr-svn and bzr-git it has helped find some of the more obscure bugs in those plugins over the last few years, to the point where there are only a handful of <a class="reference external" href="https://dev.launchpad.net/FailingBzrGitImports">issues with Git imports</a> and <a class="reference external" href="https://dev.launchpad.net/FailingBzrSvnImports">Subversion imports</a> left.</p>
</div>
<div class="section" id="bzr-git-and-dulwich">
<h4>bzr-git and dulwich</h4>
<p>bzr-git provides transparent access to Git repositories from within Bazaar and is built on top of <a class="reference external" href="http://samba.org/~jelmer/dulwich">Dulwich</a>. Dulwich is a Python library that provides access to the Git file formats and protocols that is completely independent of Bazaar. James Westby originally started it and I adopted it for bzr-git and further extended it. There are now several other projects that use it as well, including <a class="reference external" href="http://hg-git.github.com/">hg-git</a>, and <a class="reference external" href="http://www.rabbitvcs.org/">rabbitvcs</a>. Apart from James and myself, almost two dozen other people have contributed it so far.</p>
</div>
<div class="section" id="bzr-svn-and-subvertpy">
<h4>bzr-svn and subvertpy</h4>
<p>bzr-svn provides transparant access to Subversion repositories in Bazaar. When I grew frustrated with the existing Subversion Python bindings <a class="reference external" href="http://jelmer.vernstok.nl/blog/archives/218-bzr-svn-now-with-its-own-Subversion-Python-bindings.html">for various reasons</a>, I decided to create independent Python bindings for Subversion from scratch. These bindings have since been split out into a separate project - <a class="reference external" href="http://samba.org/~jelmer/subvertpy">subvertpy</a> - and other projects have since also started using them, e.g. <a class="reference external" href="http://mercurial.selenic.com/wiki/HgSubversion">hgsubversion</a> and <a class="reference external" href="http://basieproject.org/">basie</a>.</p>
</div>
</div>
<div class="section" id="using-the-daily-builds">
<h3>Using the daily builds</h3>
<p>To use the <a class="reference external" href="http://wiki.samba.org/index.php/Samba4">Samba 4</a> and <a class="reference external" href="http://www.openchange.org/">OpenChange</a> daily builds (Ubuntu Maverick only for now), run:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span>$<span class="w"> </span>apt-add-repository<span class="w"> </span>ppa:samba-team/ppa
$<span class="w"> </span>apt-add-repository<span class="w"> </span>ppa:openchange/daily-builds
</pre></div></td></tr></table></div>
<p><em>Currently Playing: Karnivool - Themata</em></p>
</div>
</div>
Proof of concept OpenChange server working2010-06-08T19:09:08+02:002010-06-08T19:09:08+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-06-08:256-Proof-of-concept-OpenChange-server-working.html<p>Seeing <a class="reference external" href="http://openchange.org/">this</a> makes me very happy. It’s taken <a class="reference external" href="http://www.openchange.org/index.php?option=com_content&view=article&id=24&Itemid=15">us</a> a couple of years to get to this point but we’ve finally made it, mostly thanks to the dedication and persistence of Julien and Brad.</p>
CtrlProxy: Looking for a new maintainer2009-09-13T21:26:13+02:002009-09-13T21:26:13+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2009-09-13:241-ctrlproxy-looking-for-a-new-maintainer.html<p>After over 7 years of working on it off and on, I’m looking for somebody to
help maintain (and eventually take over) <a class="reference external" href="http://www.ctrlproxy.org/">CtrlProxy</a>. I started working on
CtrlProxy somewhere in 2002, only a short while after Wilmer started hacking on
<a class="reference external" href="http://www.bitlbee.org/">BitlBee</a>. If I remember correctly I started working on it because I didn’t
want to run a separate dircproxy (the only real competitor at the time)
instance (with configuration) for each <span class="caps">IRC</span> network that I connected to. It was
also just a good excuse to play with the <span class="caps">IRC</span> protocol a bit.</p>
<p>Over the years, CtrlProxy has served as a playground for me to try out new and
interesting things. It’s been rewritten or severely refactored several times in
its early history, the latest time being the 3.0 release (from 2005). I’ve
tried different build systems, I’ve tried different implementation languages,
I’ve tried different configuration file formats, I’ve tried different support
libraries, I’ve tried different version control systems, I’ve tried different
documentation formats. So while it’s definitely been a very educational project
for me personally, I haven’t really had the time or the interest to dedicate to
the project that it deserved during the last few years. This was mostly because
<a class="reference external" href="http://www.samba.org/">there</a> <a class="reference external" href="http://www.wireshark.org/">were</a> <a class="reference external" href="http://www.ubuntu.com/">other</a> <a class="reference external" href="http://www.debian.org/">more</a> <a class="reference external" href="http://www.bitlbee.org/">interesting</a> <a class="reference external" href="http://www.openchange.org/"><span class="caps">FOSS</span></a> <a class="reference external" href="http://bazaar-vcs.org/">projects</a> I spent my
spare cycles on.</p>
<p>These days there are plenty of other good <span class="caps">IRC</span> proxies out there, such as
<a class="reference external" href="http://bip.t1r.net/"><span class="caps">BIP</span></a>, so I doubt CtrlProxy will be missed if it were to disappear. Despite
that, if anybody is interested in taking over, please send me an email
(<a class="reference external" href="mailto:jelmer@samba.org">jelmer@samba.org</a>) or contact me on <span class="caps">IRC</span> (jelmer on the <span class="caps">OFTC</span> and Freenode networks).</p>
<p><em>Currently Playing: Anathema - Shroud of False</em></p>
bzr-builddeb FTW2008-11-07T18:01:30+01:002008-11-07T18:01:30+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-11-07:226-bzr-builddeb-FTW.html<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span></pre></div></td><td class="code"><div><pre><span></span>%<span class="w"> </span>bzr<span class="w"> </span>branch<span class="w"> </span>deb:line6-usb-source<span class="w"> </span>debian
Retrieving<span class="w"> </span>Vcs<span class="w"> </span>locating<span class="w"> </span>from<span class="w"> </span>line6-usb-source<span class="w"> </span>Debian<span class="w"> </span>version<span class="w"> </span><span class="m">0</span>.7.4-1
Branched<span class="w"> </span><span class="m">354</span><span class="w"> </span>revision<span class="o">(</span>s<span class="o">)</span>.
%<span class="w"> </span>bzr<span class="w"> </span>merge-upstream<span class="w"> </span>https://line6linux.svn.sourceforge.net/svnroot/line6linux/driver/trunk
All<span class="w"> </span>changes<span class="w"> </span>applied<span class="w"> </span>successfully.
Using<span class="w"> </span>version<span class="w"> </span>string<span class="w"> </span><span class="m">0</span>.7.4+svn511<span class="w"> </span><span class="k">for</span><span class="w"> </span>upstream<span class="w"> </span>branch.
The<span class="w"> </span>new<span class="w"> </span>upstream<span class="w"> </span>version<span class="w"> </span>has<span class="w"> </span>been<span class="w"> </span>imported.<span class="w"> </span>You<span class="w"> </span>should<span class="w"> </span>now<span class="w"> </span>update<span class="w"> </span>the<span class="w"> </span>changelog<span class="w"> </span><span class="o">(</span>try<span class="w"> </span>dch<span class="w"> </span>-v<span class="w"> </span><span class="m">0</span>.7.4+svn511-1<span class="w"> </span><span class="s2">"New upstream snapshot."</span><span class="o">)</span>,<span class="w"> </span>resolve<span class="w"> </span>any<span class="w"> </span>conflicts,<span class="w"> </span>and<span class="w"> </span><span class="k">then</span><span class="w"> </span>commit.
%<span class="w"> </span>dch<span class="w"> </span>-v<span class="w"> </span><span class="m">0</span>.7.4+svn511-1<span class="w"> </span><span class="s2">"New upstream snapshot.</span>
<span class="s2">% bzr builddeb</span>
<span class="s2">Building using working tree</span>
<span class="s2">Preparing the build area: ../build-area</span>
<span class="s2">Purging the build dir: ../build-area/line6-usb-0.7.4+svn511</span>
<span class="s2">[...]</span>
<span class="s2">Placing result in /home/jelmer/bzr/line6-usb/result</span>
<span class="s2">% ls ../result</span>
<span class="s2">line6-usb_0.7.4+svn511-1_amd64.changes line6-usb_0.7.4+svn511-1.dsc</span>
<span class="s2">line6-usb-source_0.7.4+svn511-1_all.deb</span>
<span class="s2">line6-usb_0.7.4+svn511-1.diff.gz line6-usb_0.7.4+svn511.orig.tar.gz</span>
</pre></div></td></tr></table></div>
<p><em>Currently Playing: Phideaux - Microdeath Softstar</em></p>
Reconciling the Samba 3 and Samba 4 source code trees2008-09-15T22:05:14+02:002008-09-15T22:05:14+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-09-15:223-Reconciling-the-Samba-3-and-Samba-4-source-code-trees.html<p>While a few of us have been working very hard on Samba 4 to allow it to rock your socks off as an Active Directory Domain Controller, some of the other Samba developers have been working just as hard on improving the existing Samba 3 codebase and adding features to that. This situation has caused tension between developers as well as technical problems in the past - code with the same purpose is being developed in parallel, libraries diverge because features are only added in one branch and not in the other, one codebase is considered “obsolete” by some and the other is considered only a playground for experimental features by others.</p>
<p>As of yesterday, we now have the two codebases living in one and the same git branch. This should make it a lot easier for the two to use the same libraries. Better yet, it should allow us to reconcile the copies of various libraries that exist in both codebases, all of which have diverged to some degree in the last few years.</p>
<p>After <a class="reference external" href="http://thread.gmane.org/gmane.comp.version-control.git/94861">a few problems came up</a> merging the two branches the easy way (they both have a directory called “source” and git doesn’t deal well with renaming them to “source3” and “source4” respectively), we decided to replay the history of both branches . This has the disadvantage that all existing branches that are based on the Samba 3 and Samba 4 branches will have to be rebased against the new master branch, but it also means we keep the ability to run “git log” inside of our source directories and having it work right.</p>
<p>Other than the fact that this makes it possible to share more code between the two codebases, one of the ideas we have is also to see if it is possible to provide an Active Directory <span class="caps">DC</span> by glueing the best bits of Samba 3 and Samba 4 together (aka “<a class="reference external" href="http://wiki.samba.org/index.php/Franky">Franky</a>“) before they are eventually merged completely.</p>
<p><em>Currently Playing: Phideaux - Formaldehyde</em></p>
bzr-svn push without file properties2008-06-30T23:30:40+02:002008-06-30T23:30:40+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-06-30:220-bzr-svn-push-without-file-properties.html<p>Ever since bzr-svn started supporting “true push”, people have been complaining about the extra file properties it sets.</p>
<p>The key thing about “true” push is that it preserves the exact revisions that were present in Subversion. This lets bzr behave on Subversion branches transparently using the same <span class="caps">UI</span> you also use for “native” Bazaar branches.</p>
<p>In other words, if I push to a Subversion branch from my machine, then that branch in Subversion contains enough information for somebody else to reconstruct the exact bzr branch I had.</p>
<p>Since some Bazaar metadata can not be represented in Subversion, it is stored in Bazaar-specific Subversion properties. Unfortunately, these file properties show up in email commit notifications and <a class="reference external" href="http://trac.edgewall.org/">trac</a> and so they tend to annoy people.</p>
<p>There are two ways around this:</p>
<div class="section" id="revision-properties">
<h2>Revision properties</h2>
<p>Bazaar-specific metadata can be stored in in custom Subversion revision properties (these don’t show up in commit notifications). Unfortunately, this requires Subversion 1.5 or newer to run on the server.</p>
<p>I hope to start setting revision properties instead of file properties when possible as of the next bzr-svn release.</p>
</div>
<div class="section" id="less-strict-push">
<h2>less strict push</h2>
<p>It’s also possible to throw away any data that can not be represented in Subversion. Since this means that the remote branch won’t end up an exact same copy of the local revisions, this isn’t true push. The two branches will have diverged (no matter how slightly) after such a push so it is necessary to rebase on the remote branch after pushing.</p>
<p>This is similar to the way git-svn pushes data into Subversion - it calls it “dcommit”.</p>
<p>Since this uses rebase it has the usual disadvantages of rebases, which I won’t get into right now.</p>
<p>As of a couple of days ago, bzr-svn now also supports this mode of pushing using the “dpush” command, by popular demand.</p>
<p><em>Currently Playing: Brandi Carlile - The Story</em></p>
</div>
bzr-svn: now with its own Subversion Python bindings2008-06-22T18:17:42+02:002008-06-22T18:17:42+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-06-22:218-bzr-svn-now-with-its-own-Subversion-Python-bindings.html<p><a class="reference external" href="http://bazaar-vcs.org/BzrSvn">bzr-svn</a> has always been using the standard Python bindings that were
provided with Subversion itself. Unfortunately, I had to fix some issues in
these bindings since they were incomplete or broken and thus bzr-svn has always
depended on a development snapshot of Subversion.</p>
<p>As of today, bzr-svn is using its own Python bindings for Subversion.</p>
<p>There were several reasons for switching to our own bindings:</p>
<ul class="simple">
<li>There are no requirements for backwards compatibility within bzr-svn. This
means the <span class="caps">API</span> can be made sane without worrying about the mess it was in the
past and users who still rely on that.</li>
<li>Deployment. It took 2 years for my fixes to the Subversion Python bindings to
be part of a release. It’ll be even longer before Subversion 1.5 makes it
into most available distributions. That makes it very hard to just download
and install bzr-svn.</li>
<li>They’re in plain C, not <span class="caps">SWIG</span>. <span class="caps">SWIG</span> has a big advantage for the Subversion
folks since it can generate python, ruby, java or tcl bindings all at once
without a lot of overhead per language. However, it has issues as well that
make it a bad choice for bzr-svn.<ul>
<li>It generates inefficient code - it generates proxy classes that add more
layers in the stack</li>
<li>Bindings tend to be very much like the C <span class="caps">API</span> rather than “Pythonic”. To make
them more Pythonic, you need tons of typemaps. For example, the Python
bindings in bzr-svn provide an iterator when browsing the revision history
rather than a callback as C and the <span class="caps">SWIG</span> bindings do.</li>
<li>Hard to write - personally at least, I write bindings in C faster than in
<span class="caps">SWIG</span></li>
<li>Adds an extra dependency to the build process. Several people had trouble
building Subversion on their Mac machines because they didn’t have the right
version of <span class="caps">SWIG</span> available.</li>
</ul>
</li>
</ul>
<p>Since all of the patches that bzr-svn depended on previously were in the Python
bindings for Subversion, it is now possible to use bzr-svn with any version of
Subversion newer than 1.4.0. Of course, you do need to have the development
headers installed as well.</p>
<p><em>Currently Playing: Kathleen Edwards - Independent Thief</em></p>
Bazaar in the GNOME world2008-06-19T15:13:20+02:002008-06-19T15:13:20+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-06-19:217-Bazaar-in-the-GNOME-world.html<p>I was happy to see that <a class="reference external" href="http://blogs.gnome.org/johncarr">John Carr</a> has set up a <a class="reference external" href="http://bzr-mirror.gnome.org/">Bazaar Mirror</a> of all
projects in <span class="caps">GNOME</span> Subversion, all created using bzr-svn. There’s also a <a class="reference external" href="http://live.gnome.org/Bazaar">quick
introduction to using Bazaar for <span class="caps">GNOME</span> developers</a> on the <a class="reference external" href="http://live.gnome.org/"><span class="caps">GNOME</span> Wiki</a>.</p>
<p><a class="reference external" href="http://uwstopia.nl/">Wouter</a>, long time Bazaar user and <span class="caps">GNOME</span> dude, recently blogged about <a class="reference external" href="http://uwstopia.nl/blog/2008/06/gnome-specimen-now-in-gnome-svn/">pushing Bazaar branches</a> into <span class="caps">GNOME</span> Subversion, working around the restrictions imposed by the pre-commit hooks in <span class="caps">GNOME</span> Subversion.</p>
<p>The problems John ran into with memory usage in the Python Subversion bindings
encouraged me to continue the work on bzr-svn’s own Python bindings, thus
avoiding any dependency on unreleased versions of Subversion and several other issues.</p>
Git cutting corners2008-04-27T13:13:20+02:002008-04-27T13:13:20+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-04-27:216-Git-cutting-corners.html<p>My relationship with git is still one of love and hate. It cuts corners to
increase performance in a couple of places and that can be really bloody annoying.</p>
<p>For example, jerry renamed one of the top-level directories in Samba 3
(revision 9f672c26d63955f613088489c6efbdc08b5b2d14). Git will skip rename
detection in this revision because of the number of files it affects, thus
causing the output of “git log <path>” of this particular directory to be useless.</p>
<p>I’m the first to admit “bzr log” on directories and files in large history
projects is painfully slow, but at least it gets the output right.</p>
<p><em>Currently Playing: Brandi Carlile - The Story</em></p>
SambaXP and other travel2008-04-26T10:03:04+02:002008-04-26T10:03:04+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-04-26:215-SambaXP-and-other-travel.html<p>It’s been a busy two weeks. Wilco and I drove up to Göttingen on Sunday two weeks ago to spend some days hacking and meeting up with the other developers before the start of <a class="reference external" href="http://www.sambaxp.org/">SambaXP</a>. It was really nice to see everybody again after more than 7 months.</p>
<p>SambaXP was a bit different this year. There were three tracks during the second part of the conference this year, one more than previously and of course, there were several engineers from Microsoft attending this time! Some of the interesting talks this year included Julien’s update on OpenChange, Tridge’s talk on <span class="caps">PFIF</span>, the talk from the likewise folks and of course the talk from Microsofts’ Wolfgang Grieskamp on <span class="caps">SMB2</span>. We also had some other informal discussions with the Microsoft folks about specific topics - very useful!</p>
<p>There are <a class="reference external" href="http://sambaxp.org/index.php?id=137">some photos</a> up on the SambaXP homepage. And just to be ahead of the comments: yes, I know I need a haircut.</p>
<p>I did some initial work on several bits and pieces of code that I hope to expand over the next few months. Volker has started working on ncacn_ip_tcp support and I have been working on making the Samba 3 <span class="caps">DCE</span>/<span class="caps">RPC</span> library compatible with Samba 4. This should allow OpenChange to use Samba 3 in the future.</p>
<p>Guenther, Wilco and I made some initial progress on the policy library, allowing client-side manipulation of (group) policies in Samba. I worked with Simo on trying to get rid of an evil hack in Samba4’s event subsystem.</p>
<p>David Holder blogged about some of the IPv6 development that we did during the conference: <a class="reference external" href="http://www.ipv6consultancy.com/ipv6blog/?p=34">http://www.ipv6consultancy.com/ipv6blog/?p=34</a></p>
<p>And lots of other things I can’t remember at the moment…</p>
<p>After the conference Andrew, Wilco and I drove back to the Netherlands and I played tour guide for a bit showing Andrew around the country during the afternoon and hacking Samba together in the morning. Later this week we took the train to Brussels, Eurostar to London and visited Sam’s company
in the <span class="caps">UK</span> Midlands for a couple of days.</p>
<p>And in the midst of all this, it seems Ubuntu Hardy was released. Congratulations to all those involved!</p>
<p><em>Currently Playing: Brandi Carlile - Turpentine</em></p>
Using bzr-builddeb as a svn-buildpackage replacement2008-03-26T15:21:20+01:002008-03-26T15:21:20+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-03-26:214-Using-bzr-builddeb-as-a-svn-buildpackage-replacement.html<p><a class="reference external" href="http://samba.org/~jelmer/bzr-svn-buildpackage.diffThisslightlyevilhack">This slightly evil hack</a> to bzr-svn allows using bzr-builddeb as a drop-in replacement for svn-buildpackage, making it recognize the “mergeWithUpstream” property svn-buildpackage uses.</p>
<p><em>Currently Playing: Jeff Healey - Mess O’ Blues</em></p>
Adaption blockers Bazaar sprint2008-03-09T17:06:00+01:002008-03-09T17:06:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-03-09:213-Adaption-blockers-Bazaar-sprint.html<p>The London Bazaar sprint is over again for this year. It was really good to meet
everybody in person again and also to meet some of the folks who hadn’t been to
a sprint before.</p>
<p>Last years sprint was mainly about improving performance; this year, we
discussed adoption blockers and how to remove them. A short summary of the
<a class="reference external" href="http://bazaar-vcs.org/SprintLondonMarch08/Brainstorms">brainstorming</a> is on the wiki.</p>
<p><a class="reference external" href="http://beuno.com.ar/">Martin’s Blog</a> has some pictures.</p>
<div class="section" id="tmv">
<h2><span class="caps">TMV</span></h2>
<p>The Mars Volta concert we went to last night in Tilburg was absolutely
brilliant. Very energetic and definitely one of the best acts I’ve ever seen
live. We were standing in the back of a completely packed venue for 3 hours,
but it was very much worth it.</p>
<p><em>Currently Playing: Soft Machine - Teeth</em></p>
</div>
OpenChange Evolution plugin preview and Debian packages2008-01-21T14:42:07+01:002008-01-21T14:42:07+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-01-21:210-OpenChange-Evolution-plugin-preview-and-Debian-packages.html<p>Srini writes that a preview of the <a class="reference external" href="http://blogs.gnome.org/sragavan/2008/01/21/evolution-mapi-exchange-2007-preview/">Evolution OpenChange plugin</a> has just been
published. This plugin is now developed in the Evolution Subversion repository,
but is based on the original plugin that was written by the <a class="reference external" href="http://www.epitech.eu/">Epitech</a> team
that was assigned to OpenChange earlier this year.</p>
<p>I’ve packaged new snapshots of Samba and OpenChange for Debian/Ubuntu.
They’re available from my <a class="reference external" href="http://samba.org/~jelmer/debian/">personal apt repository</a> and will hopefully soon
also be available from Debian experimental.</p>
<p><em>Update</em>: The packages are now in <a class="reference external" href="http://packages.debian.org/experimental/">Debian experimental</a> as well as the
upcoming Intrepid release of Ubuntu. I have removed them from my personal
repository because I was running out of disk quota.</p>
GTK+ LDB Browser2007-12-16T20:49:05+01:002007-12-16T20:49:05+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-12-16:204-GTK+-LDB-Browser.html<p>As some may have noticed, a large portion of my Samba 4 work during the last
few months has been focussed on adding Python bindings for our various public
libraries and the refactoring necessary to make it possible to add Python
bindings. So far, we have bindings for <span class="caps">LDB</span> and <span class="caps">TDB</span> but I intend to add bindings
for most of our public <span class="caps">API</span> so it is possible to, for example, open Windows
registry files, join domains, etc. from Python.</p>
<p><a class="reference external" href="http://ldb.samba.org"><span class="caps">LDB</span></a> is our <span class="caps">LDAP</span>-like embedded database, and is for <span class="caps">LDAP</span> what sqlite is for
<span class="caps">SQL</span>. Last night I decided to see how hard it would be to write a graphical
browser for <span class="caps">LDB</span> using Python, and it turned out to be quite easy, thanks to
PyGTK. There is a screenshot of what it looks like <a class="reference external" href="http://samba.org/~jelmer/gtkldb.png">here</a>. Packages with the
Python bindings for <span class="caps">LDB</span> are <a class="reference external" href="http://packages.debian.org/python-ldb">already in Debian</a>.</p>
<p>The sources for gtkldb are available in the samba-gtk bzr branch at
<a class="reference external" href="http://people.samba.org/bzr/jelmer/samba-gtk/trunk">http://people.samba.org/bzr/jelmer/samba-gtk/trunk</a>, along with some of the <span class="caps">GTK</span>+
frontends for Samba 4 I wrote earlier (gregedit, gwcrontab, gwsvcctl, gepdump
and gwsam).</p>