Google Summer of Code 2012 – Clang support for build services in Debian – Final Result

In the process of installing and configuring wanna-build, I had lots of problems. Because there was almost no documentation on the subject (and some of the existing was outdated), I had to figure out how everything works by reading the source code and asking people on IRC and mailing lists. I use the most recent version (from git) and since it’s not stable, there were some problems and bugs. I fixed several bugs in order to set the system up. I also documented every step so now there’s documentation on everything I have done. I uploaded the documentation on installing and configuring wanna-build [0].

I installed and configured reprepro on a local machine in order to use it with wanna-build. In the process I made step-by step setup instructions [1].

I installed and configured sbuild. The lastest version of sbuild contains filters but they don’t work. Their purpose is to keep only the necessary variables in the environment, but apparenly the configuration file which should set them is never read. That leaves the filter list empty which causes the environment to be empty. I had to disable them to keep my work going.

I installed buildd. I had some problems setting it up – there was a bug in the scripts related to parsing the configuration file. I was pointed to a helpful bug report on the topic and with some additional changes I eventually got it working. In the process I had to read some of the source code of buildd.

I made a patch that fixes a bug in the manual page of reprepro and got it approved.

I contacted the Alioth admin team via the web site, requesting to use their server for my project “Clang for build services”. I got approved and I created a git repository where I’m now uploading my work.

The repository that I use to upload changes to sbuild and buildd is [2].

I add the name of the used compiler to the control file and to the name of the deb package. I wrote a function for parsing the debian/rules file looking for hard-coded compiler there. If there is such compiler, which is different from the one chosen as a default, the building proccess exits with a clear error.

I made changes to wanna-build, dpkg, sbuild and buildd to fix bugs in them and make it possible to use a compiler different from gcc for the build process.
After my work a build compiler can be set easily. If a package can not be built with that compiler, the build process fails with a clear error. I also modified the pgstatus web interface to display build logs and create reports on user request.

I wrote complete documentation about setting up wanna-build, reprepro and buildd ([0] and [1]). This makes a fully functional buildd infrastructure. Now we have a full step by step tutorial.

Me and Sylvestre Ledru who was my great mentor, modified the buildd daemon so it will be possible to be run several instances of it for different architectures on a single physical server.

I made a lot of changes to pgstatus – this is a web interface to the wanna-build database. I had to add some features and fix several problems. In the process I read some of the code of pgstatus and buildd-tools. Now I have a better understanding of how they work.

I added a new feature in pgstatus that allows the user to create a full report of the state of all builds. When a designated reports page is opened, a table with the current state of the builds is shown, together with a list of all the reports saved to files for the current distribution and architecture. The reports can be downloaded directly from the
interface. They are stored in JSON format.

The log files were not accessible from pgstatus because of lack of implementation and bugs. I created a script that reads emails sent by wanna-build and extracts log files from them.

I made changes to wanna-build so that it feeds the database with the necessary information about package history. I updated the wanna-build patch to include these changes.

As a result the buildd logs can now be opened from the web interface of pgstatus.

I applied a patch that fixes bug [3] so that config_schroot.sh (this script installs and sets the default compiler) starts with the necessary privileges.

I added a new feature to sbuild that allows scripts to be run right after the dependencies are installed in the chroot and before the actual package starts building. Now anybody can configure sbuild to run such a script by modifying the appropriate configuration file. By adding the following lines:

$external_commands = {
“chroot-pre-build-commands” => [
['/root/purge-gcc.sh'],
],
};

The script /root/purge-gcc.sh is run by sbuild in the chroot.

This script can be necessary to set everything in the environment so that packages are built exactly as needed. Me and my mentor used this to replace gcc with a script that removes all gcc-specific packages so that we can be sure whether a package uses clang or not.

I have a fully functional buildd infrastructure installed on my development system. In order to get it running, I made several patches:

dpkg.patch – The main patch with a lot of changes related to compiler support for building.
wanna_build.patch – Fixes bugs and makes wanna-build feed the database with package history information.
sbuild.patch – Fixes bugs and work-arounds the problem that some sbuild scripts are called without reading the configuration file of sbuild. Adds configuration option that allows scripts to be run right after the dependencies are installed in the chroot and before the actual package starts building.
pgstatus.patch – Fixes bugs and adds reports page to pgstatus.

Me and my mentor contacted the Debian wanna-build team to connect our new service with Debian. We explained what the current state of the project is and what we need from them.

I thank my mentor – Sylvestre Ledru for his presentation on DebConf 12 ([4], [5]). He showed our project and it was seen that people are interested in it. A nice resume of his talk has been made by Michael Larabel on Phoronix [6].

I tested everything thoroughly to make sure my changes don’t break anything. I am happy to say that the project is now finished.

====

[0] http://wiki.debian.org/DebianWannaBuildInfrastructureOnOneServer
[1] http://wiki.debian.org/SetupBuildServiceForWanna-build
[2] http://anonscm.debian.org/gitweb/?p=pkg-llvm/clang-build.git;a=summary
[3] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608840
[4] http://sylvestre.ledru.info/blog/media/5/rebuild-debian-presentation-clang-July-10.pdf
[5] http://meetings-archive.debian.net/pub/debian-meetings/2012/debconf12/high/884_Build_Debian_with_another_compiler.ogv
[6] http://www.phoronix.com/scan.php?page=news_item&px=MTEzODQ

Google Summer of Code 2012 – Clang support for build services in Debian

This project is about extension of the Debian infrastructure with ability to build packages with clang. The key idea is to have an alternative way to build packages and easily switch between clang and gcc/g++. Clang is faster than gcc, uses less memory, and produces faster code. It is getting better with time. It is available under the BSD license. It is well documented. It can help programmers find and analyze easily their errors, because clang has much more detailed error and warning messages and it allows better diagnostics. It also provides a good alternative for programmers using gcc in the coding and compiling processes. It enforces better code quality.

In the compiling process, compiler choice is usually done by setting the environment variable CC. When Makefile is generated, it contains initialization of the CC variable based on the current environment. If CC is not set, the default compiler is gcc. CC is always used by make. I proved this with tests (I set CC=/usr/bin/clang).
There must be an easy way to switch the default compiler between gcc/g++ and clang. One way would be by issuing a command like
“dpkg-reconfigure default-compiler”. I think that debuild, sbuild can be changed to take optional compiler argument (something like –clang –gcc). If this argument is not specified, the default compiler is used (according to CC)
Also I think that when the default compiler is set to clang, some special cases should be checked.
Example: There is gcc hard coded in the Makefile. In this case there should be a dialog box that offers several options to choose from:
1. Leave gcc in this line.
2. Change gcc with the CC variable.
3. Edit with the default text editor.
I have seen the 3rd option in many applications, like svn or crontab. When some file modification is required, the default editor is launched to modify the file.
Also, I’ll check what are the compiling problems of the current packages that can’t compile by just changing gcc with clang. This way I’ll see what checks for special cases to implement.