Source Mage GNU/Linux

Programs
Speedtouch Project
Snoopy Project
eciadsl Project
The bread recipe
Source Mage
BeOS
My resume
My blog
Notes
Donation

en fr

Software installation specification from precompiled binaries

Benoît Papillault, 2004-11-23

Goals

Software installation in the Source Mage GNU/Linux distribution is done by using the cast command which download and compile software from its source code. This option is not realistic when doing installation CD-ROM. Thus we will analyze what cast does to try to reproduce it by using files present under /var/cache/sorcery.

Differents types of spells

There exist currently two types of spells : BUILD_API=1 and BUILD_API=2. This is a shell variable indicating spell structure. It can be defined at 3 locations (we take the glibc spell from the stable grimoire as an example here):

At the grimoire level: /var/lib/sorcery/codex/stable/API_VERSION
At the section level: /var/lib/sorcery/codex/stable/libs/API_VERSION
At the spell level: /var/lib/sorcery/codex/stable/libs/glibc/DETAILS

Absence of this variable is treated like BUILD_API=1.

What `cast` does (BUILD_API=1)

We will detail only what is relevant for the next paragraphs. All the compilation is watched by the installwatch tool, next the POST_INSTALL file is executed if it exists.

Once this is done, the binary cache in /var/cache/sorcery as well as the md5sum file is built. It means that the POST_INSTALL must not modify those files. Otherwise, the md5sum checking done by cleanse will fail.

What `cast` does (BUILD_API=2)

The process is identical, except that the POST_INSTALL script is replaced by the FINAL script.

Installation from precompiled binaries

The installation is done in 4 steps:

dependencies installation using the same process recursively. Thos dependencies are listed in the DEPENDS file from the grimoire with the depends and optional_depends functions.
Warning: sorcery does not allow for optional dependencies tracking. For instnance, openssl is an optional dependency of irssi. If you compiled irssi and selected openssl, the resulting binary will work only if openssl is installed. So, if you later install irssi from the binary cache, sorcery will not install openssl. However, those dependencies (optional or not) are recorded in /var/state/sorcery/depends.
binary file extraction from the archive in /var/cache/sorcery. This archive could be downloaded from local support (existing system, mounted partition, CD-ROM, nfs mounts) or network support (http, ftp, ...).
When using a network suport, we need a way to get the list of available archives. The easiest way is to create a file ls containing the result of the ls command.
updating sorcery files:
- /var/state/sorcery/packages
- /var/state/sorcery/depends
- /var/cache/sorcery/$SPELL-$VERSION-$HOST.tar.bz2 (optional)
launching a post-installation script: POST_INSTALL for BUILD_API=1 and FINAL for BUILD_API=2. This script, as well as the BUILD_API determination, depends on the grimoire. Thus, there is a potential risk if the grimoire does not match the grimoire used at compile time.

File format and location

The various files needed to install software from precompiled binaries may exist in various forms:

on an installed system: (what the resurect functionality of cast does approximately)
- installed version: /var/state/sorcery/packages
- dependencies: /var/state/sorcery/depends
- precompiled binaries (optionel): /var/cache/sorcery/${SPELL}-${VERSION}-${HOST}.tar.bz2 file list: /var/log/sorcery/install/${SPELL}-${VERSION}
- post-installation script:
  - POST_INSTALL if BUILD_API=1
  - FINAL if BUILD_API=2
in a remote (ftp/http/rsync/cvs/scp) or local (partition, nfs) system. This system will be made of a unique directory containing all the precompiled binaries as well as a ls file. It will be symbolicaly accessible through the URL ${BASE} which will be used for the description.
- precompiled binaries list in the ${BASE}/ls file.
- ${BASE}/${SPELL}_${VERSION}_${HOST}.tar.bz2 containing:
  - the list of files to install, for instance:
```
usr/bin/which
usr/share/info/which.info
usr/share/man/man1/which.1
var/log/sorcery/compile/which-2.16.bz2
var/log/sorcery/md5sum/which-2.16
var/log/sorcery/install/which-2.16
```
  - install/depends
    will be created from the matching line in /var/state/sorcery/depends. The whole line must be copied to be later restored in the system to install. For instance:
```
irssi:glib2:on:required::
irssi:ncurses:on:required::
irssi:openssl:on:optional::--disable-ssl
irssi:perl:off:optional:--with-perl=module:--with-perl=no
```
  - install/postinst
    is a copy of POST_INSTALL or FINAL.
on a ISO system. This is an hybrid system where we try to gain disk space. We will find software in both versions: either already installed, either in tar.bz2 form in the directory /install.

Tools

In order to use the system presented here, we need two tools: a tool to create tar.bz2 file containing an install directory (spell-binary-make) and a tool to install them (spell-binary-install).

spell-binary-make

Usage: [BASE=/install] [INSTALL_ROOT=/mnt/root] spell-binary-make spell
Requirement: the spell must be already installed, as well as all its dependencies.
Output: a file ${SPELL}_${VERSION}_${HOST}.tar.bz2 in the ${BASE} directory for the spell and all its dependencies.
Note: the spell name can be modified in some cases. The caracter _ is used as delimitor and thus caracters _ are replaced by - in spell names.
BASE is the output directory containing ${SPELL}_${VERSION}_${HOST}.tar.bz2 and INSTALL_ROOT is the source system's root directory. Default values: BASE=. et INSTALL_ROOT=/

spell-binary-install

Usage: [BASE=/install] [INSTALL_ROOT=/mnt/root] spell-binary-install spell
BASE is the input directory containing ${SPELL}_${VERSION}_${HOST}.tar.bz2 (it can be a URL) and INSTALL_ROOT is the destination system's root directory. Default values: BASE=. et INSTALL_ROOT=/
If no ${SPELL}_${VERSION}_${HOST}.tar.bz2 file exist, an installation is attempted from the running system. This mode is used by the ISO.

Implementation

Both tools are implemented by bash shell scripts and can be downloaded:

This implementation has some restrictions:

It is now possible to make a package with a spell that has no cache file
The use of spell-binary-make with something different than INSTALL_ROOT=/ can lead to erroneous behaviour
The use of spell-binary-install with an URL using http/ftp is possible in this implementation
The use of spell-binary-install with something different than INSTALL_ROOT=/ can produce bug in the post-installation script
Alien files are not installed

Possible use

ISO size reduction

This system avoids to have precompiled binaries if a spell is already installed on the ISO itself.

Example:

BASE=/install INSTALL_ROOT=/mnt/root spell-binary-install irssi

Installation from another support or network location

This system allows what is commonly called netinstall and will allow to create an ISO without any precompiled binaries. This can also be used to update precompiled binaries without the need for releasing a new ISO, thus reducing the cost of updates and maintainance.