path: root/doc/README.devel

== GETTING STARTED ==

Git repo: git://dev.laptop.org/git/projects/olpc-os-builder
Mailing list: http://lists.laptop.org/listinfo/devel

olpc-os-builder can be run in "uninstalled mode" from a standard git
checkout.

 1. Install build and run-time dependencies

    These can be found in the olpc-os-builder.spec file in Fedora GIT. Instead
    of going to the hassle to look them up, you could just run:

       yum install olpc-os-builder

    This will ensure that all dependencies are present, and having a systemwide
    installation of olpc-os-builder will in no way affect your "uninstalled"
    version.

 2. Build the relevant binaries, from the git checkout:

       make

 3. Produce a build: (must be run as root)

       ./osbuilder.py examples/f14-xo1.5.ini

 4. View and use build result files:

       ls -l build/output


== DESIGN ==

olpc-os-builder is composed of a series of modules, and an INI-style
configuration supplied by the user, which selects the modules to be used
and any settings to apply to them.

The build process is composed of a series of stages. In a nutshell, the
progression of these stages is modelled around the following actions:
 1. Prepare
 2. Generate kickstart file
 3. Run image-builder with kickstart file
 4. Run some final modifications on resultant image
 5. Convert image into format appropriate for distribution to XOs
 6. Finalize and cleanup

Each module has its own directory. A module can read the configuration
supplied by the user, which may well have settings specific to the module
in question. The convention is that each module should look out for its
own section in the config file, and such a section should have the same
name as the module itself.

A module can define default configuration values, by providing a "defaults.ini"
file in the module directory. While it is possible to provide default values
for any section in this file, it is advised that all modules only provide
default values for a section with the same name as the module itself. (The
exception being 'base' which is where the [global] defaults are defined)

Each module is composed of a series of parts, which are included in the
module's directory. These are the files that make up the implementation
of what the module tries to achieve.
 - Each part belongs to one specific stage,
 - A module can present multiple parts for each stage
 - A module can provide parts for any subset of stages (it doesn't have to
   provide a part for every single one)
 - Each part is be labelled numerically to determine in which order it gets
   executed within the execution of a stage (relative to parts provided from
   other modules, which all share the same numbering namespace)
 - A part can be a shell script, or a Python program, or a plain text file that
   simply gets included in the output object.

The filename format for a part is:
	<stage>.<order>.<name>.<extension>

 stage		one of the stages documented below

 order		a 2-digit number indicating at which point during the execution of
			a stage the part should be executed. When a stage is executed, all
			of the parts from all of the modules are placed into a collection,
			ordered by based on this number, and then executed from 00 to 99.

 name		A textual name for the part

 extension	.sh (shell), .py (python), or .inc (plain text for inclusion)

Shell parts are always run under 'set -e' so you must be careful to trap
any expected failures that are part of your regular script flow.

Each stage either has a purpose of generating content for a specific output
file (i.e. the kickstart file), or for running scripts/programs to formulate
the output files.

All of the stages that begin with 'ks' are for generating the kickstart file.
Any output on stdout from the parts within will be included in the kickstart
file.

All of the other stages do not have a specific output file, but the part
implementations themselves can (and do) generate their own output files
in the output directory (explained below). It is not possible to use parts
with the .inc extension in these other stages, they are only for scripts
and programs.

Much of the core functionality of the application (e.g. running image-builder)
is embedded within the 'base' module.


== STAGES ==

- prepare
Any initial preparations for the road ahead


- ksmain
Core kickstart options to be included at the top of the kickstart file
(e.g. keyboard, lang, user)


- kspkglist
Entries to be included in the kickstart %packages list


- kspost
Each kspost part becomes its own bash-interpreted %post script.
These scripts are run under "set -e" meaning that if any command fails,
the script will abort with error. These scripts are run with
"%post --erroronfail" which means that any such error will cause the whole
build to abort.

If you want to run a script outside of the chroot, include "nochroot" in the
name of the part.

Note that if you provide a .sh or .py script as a kspost part, this script
will be executed in order to make a %post shell script. Therefore the task
of your kspost part is *always* to produce shell code to be executed later
(at build time). This is in contrast to your expectation that your kspost
script would be called at %post time -- this is not the way it works.
These 'ks' stages are *only* for generating the kickstart file.


- build
Once this stage is reached, the kickstart file is complete. image-creator
is now executed by the base module within this stage in order to produce
the intermediate image.


- mountfs
This stage is responsible for taking the output from image-creator and
locating a filesystem image, moving it into the intermediates directory
where it can be written to, and mounting it at the fsmount directory.


- preimage
This is where any final modifications are made to the filesystem before
it is transformed into an XO-ready image. Note that where possible, all
filesystem modifications should be made as kickstart %post scripts
(see the kspost stage). This is only for the exceptional cases, and things
that really really have to run at the very end.


- image
This is where any appropriate modules should take the contents of the fsmount
and transform it into an XO-ready image.


- postimage
In this stage, modules can use the fsmount to produce any additional output,
such as a package list or a tarball of the entire filesystem.


- unmountfs
This stage is responsible for unmounting the intermediate filesystem that
was mounted by fsmount.


- finalize
Anything that needs to be run at the very end of a successful build.


- cleanup
This stage is special - if any other stage fails, then this stage is executed
and then the whole build system bails out. Error codes from these parts
are ignored.

This stage does not need to worry about cleaning the output or intermediates
directories, since they are cleaned automatically when appropriate.

The usual task to do here is to unmount anything that was mounted by another
part in the module, bearing in mind that it might not be mounted.


== LIBRARY ==

Parts can take advantage of files in the library directory 'libdir' in
order to simply access to the build number, the various directories listed
below, and to read the configuration.

In a .py part, simply:
	import ooblib

In a shell script:
	. $OOB__shlib

The documentation for these libraries is available in the files themselves.


== DIRECTORIES ==

Parts are concerned with a series of directories:

- fsmount
This is where the fsmount stage should mount the filesystem for tweaking
and conversion into XO-ready images.

- libdir
This is where shared library code is kept for the parts, simplifying various
functions such as reading the build number.

- bindir
This is where some C programs that are part of the distribution can be
found.

- cachedir
Modules can write files to this directory which will generally not be deleted
between builds. For example, files that are downloaded can be saved here to
avoid having to download them on every build.
This directory can be legally deleted by the user between builds, so modules
must be able to function even when the cache has been cleared (e.g. by
re-downloading the required files)

- intermediatesdir
Parts can output files here to be used as inputs to other parts (even from
different modules).
This directory is cleaned before a build is started, and after completion
of a successful build.
A "shared" subdirectory is automatically created and this is bind-mounted
inside the build root at /build_shared. It is a useful place for sharing
files inside and outside the build environment. It is available for parts
with order number 06 to 84 inclusive.

- outputdir
This is the final directory that is presented to the user. All final
'deliverables' should be placed here.
This directory is cleaned before a build is started.


== CONVENTIONS IMPLEMENTED BY CORE MODULES ==

- Build number
Each build has a number. A module responsible for selecting a number should
write it to a file named "buildnr" in the intermediatesdir during the prepare
stage.

- Output file naming
Output files are generally named with the prefix 'os', followed by the build
number, followed by a '.' and followed by some text to indicate what it is.
e.g. os43.packages.txt for the package list

- No overlap in output files
Modules should avoid writing to the same output files.
For example, sd_card_image and jffs2_image should name their resultant .img
files differently. (e.g. osXX.img for jffs2, and osXX.disk.img for SD)


== RELEASING AN OLPC OS BUILD ==

When releasing an official OLPC OS build, you should do the following:
 - Modify the config so that [base].official=1
 - Make and test the build
 (assuming test success...)
 - Bump olpc-os-builder version number
 - Modify config to set suggested_oob_version in [base] to the first two
   components of the new version number (e.g. 1.3)
 - Delete [base].official setting from config
 - Save config in examples/ directory with appropriate name
 - Tag and release new olpc-os-builder

The base.official option basically requires that the user supplies a
base.customization_info setting. If they don't, "unofficial" will be used.
However, if base.official is 1, customization_info can be left empty and
no "unofficial" tag will be appended. This will allow us to easily distinguish
between official OLPC releases, and releases that are made by third parties.

Bumping the version and noting that version in the config is important in
order to provide a firm record for the environment under which the build
was made. If deployments try to replicate the same build but use a different
oob version with different module code, then they will end up with the
undesirable result of an OS build that is somewhat different from the official.


== VERSION NUMBERS ==

The olpc-os-builder version number has 3 components, e.g. v1.2.3

The first component relates to the version of Fedora the tool works with.
It must be incremented every time the tool is modified to produce builds based
on a new version of Fedora.

The second component refers to the minor release of the official OLPC OS
release that the tool was used to build. It must be incremented when starting
work on a new minor release. For example, v1.2.0 was used to build
OLPC OS 10.1.2, this was incremented to v1.3.0 when development started on OLPC
OS 10.1.3.

The third version component starts at 0 for a given OLPC OS release. The
official OLPC OS release is always made with 0 as the third component
(e.g. v1.3.0). If new versions of olpc-os-builder are released that still
build the same OLPC OS release, this number gets incremented (see below for
more information).


== CODE FREEZES ==

By definition, OLPC OS releases are always made with olpc-os-builder releases
that have 0 as the 3rd version component (e.g. v1.3.0). As an aim of the tool
is to allow for exact reconstruction of OLPC OS releases into the future,
the branch where this olpc-os-builder was made automatically goes into a code
freeze.

In this frozen state, further commits can be made, but only for small patches
which undoubtedly do not affect the build output when the standard OLPC OS
configuration is used. The equivalent functionality or fixes must already be
present in the master branch. If there is any doubt as to whether the change
would affect the standard build output, it is not applied.

Examples of commits that are permitted are fixes to bugs that prevented the
build tool from running at all under certain environments, small changes to
modules not used by the official OLPC OS configuration, etc.