Build Wrappers Overview

The ROCK Linux build scripts auto-detect which files are created by a package

build and use this information to create the package file-list. There are

three techniques which can be used to auto-detect this file list (which one

is used can be selected in the build configuration as part of the "expert

options"):

• flist-wrapper

  The flist-wrapper is the recommended way of doing it. It's an "elf preload

  library". This is a library which is loaded _before_ the c-library and so

  can overwrite functions of the c library. The flist-wrapper lib is overwriting

  all functions which can be used to create files and is writing the filenames

  to a log file before running the original c library function. This log file is

  used to create the package file list after the build has been completed.

  The problem with the flist-wrapper lib is that it doesn't work with statically

  built binaries or with non glibc c libraries. In such environments (e.g. when

  building a dietlibc based system), other techniques - such as those described

  below - must be used for creating the file list.

   

• strace

  This one is rather easy: The whole build is running in a process-tree which is

  monitored by strace. The log file created by strace is later used to create the

  package file list. However: this only works on architectures which have

  an strace command - which is not the case for all architectures the Linux

  kernel and glibc have been ported to.

   

• find

  This one is rather ugly (but sometimes the only alternative): When the build is

  started, a 'timestamp-file' is created. After the build has been completed,

  the find program is used to find all files which are newer than the

  'timestamp-file'. This is really slow and has the disadvantage that

  nothing else can create or modify files, except the build process, while the

  build is running. This should only be used if no other system for creating

  the file list can be used.

   

• install-wrapper

  It is planed to also implement a flist-creation system using the

  install-wrapper (see the other article in this Rolling ROCK issue).

  The install-wrapper is called whenever a build process is using the

  commands install, cp, mv or ln.

  However: Some builds are also installing files using other commands and

  they would not show up in the file-list created by the install-wrapper.

For the package, target or architecture maintainer it shouldn't be necessary

to know how the flist wrapper works internally. However: it can't hurt - so here

are a few additional internals for the flist wrapper library:

The flist wrapper library (as all the other wrappers) is created by the

Build-Tools script and is stored in the build/*/tools.*

directory. It is configured using two environment variables which are

set in the Build-Pkg script:

• FLWRAPPER_WLOG

  The file which contains the names of the file the build process is writing

  to. This is src.*/fl_wrapper.wlog in native or cross builds and

  src.*/R.src/fl_wrapper.wlog in chroot builds.

   

• FLWRAPPER_RLOG

  The flist wrapper library also logs all files the build process is reading

  from or is executing. This information is used to generate the package build

  dependencies in a reference build. You can read the results of this analysis

  in scripts/dep_db.txt or in the package *.cache files. This

  log file is named src.*/fl_wrapper.rlog (or in chroot builds

  src.*/R.src/fl_wrapper.rlog).

The sources for the flist wrapper library can be found at

misc/tools-source/fl_wrapper.c. This C source file is auto-generated

by misc/tools-source/fl_wrapper.c.sh. The logic, which is now

implemented in fl_wrapper.c.sh, was implemented in

fl_wrapper.c using cpp macros - but it turned out that

it is easier to read and maintain as it is now.

The Build-Tools script creates some files like install,

cp, mv and ln in build/*/tools.*/wrapper.

This directory is added to the begin of the PATH environment variable in the

parse-config script and so this programs are used instead of the standard

tools when one of this commands is used during the build.

This wrapper files are just symbolic links to the install_wrapper

program which is modifying the parameter list based on the

INSTALL_WRAPPER_FILTER variable and then running the original

program from the standard bin directories.

The INSTALL_WRAPPER_FILTER variable is initialized to an empty text

in scripts/parse-config and can be extended in the package, target,

architecture or misc parse-config files and the package

*.conf file using the var_append and var_insert functions.

The INSTALL_WRAPPER_FILTER variable must contain a shell command

which is reading the destination filenames for install and the other

commands from input and writing the new filenames to the output. Multiple

commands might be concatenated using the pipe character.

So, e.g. if a package installs a usr/share/man/man1/ls.1 (which creates a

shared-file conflict with coreutils), it's possible to add something like

var_append INSTALL_WRAPPER_FILTER "|" \

'sed "s,share/man/man1/ls.1\$,share/man/man1/ls_x.1,"'

to the package *.conf file and the file will be installed using the

new name ls_x.1.

Use this wrapper with care! It's possible to break a lot with it

because all install, cp, mv and ln

commands during the build are wrapped with it - not only those executed

in make install.

The install wrapper is writing a log file containing the old and new

parameter lists so it's easier do debug problems. This log file is named

src.*/install_wrapper.log (or in chroot builds

src.*/R.src/install_wrapper.log). You might want to run

Build-Pkg with the option noclearsrc if you want

the log file (and the entire src.* directory) be kept for debugging

reasons.

The source code for the install wrapper shell script can be found at

misc/tools-source/install_wrapper.sh.

The Build-Tools script creates a cmd_wrapper application and some

shell scripts like cc, gcc, gcc-2 and

gcc-3 in build/*/tools.*/wrapper. This directory is added

to the begin of the PATH environment variable in theparse-config

script and so this programs are used instead of the standard tools when one

of this commands is used during the build.

The cmd_wrapper program is simply running the 'real' applications

from the standard system bin or sbin directories with a

modified parameter list. A set of environment variables

(CMD_WRAPPER_*) is used to configure cmd_wrapper and

define how the parameter list should be modified:

• CMD_WRAPPER_LOGFILE

  The log file for debugging problems with <tt>cmd_wrapper</tt>. Usually this

  is <tt>cmd_wrapper.log</tt> in the <tt>src.*</tt> directory.

   

• CMD_WRAPPER_BYPASS

  If this variable is set to <tt>1</tt>, the <tt>cmd_wrapper</tt> is neither

  modifying the parameters nor running any other wrappers.

   

• CMD_WRAPPER_OTHERS and CMD_WRAPPER_APPEND_PO

  The CMD_WRAPPER_OTHERS variable contains a colon-separated list

  of other wrapper programs which should be executed before doing any

  modification in the parameter list. This other wrappers do get the program

  name passed as first parameter and and all parameters passed to

  cmd_wrapper as further parameters. The cmd_wrapper

  expects the other wrapper to run it's 1st parameter and so again execute

  the cmd_wrapper.

   

  The CMD_WRAPPER_APPEND_PO variable contains a list of parameters which

  should be added to the parameter list before the other wrappers are

  started. This is used e.g. for the dietlibc subsystem to call the compiler with

  the 'diet' program transparently for the package makefiles.

   

• CMD_WRAPPER_INSERT and CMD_WRAPPER_APPEND

  The value of CMD_WRAPPER_INSERT is inserted at the begin of the

  parameter list, the value of CMD_WRAPPER_APPEND is appended to

  the end of the parameter list.

   

• CMD_WRAPPER_REMOVE

  The parameters listed in this variable are removed from the parameter list.

  It's possible to pass shell-wildcards (like -O*) in this variable.

  Parameters which are added by CMD_WRAPPER_INSERT or

  CMD_WRAPPER_APPEND are not removed - only parameters passed on the

  command line.

   

• CMD_WRAPPER_FILTER

  The shell command specified in this variable (multiple shell-commands separated

  by a pipe-character) is executed and passed the so far modified parameter list

  (parameters separated by a newline-char - i.e. one parameter per line). The

  output of this command (one parameter per line) is finally passed as parameter

  list the the real application. This can be used for complex modifications of

  the parameter list.

This CMD_WRAPPER_* variables are never set directly by the build

scripts - they are set by the wrapper shell scripts based on other

*_WRAPPER_* variables. E.g. the 'gcc-3' wrapper script is setting

CMD_WRAPPER_* to the merged values of CC_WRAPPER_*,

GCC_WRAPPER_* and GCC3_WRAPPER_*.

So if a parameter should be added every time a c-compiler is used, the

parameter should be added to CC_WRAPPER_APPEND. But if the parameter

is gcc-3 specific, GCC3_WRAPPER_APPEND is used instead.

The cmd_wrapper application is built directly by the

Build-Tools script, but the separate wrappers are created

by wrappers.in files which may be placed everywhere in

package/, misc/ and target/<current-target>/.

See package/base/gcc3/wrappers.in for a simple example of such a

wrappers.in file.

The source code for the cmd_wrapper application can be found at

misc/tools-source/cmd_wrapper.c.