How does SBuild perform what it does?

Back to Build developer
Back to SBuild manual home

Programmers like to have an overview of how their tool behaves, the 1000-feet view of the steps it takes behind the scene to perform some task. Especially, SBuild claiming that it is a highly programmable and a highly customizable tool for programmers, we need to provide this overview.

Here is a first approximation of what happens when you run a build script on your command line:

1. SBuild initialization: Your script first finds the SBuild tool itself and loads it. Also the SBuild toolkits are loaded.
2. Project local initialization: Whatever you defined in your SBuild project, for the intention of all your build scripts in it, is loaded
3. Construction of the target tree: Your desired root target is identified and, starting from it, a tree is made by finding all the subtargets recursively. No action is taken yet
4. Running SCons: For most actions to be performed on targets, SBuild starts SCons and hands over the control until the end of execution.

By "loading" we understand the following: the file(s) to load is found, then it is understood (parsed), then it is executed and the functionality provided in the file is exposed for the rest of the program. For example, suppose we talk about "loading the project local variants", which happen to live in a file named _sb_vars.py, then these are the steps that we designate all together as "loading":

• The file _sb_vars.py is located on disk
• The file _sb_vars.py is parsed by Python (may fail because of Python syntax errors)
• The content of the file is interpreted by SBuild (may fail, for example, if you specified for a variant a value that is not an allowed value, through command line or through shell environment)
• The variants described in _sb_vars.py become available to the rest of the program (as sb.vars.<name>)

The same "loading" process applies to SBuild toolkits and to project customization settings.

In general, each step that SBuild takes is announced by clear messages in stdout. Just watch them and you will know. Can increase verbosity with option -D.

Back to top

SBuild initialization

Your script starts with Python importing the file _sb_bootstrap.py. This file must be present in each and every SBuild project. The chief reason to be of this file is to fetch the SBuild tool. Once that is done, the namespace sb becomes available to the rest of the program.

Locating SBuild

The way the SBuild tool is found is not frozen. The file _sb_boostrap.py is free to use any Python code to achieve its goal. It will certainly depend on how the SBuild tool is installed on your development machine. Typically, _sb_bootstrap.py looks for the existence on disk of one hard-coded directory (or several, in turn) and tries to load SBuild from there. By convention, failing that, it will look then for a shell environment variable called SBUILD_LOCATION and try to load from the directory mentioned in it. Technically, the directory that you put in SBUILD_LOCATION is the root directory of a so-called "SCons local installation" (the SBuild code is just a subdirectory inside SCons).

Loading the toolkits

The toolkits are just directories with Python code that enrich the SBuild with new supported target types. The list of the toolkits to load is physically in _sb_boostrap.py and the toolkits may be subdirectories in the SBuild tool or subdirectories in your SBuild project (so next to your build script). For example, the C/C++ toolkit of SBuild lives in the directory tk_ccpp, which is a subdirectory somewhere in the SBuild tool.

Loading toolkit variants

Toolkits have, among others, their toolkit parameters and their toolkit variants. A toolkit usually has a file <toolkit_name>_vars.py. This file holds, among others, definition of variants and definition of SBuild API extensions. Loading this file is an integral part of loading the toolkit and that happens before project local initialization. After a toolkit is loaded, its variants are available in sb.vars. For example, since tgtplatform is a variant specified in ccpp_vars.py, after tk_ccpp was loaded, you can use sb.vars.tgtplaform. Notably you can use it in your SBuild project files like _sb_vars.py and in your build scripts. SBuild API extensions are just ordinary Python functions that are defined in the file <toolkit_name>_vars.py. After toolkit loading they become available as sb.<name>() for the rest of the program.

Command line parsing

Reminder: The command line may have options, -<letter>, or keyword arguments, <key>=<value>. Any other kind of argument is not allowed. Many options are in fact aliases to save typing of keyword arguments. For example, -R is an alias for action=run(). Run with -h to learn more.

Parsing the command line involves making a Python dictionary sb.opts where all keyword arguments are stored. The keyword arguments can be 1 of 2 things:

• Toolkit parameters (ex. lint_usage=0). These are taken in consideration during the toolkit loading to change the hard-coded default value.
• Variant values (ex. bldopt=dbg). These are used by Sbuild twice! Once to change the variant value after the variant become known to SBuild (after the file ***_vars.py containing the variant was parsed). And a second time after the project local customization was executed (so that the command line prevails over settings in _sb_custom.py).

For both toolkit parameters and variant values, if a shell environment variable specifies a value and the command line specifies a different value, the command line wins. Use the keyword argument explain=sum to your build script to make it dump the current values after all initialization is done.

Back to top

Reminder: an SBuild project is just a directory containing build scripts. The SBuild project also contains a few files, names _sb_***.py, files that contain things to be shared by all build scripts in this project. Two of them are important for the discussion here (both are optional):

• _sb_vars.py. This file contains variant specification, sb.V(), external specifications, sb.E(), SBuild extensions (ordinary Python functions). It may also contain hooks (see below).
• _sb_custom.py. This file contain settings in the form <key> = <value>, pretty much like the keyword arguments on the command line. Usually, the <key>=<value> in this file are conditional (under some "if" statement")

The important thing is that _sb_vars.py is loaded before _sb_custom.py. The way to remember it is that _sb_vars.py defines variants and _sb_custom.py changes their value, which can be reasonably done only after they have been defined. Since _sb_vars.py comes first, a function func() defined in it can be used as sb.func() in _sb_custom.py and, of course, in the build scripts.

Important to know, these 2 files, _sb_vars.py and _sb_custom.py, are seen by SBuild merely as 2 name spaces. This means that they may be split over several physical files (that are brought-in using the Python statements import of exec).

Pitfalls of delayed interpretation of _sb_custom.py

Assuming that you have a variant in your _sb_vars.py named myvar with default value "foo". Then the following piece of Python code in your _sb_custom.py will most likely puzzle you:

# here want to change the value of variant myvar
myvar = "bar"
print "Here the value of myvar:", sb.vars.myvar.get()

It will print out "foo". Indeed, "foo" is still the current value of myvar at that moment. The value will change to "bar" but only after the _sb_custom.py is completely parsed by Python (and possibly never, if for example you also have on the command line myvar=somethingelse). The myvar, in myvar = "bar" above, is merely a Python string variable. It is a request to change the value of the variant not the real variant, pretty much like myvar=bar on the command line is merely a request to change the variant, not the variant itself.

You may wonder why not allow a syntax like sb.vars.myvar.set("bar"), which takes effect immediately and removes any possibility for non-intuitive behaviour. Know that this set() exists but it is reserved for internal SBuild usage, for technical reasons explained further below.

When is it final?

The entire initialization is done when both the SBuild initialization and project local initialization are done. That's the moment when the function sb.init() at the top of your build script finished.

Note that variants are read-only for the build scripts. You can change their value from several places (command line, shell environment, project local customization (_sb_custom.py)) but not from the build scripts. This is required in order to keep the target tree consistent: if set() were allowed, then situations would be possible where build scripts may get() different values for one and the same variant, depending if the build script doing get() is "after" or "before" the script doing set() when the target tree is constructed.

Back to top

Constructing the target tree

What happens after initialization? The target specifications in your script are looked at (all those sb.T()). Spread around and between target specs, arbitrary Python code (including SBuild extensions of your own) may change your target specs in some way.

Then Python reaches the bottom of your script, the function call sb.run_or_collect(). If this is a top level build script (meaning the build script on your command line), that call 1. constructs the target tree and then 2. performs actions on the targets in this tree. If this script is a lower-level build script (meaning loaded automatically by the top script while looking for subtargets) then this call does basically nothing.

Resolving the root target

The crucial thing  that SBuild starts with is the root target, although you rarely notice or do anything about it. That is because you choose a build script and each script has one default target (or several). So, in order to build something, you choose implicitly a root target by choosing a script.

There are several ways SBuild may choose or even construct on the fly the root target:

• if argument tgt=toto is on the command line, then SBuild looks in the top script for the target with the id toto and that becomes the root target
• if argument pat=this and/or antipat=that on the command line, then SBuild constructs on the fly an alias target having as subtargets all targets in the top build script with their id matching this and not matching that.
• if no such argument, then SBuild looks in the top script for a Python list named default_targets. If found, SBuild constructs on the fly an alias target having as subtargets all targets listed in default_targets.
• if not found, then SBuild looks in the top script for a target with the id toto assuming the script is called build_toto.py and makes that the root target. If all that fails the build script stops (in fact, each script named build_something.py must have a target with the id something, even if that target is not part of the current build at all).

Constructing the tree imports lower build scripts

One the root target identified, SBuild can start looking for subtargets. A target spec usually has the attribute sub_tgts and that is a list of target queries. Such query may point to another target spec in the same build script but may also point to a target spec in another build script. Therefore, while constructing the tree of targets, SBuild loads lower level build scripts (and may fail if problems in those scripts).

Importing these lower build scripts cannot be done earlier. Imagine that you change the root target from the command line, then a different tree is to be constructed and possibly other lower script imported. Use the verbose mode (option -D) to see exactly what files are imported when by SBuild. Dump the SBuild tree of targets using the -T option. It also shows to you in which build script each involved target spec lives.

Back to top

Running SCons

Once the target tree constructed in memory, SBuild considers the action that you requested. Some actions, named SBuild actions, can be and are performed directly by SBuild like, for example printing out sources (option -S) and running (option -R). But the most interesting actions are delegated by SBuild to SCons, like incremental building (option -b) and cleaning (option -c). When SBuild starts SCons, a clear message on screen announces that. Once SCons took control, it will never give it back (although SBuild gets a brief chance to say good bye, through a Python atexit hook).

The file sconscript.py

This is an SCons build description. It drives what SCons performs for SBuild. It contains the following steps:

• Capture the SCons API. After this moment, the SCons API is available to SBuild toolkits in the namespace sc.
• Set some SCons settings according to command line options and check the version of SCons.
• Create the SCons Construction Environment
• Let the SBuild toolkits enrich the SCons construction environment. This is the important moment when the SCons Tool wrappers are loaded. "Loaded" means that they are parsed and their function generate(env) is called (env being the SCons Construction Environment create above, which is input and output in these calls). Which SCons Tool wrappers exactly are loaded depends on current value of the variant toolchain of the C/C++ toolkit. Just watch the echo messages and you'll see.
• Convert the SBuild tree of targets into an SCons tree of files. This steps transforms the dependency tree of SBuild, where the nodes are SBuild targets, into a dependency tree where the nodes are individual files (SCons only works with files). This SCons tree is larger than the SBuild tree (in number of nodes and in number of levels).
• Pass over to SCons the root of the SCons dependency tree.

Following that, SCons performs actions on the files in the SCons tree, like building, dumping command lines, etc. SBuild tell SCons what action by preparing the sys.argv.

You can dump the SCons file tree using option -t or -O (take care, for large builds, it takes quite a while and the output may be huge, better display first the SBuild tree, with -T, to get an idea of the size of the build).

Back to top

Hooks

You can define hooks in your _sb_vars.py file, meaning Python functions that SBuild executes at given moments in time. Hooks are merely Python function with a particular name: hook_<when_to_call_it>. For example, a popular hook is hook_post_init() and it is usually used to print out the value of some variants that are frequently used. Currently, SBuild has the limitation that only one such hook function can be defined for each moment where you want to hook something.

The hooks currently supported are:

• hook_post_init(module_name)
• hook_pre_collect(script_name)
• hook_per_target(tgt, level)
• hook_pre_scons()

Back to top

Back to Build developer
Back to SBuild manual home