How does SBuild perform what it does?
|Table of content|
Programmers like to have an overview of how their tool behaves, the 1000-feet view of the steps it takes behind the scene to perform some task. Especially, SBuild claiming that it is a highly programmable and a highly customizable tool for programmers, we need to provide this overview.
Here is a first approximation of what happens when you run a build script on your command line:
By "loading" we understand the following: the file(s) to load is found, then it is understood (parsed), then it is executed and the functionality provided in the file is exposed for the rest of the program. For example, suppose we talk about "loading the project local variants", which happen to live in a file named _sb_vars.py, then these are the steps that we designate all together as "loading":
The same "loading" process applies to SBuild toolkits and to project customization settings.
In general, each step that SBuild takes is announced by clear messages in stdout. Just watch them and you will know. Can increase verbosity with option -D.
Your script starts with Python importing the file _sb_bootstrap.py. This file must be present in each and every SBuild project. The chief reason to be of this file is to fetch the SBuild tool. Once that is done, the namespace sb becomes available to the rest of the program.
The way the SBuild tool is found is not frozen. The file _sb_boostrap.py is free to use any Python code to achieve its goal. It will certainly depend on how the SBuild tool is installed on your development machine. Typically, _sb_bootstrap.py looks for the existence on disk of one hard-coded directory (or several, in turn) and tries to load SBuild from there. By convention, failing that, it will look then for a shell environment variable called SBUILD_LOCATION and try to load from the directory mentioned in it. Technically, the directory that you put in SBUILD_LOCATION is the root directory of a so-called "SCons local installation" (the SBuild code is just a subdirectory inside SCons).
Loading the toolkits
The toolkits are just directories with Python code that enrich the SBuild with new supported target types. The list of the toolkits to load is physically in _sb_boostrap.py and the toolkits may be subdirectories in the SBuild tool or subdirectories in your SBuild project (so next to your build script). For example, the C/C++ toolkit of SBuild lives in the directory tk_ccpp, which is a subdirectory somewhere in the SBuild tool.
Loading toolkit variants
Toolkits have, among others, their toolkit parameters and their toolkit variants. A toolkit usually has a file <toolkit_name>_vars.py. This file holds, among others, definition of variants and definition of SBuild API extensions. Loading this file is an integral part of loading the toolkit and that happens before project local initialization. After a toolkit is loaded, its variants are available in sb.vars. For example, since tgtplatform is a variant specified in ccpp_vars.py, after tk_ccpp was loaded, you can use sb.vars.tgtplaform. Notably you can use it in your SBuild project files like _sb_vars.py and in your build scripts. SBuild API extensions are just ordinary Python functions that are defined in the file <toolkit_name>_vars.py. After toolkit loading they become available as sb.<name>() for the rest of the program.
Command line parsing
Reminder: The command line may have options, -<letter>, or keyword arguments, <key>=<value>. Any other kind of argument is not allowed. Many options are in fact aliases to save typing of keyword arguments. For example, -R is an alias for action=run(). Run with -h to learn more.
Parsing the command line involves making a Python dictionary sb.opts where all keyword arguments are stored. The keyword arguments can be 1 of 2 things:
For both toolkit parameters and variant values, if a shell environment variable specifies a value and the command line specifies a different value, the command line wins. Use the keyword argument explain=sum to your build script to make it dump the current values after all initialization is done.
Project local initialization
Reminder: an SBuild project is just a directory containing build scripts. The SBuild project also contains a few files, names _sb_***.py, files that contain things to be shared by all build scripts in this project. Two of them are important for the discussion here (both are optional):
The important thing is that _sb_vars.py is loaded before _sb_custom.py. The way to remember it is that _sb_vars.py defines variants and _sb_custom.py changes their value, which can be reasonably done only after they have been defined. Since _sb_vars.py comes first, a function func() defined in it can be used as sb.func() in _sb_custom.py and, of course, in the build scripts.
Important to know, these 2 files, _sb_vars.py and _sb_custom.py, are seen by SBuild merely as 2 name spaces. This means that they may be split over several physical files (that are brought-in using the Python statements import of exec).
Pitfalls of delayed interpretation of _sb_custom.py
Assuming that you have a variant in your _sb_vars.py named myvar with default value "foo". Then the following piece of Python code in your _sb_custom.py will most likely puzzle you:
It will print out "foo". Indeed, "foo" is still the current value of myvar at that moment. The value will change to "bar" but only after the _sb_custom.py is completely parsed by Python (and possibly never, if for example you also have on the command line myvar=somethingelse). The myvar, in myvar = "bar" above, is merely a Python string variable. It is a request to change the value of the variant not the real variant, pretty much like myvar=bar on the command line is merely a request to change the variant, not the variant itself.
You may wonder why not allow a syntax like sb.vars.myvar.set("bar"), which takes effect immediately and removes any possibility for non-intuitive behaviour. Know that this set() exists but it is reserved for internal SBuild usage, for technical reasons explained further below.
When is it final?
The entire initialization is done when both the SBuild initialization and project local initialization are done. That's the moment when the function sb.init() at the top of your build script finished.
Note that variants are read-only for the build scripts. You can change their value from several places (command line, shell environment, project local customization (_sb_custom.py)) but not from the build scripts. This is required in order to keep the target tree consistent: if set() were allowed, then situations would be possible where build scripts may get() different values for one and the same variant, depending if the build script doing get() is "after" or "before" the script doing set() when the target tree is constructed.
What happens after initialization? The target specifications in your script are looked at (all those sb.T()). Spread around and between target specs, arbitrary Python code (including SBuild extensions of your own) may change your target specs in some way.
Then Python reaches the bottom of your script, the function call sb.run_or_collect(). If this is a top level build script (meaning the build script on your command line), that call 1. constructs the target tree and then 2. performs actions on the targets in this tree. If this script is a lower-level build script (meaning loaded automatically by the top script while looking for subtargets) then this call does basically nothing.
Resolving the root target
The crucial thing that SBuild starts with is the root target, although you rarely notice or do anything about it. That is because you choose a build script and each script has one default target (or several). So, in order to build something, you choose implicitly a root target by choosing a script.
There are several ways SBuild may choose or even construct on the fly the root target:
Constructing the tree imports lower build scripts
One the root target identified, SBuild can start looking for subtargets. A target spec usually has the attribute sub_tgts and that is a list of target queries. Such query may point to another target spec in the same build script but may also point to a target spec in another build script. Therefore, while constructing the tree of targets, SBuild loads lower level build scripts (and may fail if problems in those scripts).
Importing these lower build scripts cannot be done earlier. Imagine that you change the root target from the command line, then a different tree is to be constructed and possibly other lower script imported. Use the verbose mode (option -D) to see exactly what files are imported when by SBuild. Dump the SBuild tree of targets using the -T option. It also shows to you in which build script each involved target spec lives.
Once the target tree constructed in memory, SBuild considers the action that you requested. Some actions, named SBuild actions, can be and are performed directly by SBuild like, for example printing out sources (option -S) and running (option -R). But the most interesting actions are delegated by SBuild to SCons, like incremental building (option -b) and cleaning (option -c). When SBuild starts SCons, a clear message on screen announces that. Once SCons took control, it will never give it back (although SBuild gets a brief chance to say good bye, through a Python atexit hook).
The file sconscript.py
This is an SCons build description. It drives what SCons performs for SBuild. It contains the following steps:
Following that, SCons performs actions on the files in the SCons tree, like building, dumping command lines, etc. SBuild tell SCons what action by preparing the sys.argv.
You can dump the SCons file tree using option -t or -O (take care, for large builds, it takes quite a while and the output may be huge, better display first the SBuild tree, with -T, to get an idea of the size of the build).
You can define hooks in your _sb_vars.py file, meaning Python functions that SBuild executes at given moments in time. Hooks are merely Python function with a particular name: hook_<when_to_call_it>. For example, a popular hook is hook_post_init() and it is usually used to print out the value of some variants that are frequently used. Currently, SBuild has the limitation that only one such hook function can be defined for each moment where you want to hook something.
The hooks currently supported are: