sunnuntai 31. joulukuuta 2017

These three things could improve the Linux development experience dramatically, #2 will surprise you

The development experience on a modern Linux system is fairly good, however there are several strange things, mostly due to legacy things no longer relevant, that cause weird bugs, hassles and other problems. Here are three suggestions for improvement:

1. Get rid of global state

There is a surprisingly large amount of global (mutable) state everywhere. There are also many places where said global state is altered in secret. As an example let's look at pkg-config files. If you have installed some package in a temporary location and request its linker flags with pkg-config --libs foo, you get out something like this:

-L/opt/lib -lfoo

The semantic meaning of these flags is "link against libfoo.so that is in /opt/lib". But that is not what these flags do. What they actually mean is "add /opt/lib to the global link library search path, then search for foo in all search paths". This has two problems. First of all, the linker might, or might not, use the library file in /opt/lib. Depending on other linker flags, it might find it somewhere else. But the bigger problem is that the -L option remains in effect after this. Any library search later might pick up libraries in /opt/lib that it should not have. Most of the time things work. Every now and then they break. This is what happens when you fiddle with global state.

The fix to this is fairly simple and requires only changing the pkg-config file generator so it outputs the following for --libs foo:

/opt/lib/libfoo.so

2. Get rid of -lm, -pthread et al

Back when C was first created, libc had very little functionality in it. Because of reasons, new functionality was added it went to its own library that you could then enable with a linker flag. Examples include -lm to add the math library and -ldl to get dlopen and friends. Similarly when threads appeared, each compiler had its own way of enabling them, and eventually any compiler not using -pthread died out.

If you look at the compiler flags in most projects there are a ton of gymnastics for adding all these flags not only to compiler flags but also to things like .pc files. And then there is code to take these flags out again when e.g. compiling on Visual Studio. And don't even get me started on related things like ltdl.

All of this is just pointless busywork. There is no reason all these could not be in libc proper and available and used always. It is unlikely that math libraries or threads are going to go away any time soon. In fact this has already been done in pretty much any library that is not glibc. VS has these by default, as does OSX, the BSDs and even alternative Linux libcs. The good thing is that Glibc maintainers are already in the process of doing this transition. Soon all of this pointless flag juggling will go away.

3. Get rid of 70s memory optimizations

Let's assume you are building an executable and that your project has two internal helper libraries. First you do this:

gcc -o myexe myexe.o lib1.a lib2.a

This gives you a linker error due to lib2 missing some symbols that are in lib1. To fix this you try:

gcc -o myexe myexe.o lib2.a lib1.a

But now you get missing symbols in lib1. The helper libraries have a circular dependency so you need to do this:

gcc -o myexe myexe.o lib1.a lib2.a lib1.a

Yes, you do need to define lib1 twice. The reason for this lies in the fact that in the 70s memory was limited. The linker goes through the libraries one by one. When it process a static library, it copies all symbols that are listed as missing and then throws away the rest. Thus if lib2 requires any symbol that myexe.o did not refer to, tough luck, all those symbols are gone. The only way to access them is to add lib1 to the linker line and have it processed in full for a second time.

This simple issue can be fixed by hand but things get more complicated if the come from external dependencies. The correct fix for this would be to change the linker to behave roughly like this:
  • Go through the entire linker line and find all libraries.
  • Look which point to same physical files and deduplicate them
  • Wrap all of these in a single -Wl,--start-group -Wl,--end-group
  • Do symbol lookup once in a global context
This is a fair bit of work and may cause some breakage. On the other hand we do know that this works because many linkers already do this, for example Visual Studio and LLVM's new lld linker.

tiistai 26. joulukuuta 2017

Creating an USB image that boots to a single GUI app from scratch

Every now and than you might want or need to create a custom Linux install that boots from a USB stick, starts a single GUI application and keeps running that until the user turns off the power. As an example at a former workplace I created an application for downloading firmware images from an internal server and flashing those. The idea there was that even non-technical people could walk up to the computer, plug in their device via USB and push a button to get it flashed.

Creating your own image based on latest stable turns out to be relatively straightforward, though there are a few pitfalls. The steps are roughly the following:
  1. Create a Debian boostrap install
  2. Add dependencies of your program and things like X, Network Manager etc
  3. Install your program
  4. Configure the system to automatically login root on boot
  5. Configure root to start X upon login (but only on virtual terminal 1)
  6. Create an .xinitrc to start your application upon X startup
Information on creating a bootable Debian live image can easily be found on the Internet. Unfortunately information on setting up the boot process is not as easy to find, but is instead scattered all over the place. A lot of documentation still refers to the sysvinit way of doing things that won't work with systemd. Rather than try to write yet another blog post on the subject I instead created a script to do all that automatically. The code is available in this Github repo. It's roughly 250 lines of Python.

Using it is simple: insert a fresh USB stick in the machine and see what device name it is assigned to. Let's assume it is /dev/sdd. Then run the installer:

sudo ./createimage.py /dev/sdd

Once the process is complete and you can boot any computer with the USB stick to see this:


This may not look like much but the text in the top left corner is in fact a PyGTK program. The entire thing fits in a 226 MB squashfs image and takes only a few minutes to create from scratch. Expanding the program to have the functionality you want is then straightforward. The Debian base image takes care of all the difficult things like hardware autodetection, network configuration and so on.

Problems and points of improvement

The biggest problem is that when booted like this the mouse cursor is invisible. I don't know why. All I could find were other people asking about the same issue but no answers. If someone knows how to fix this, patches are welcome.

The setup causes the root user to autologin on all virtual terminals, not just #1.

If you need to run stuff like PulseAudio or any other thing that requires a full session, you'll probably need to install a full DE session and use its kiosk mode.

This setup runs as root. This may be good. It may be bad. It depends on your use case.

For more complex apps you'd probably want to create a DEB package and use it to install dependencies rather than hardcoding the list in the script as is done currently.

lauantai 23. joulukuuta 2017

"A simple makefile" is a unicorn

Whenever there is a discussion online about the tools to build software, there is always That One Person that shows up and claims that all build tools are useless bloated junk and that you should "just write a simple Makefile" because that is lean, efficient, portable and does everything anyone could ever want.

Like every sentence that has the word "just", this is at best horribly simplistic but mostly plain wrong. Let's dive in more detail into this. If you look up simple Makefiles on the Internet, you might find something like this page. It starts with a very simple (but useless) Makefile and eventually improves it to this:

IDIR =../include
CC=gcc
CFLAGS=-I$(IDIR)

ODIR=obj
LDIR =../lib

LIBS=-lm

_DEPS = hellomake.h
DEPS = $(patsubst %,$(IDIR)/%,$(_DEPS))

_OBJ = hellomake.o hellofunc.o 
OBJ = $(patsubst %,$(ODIR)/%,$(_OBJ))


$(ODIR)/%.o: %.c $(DEPS)
$(CC) -c -o $@ $< $(CFLAGS)

hellomake: $(OBJ)
gcc -o $@ $^ $(CFLAGS) $(LIBS)

.PHONY: clean

clean:
rm -f $(ODIR)/*.o *~ core $(INCDIR)/*~ 

Calling this "simple" is a bit of a stretch. This snippet contains four different kinds of magic expansion variables, calls three external commands (two of which are gcc, just with different ways) and one Make's internal command (bonus question: is patsubst a GNU extension or is it available in BSD Make? what about NMake?) and requires the understanding of shell syntax. It is arguable whether this could be called "simple", especially for newcomers. But even so, this is completely broken and unreliable.

As an example, if you change any header files used by the sources, the system will not rebuild the targets. To fix these issues you need to write more Make. Maybe something like this example, described as A Super-Simple Makefile for Medium-Sized C/C++ Projects:

TARGET_EXEC ?= a.out

BUILD_DIR ?= ./build
SRC_DIRS ?= ./src

SRCS := $(shell find $(SRC_DIRS) -name *.cpp -or -name *.c -or -name *.s)
OBJS := $(SRCS:%=$(BUILD_DIR)/%.o)
DEPS := $(OBJS:.o=.d)

INC_DIRS := $(shell find $(SRC_DIRS) -type d)
INC_FLAGS := $(addprefix -I,$(INC_DIRS))

CPPFLAGS ?= $(INC_FLAGS) -MMD -MP

$(BUILD_DIR)/$(TARGET_EXEC): $(OBJS)
$(CC) $(OBJS) -o $@ $(LDFLAGS)

# assembly
$(BUILD_DIR)/%.s.o: %.s
$(MKDIR_P) $(dir $@)
$(AS) $(ASFLAGS) -c $< -o $@

# c source
$(BUILD_DIR)/%.c.o: %.c
$(MKDIR_P) $(dir $@)
$(CC) $(CPPFLAGS) $(CFLAGS) -c $< -o $@

# c++ source
$(BUILD_DIR)/%.cpp.o: %.cpp
$(MKDIR_P) $(dir $@)
$(CXX) $(CPPFLAGS) $(CXXFLAGS) -c $< -o $@


.PHONY: clean

clean:
$(RM) -r $(BUILD_DIR)

-include $(DEPS)

MKDIR_P ?= mkdir -p

It's unclear what the appropriate word to describe this thing is, but simple would not be at the top of the list for many people.

Even this improved version is broken and unreliable. The biggest issue is that changing compiler flags does not cause a recompile, only timestamps do. This is a common reason for silent build failures. It also does not provide for any way to configure the build depending on the OS in use. Other missing pieces that should be considered entry level features for build systems include:

  • No support for multiple build types (debug, optimized), changing build settings requires editing the Makefile
  • Output directory is hardcoded, you can't have many build directories with different setups
  • No install support
  • Does not work with Visual Studio
  • No unit testing support
  • No support for sanitizers apart from manually adding compiler arguments
  • No support for building shared libraries, apart from manually adding compiler arguments (remember to add -shared in your object file compile args ... or was it on link args ... or was it -fPIC)
  • No support for building static libraries at all
  • And so on and so on

As an example of a slightly more advanced feature, cross compilation is not supported at all.

These are all things you can add to this supposedly super simple Makefile, but the result will be a multi-hundred (thousand?) line monster of non-simplicityness.

Conclusions

Simple makefiles are a unicorn. A myth. They are figments of imagination that have not existed, do not exist and will never exist. Every single case of a supposedly simple Makefile has turned out to be a mule with a carrot glued to its forehead. The time has come to let this myth finally die.

keskiviikko 6. joulukuuta 2017

Comparing C, C++ and D performance with a real world project

Some time ago I wrote a blog post comparing the real world performance of C and C++ by converting Pkg-config from C to C++ and measuring the resulting binaries. This time we ported it to D and running the same tests.

Some caveats

I got comments that the C++ port was not "idiomatic C++". This is a valid argument but also kind of the point of the test. It aimed to test the behavior of ported code, not greenfield rewrites. This D version is even more unidiomatic, mostly because this is the first non-trivial D project I have ever done. An experienced D developer could probably do many of the things much better than what is there currently. In fact, there are parts of the code I would do differently based solely on the things I learned as the project progressed.

The code is available in this Github repo. If you wish to use something else than GDC, you probably need to tweak the compiler flags a bit. It also does not pass the full test suite. Once the code was in good enough condition to pass the Gtk+ test needed to get the results on this post, motivation to keep working on it dropped a fair bit.

The results

The result array is the same as in the original post, but the values for C++ using stdlibc++ have been replaced with corresponding measurements from GDC.

                                    GDC   C++ libc++       C

Optimized exe size                364kB        153kB    47kB
minsize exe size                  452kB        141kB    43kB
3rd party dep size                    0            0   1.5MB
compile time                       3.9s         3.3s    0.1s
run time                          0.10s       0.005s  0.004s
lines of code                      3249         3385    3388
memory allocations                  151         8571    5549
Explicit deallocation calls           0            0      79
memory leaks                          7            0   >1000
peak memory consumption            48.8kB         53kB    56kB

Here we see that code size is not D's strong suite. As an extra bit of strangeness the size optimized binary took noticeably more space than the regular one.  Compile times are also unexpectedly long given that D is generally known for its fast compile times. During development GDC felt really snappy, though, printing error messages on invalid code almost immediately. This would indicate that the slowdown is coming from GDC's optimization and code generation passes.

The code base is the smallest of the three but not by a huge margin. D's execution time is the largest of the three but most of that is probably due to runtime setup costs, which are amplified in a small program like this.

Memory consumption is where things get interesting. D uses a garbage collector by default whereas C and C++ don't, requiring explicit deallocation either manually or with RAII instead. The difference is clear in the number of allocations done by each language. Both C and C++ have allocation counts in the thousands whereas D only does 151 of them. Even more amazingly it manages to beat the competition by using the least amount of memory of any of the tested languages.

Memory graphs

A massif graph for the C++ program looked like this:


This looks like a typical manual memory management graph with steadily increasing memory consumption until the program is finished with its task and shuts down. In comparison D looks like the following:


D's usage of a garbage collector is readily apparent here. It allocates a big chunk up front and keeps using it until the end of the program. In this particular case we see that the original chunk was big enough for the whole workload so it did not need to grow the size of the memory pool. The small jitter in memory consumption is probably due to things such as file IO and work memory needed by the runtime.

The conversion and D as a language

The original blog posted mentioned that converting the C program to C++ was straightforward because you could change things in very small steps (including individual items in structs) while keeping the entire test suite running for the entire time. The D conversion was the exact opposite.

It started from the C++ one and once the files were renamed to D, nothing worked until all of the code was proper D. This meant staring at compiler failure messages and fixing issues until they went away (which took several weeks of work every now and then when free time presented itself) and then fixing all of the bugs that were introduced by the fixes. A person proficient in D could probably have done the whole thing from scratch in a fraction of the time.

As a language D is a slightly weird experience. Parts of it are really nice such as the way it does arrays and dictionaries. Much of it feels like a fast, typed version of Python, but other things are less ergonomic. For example you can do if(item in con) for dictionaries but not for arrays (presumably due to the potential for O(n) iterations).

Perhaps the biggest stepping stone is the documentation. There are nice beginner tutorials  but intermediate level documentation seems to be scarce or possibly it's just hard to bing for. The reference documentation seems to be written by experts for other experts as tersely as possible. For comparison Python's reference documentation is both thorough and accessible in comparison. Similarly the IDE situation is unoptimal, as there are no IDEs in Ubuntu repositories and the Eclipse one I used was no longer maintained and fairly buggy (any programming environment that does not have one button go-to-definition and reliably working ctrl+space is DOA, sorry).

Overall though once you get D running it is nice. Do try it out if you haven't done so yet.

perjantai 10. marraskuuta 2017

Aiming for C++ sorting speed with a plain C API

A well known performance measurement result is that C++ standard library's std::sort function is a lot faster than C library's equivalent qsort. Most people, when they first hear of this, very strongly claim that this is not possible, C is just as fast (if not faster) than C++, this is a measurement error, the sorting algorithms used are different and so on. Then they run the experiment themselves and find that C++ is indeed faster.

The reason for this has nothing to do with how the sorting function is implemented but everything to do with the API. The C API for sorting, as described in the man pages looks like this:

void qsort(void *base, size_t nmemb, size_t size,
           int (*compar)(const void *, const void *));

The interesting point here is the last argument, which is a function pointer to a comparison function. Because of this, the sort implementation can not inline this function call but must instead always call the comparator function, which translates to an indirect jump.

In C++ the sort function is a template. Because of this the comparison function can be inlined in the implementation. This turns out to have a massive performance difference (details below). The only way to emulate this in plain C would be to ship the sort function as a preprocessor monster thingy that people could then use in their own code. This leads to awful and hard to maintain code, so this is usually not done. It would be nice to be able to provide a similar fast sort performance with a stable plain C API but due to the way shared libraries work, it's just not possible.

So let's do it by cheating.

If we know that a compiler is available during program execution, we can implement a hybrid solution that achieves this. Basically we emulate how JIT compilers work. All we need is something like this:

sorter opt_func = build_sorter("int",
    "(const int a, const int b) { return a < b; }");
(*opt_func)(num_array, NUM_INTS);

Here build_sorter is a function that takes as arguments the type of the item being sorted, and a sorting function as source code. Then the function calls into the external compiler to create an optimised sorting function and returns that via a function pointer that can be called to do the actual sort.

Full source code is available in this Github repo. Performance measurements for 100 million integers are as follows.

C++ is almost twice as fast as plain C. On the fly code generation is only 0.3 seconds slower than C++, which is the amount of time it takes to compile the optimised version. Codegen uses the C++ sorting function internally, so this result is expected.

Thus we find that it is possible to provide C++ level performance with a plain C API, but it requires the ability to generate code at runtime.

Extra bonus

During testing it was discovered that for whatever reason the C++ compiler (I used GCC) is not able to inline free functions as well as lambdas. That is, this declaration:

std::sort(begin, end, sorting_function);

generates slower code than this one:

std::sort(begin, end, [](sorting_lambda_here));

even though the contents of both comparison functions is exactly the same (basically return a < b). This is the reason the source code for the sort function above is missing the function preamble.

sunnuntai 3. syyskuuta 2017

Comparing C and C++ usage and performance with a real world project

The relative performance of C and C++ is the stuff of folk legends and Very Strong Opinions. There are microbenchmarks that can prove differences in performance in any direction one could wish for, but, as always, they are not conclusive in any way. For an actual comparison you'd need to have a complete, non-trivial program in one language, translate it to the other one without doing any functional changes and then comparing the results. The problem here is that this sort of a conversion does not exist.

So I made one myself.

I took the well known pkg-config program which is written in plain C using GLib and converted it to C++. The original code is available at Freedesktop and the converted version is on Github (branches master and cpp). The C++ version does not have any dependencies outside of C++ standard library whereas the C version depends on GLib and by extension pcre (which is an internal dependency of Glib, pkg-config does not use regular expressions).

All tests were run on Ubuntu 1704. The C++ version was tested both with GCC/stdlibc++ and Clang/libc++. Measurements were done with the gtk+ test in pkg-config's test suite.

The results in a single array

                           C++ stlibc++   C++ libc++       C

Optimized exe size                180kB        153kB    47kB
minsize exe size                  100kB        141kB    43kB
3rd party dep size                    0            0   1.5MB
compile time                       3.9s         3.3s    0.1s
run time                          0.01s       0.005s  0.004s
lines of code                      3385         3385    3388
memory allocations                 9592         8571    5549
Explicit deallocation calls           0            0      79
memory leaks                          0            0   >1000
peak memory consumption           136kB         53kB    56kB

Binary sizes and builds

The first thing to note is that the C version is a lot smaller than the corresponding C++ executables. However if you factor in the size of the external third party dependency binaries, i.e. the shared libraries of GLib and pcre, the C version is an order of magnitude bigger. One could argue endlessly what is the correct way to calculate these sizes (because a system provided library is shared among many users) but we're not going to do that here.

C++ is known for its slow build times and that is also apparent here. Again it should be noted that compiling the C dependencies takes several minutes so if you are on a platform where dependencies are built from source, the C version is a lot slower.

The runtime is fast for all three versions. This is expected because pkg-config is a fairly simple program. Stdlibc++ is slower than the other two, whose runtime is within measurement error of each other.

Memory

Memory and resource management has traditionally been the problem of C, where the programmer is responsible for shepherding and freeing every single resource. This can be clearly seen in the result table above. Perhaps the most striking fact is that there are 79 explicit (that is, written and maintained by the developer) resource release calls. That means that more than 2% of all statements in the entire code base are resource deallocation calls.

Every manual resource deallocation call is a potential bug. This is confirmed by the number of memory leaks as reported by Valgrind. There are more than 1000 of them, several dozen of which are marked as "definitely lost". The C++ implementation on the other hand uses value types such as std::string and RAII consistently. Every resource is deallocated automatically by the compiler, which, as we can see, does it perfectly. There are no resource leaks.

Memory consumption is also interesting. The C version works by creating an array of package objects and strings. Then it creates a hash table with pointers that point to said array. This is the classical C "sea of aliased pointers" problem, where the developer must keep track of the origin and meaning of every single pointer with no help from the compiler.

The C++ version has no pointers but instead uses value types. This means that all data is stored twice: once in the array and a second time in the hash table. This could probably be optimized away but was left as is for the purposes of this experiment. Even with this duplication we find that the version using libc++ uses less memory than the Glib one. Stdlibc++ uses a fair bit more memory than the other two. To see why, let's look at some Massif graphs starting with stdlibc++.


This shows that for some reason stdlibc++ allocates one chunk of 70 kB during startup. If we ignore this allocation the memory consumption is about 60 kB which is roughly the same as for the other two executables.

Plain C looks like this.


The most notable thing here is that Massif can not tell the difference between different allocation sources but instead lumps everything under g_malloc0. The C++ version shows allocations per container type which is extremely useful.

Finally, here is the chart for libc++.



Libc++ does not have an initial allocation like stdlibc++, so its memory usage is lower. Its containers also seem to be more optimized, so it uses less memory overall. Memory consumption could probably be reduced by using a linear probing hash map (which is also what Glib does internally) rather than the node-based one as required by the C++ standard but it would mean having an external dependency which we want to avoid.

The conversion job

One of the many talking points of Rust is that converting C to it is easy. This is spoken of in quotes such as "Rust is the only language that allows you to convert existing C code into a memory safe language piece by piece" (link to original purposefully omitted to protect the innocent). Depending on your definition of a "memory safe language" this statement is either true or complete bunk.

If you are of the opinion that Rust is the only memory safe language then the statement is obviously true.

If not then this statement is fairly vacuous. Every programming language that has support for plain C ABI and calling conventions, which is to say almost every one of them, has supported transitioning from C code one function at a time. Pascal, D, Java with JNI, even Fortran have been capable of doing this for decades.

C++ can also do this but it goes even further: it supports replacing C structures one element at a time. Pkg-config had many structs which consisted of things like GLists of char pointers. In any other programming languages changing this element means converting the entire struct from C into your new language in a single step. This means changing all code that uses said struct into the new language in one commit, which is usually huge and touches a large fraction of the code base.

In C++ you can convert only a fraction of the struct, such as replacing one of the stringlists with a std::vector<std::string>. Other elements of the struct can remain unchanged. This means smaller, more understandable commits. The extra bonus here is that these changes do not affect functionality in any way. There are no test suite regressions during the update process, even when working with frankenstein structs that are half C and half C++.

Following this train of thought to its final station yields slightly paradoxical results. If you have a legacy C code base that you want to convert to D or Rust or whatever, it might make sense to convert it to C++ first. This allows you to do the hard de-C-ification in smaller steps. The result is modern C++ with RAII and value types that is a lot simpler to convert to the final target language.

The only other programming language in common use that is capable of doing this is Objective C but it has the unfortunate downside of being Objective C.

Conclusions

Converting an existing C program into C++ can yield programs that are as fast, have fewer dependencies and consume less memory. The downsides include a slightly bigger executable and slower compilation times.

tiistai 29. elokuuta 2017

Dependencies, why are they so hard to get right?

The life cycle of any programming project looks roughly like this.


We start with the plain source code. It gets processed by the build system to produce so called "build tree artifacts". These are executables, libraries and the like but they are slightly special. They are stored inside the build tree and can not usually be used directly. Every build system has its own special magic sprinkled in the outputs. The files inside a build tree can not be run directly (usually) and the file system layout can be anything. The build tree is each build system's internal implementation detail, which is usually not documented and definitely not stable. The only thing that can reliably operate on items in the build directory is the build system itself.

The final stage is the "staging directory" which usually is the system tree, as an example /usr/lib in Unix machines but can be e.g. an app bundle dir on OSX or a standalone dir that is used to generate an MSI installer package on Windows. The important step here is installation. Conceptually it scrubs all traces of the build system's internal info and make the outputs conform to the standards of the current operating system.

The different dependency types

Based on this there are three different ways to obtain dependencies.


The first and simplest one is to take the source code of your dependency, put it inside your own project and pretend it is a native part of your project. Examples of this include the SQLite amalgamation file and some header-only C++ libraries. This way of obtaining dependencies is not generally recommended or interesting so we'll ignore it for the remainder of this post.

Next we'll look into the final case. Dependencies that are installed on the system are relatively easy to use as they are guaranteed to exist before any compilation steps are undertaken and they don't change during build steps. The most important thing to note here is that these dependencies must provide their own usage information in a build system independent format that is preferably fully declarative. The most widely accepted solution here is pkg-config but there can be others, as long as it is fully build system independent.

Which leaves us the middle case: build system internal dependencies. There are many implementations of this ranging from Meson subprojects to CMake internal projects and many new languages such as D and Rust which insist on compiling all dependencies by themselves all the time. This is where things get complicated.

Since the internal state of build trees are different, it is easy to see that you can not mix two different build systems within one single build tree. Or, rather, you could but it would require one of them to be in charge and the other one to do all of the following:
  • conform to the file layout of the master project
  • conform to the file format internals of the master project (which, if you remember, are undocumented and unstable)
  • export full information about what it generates, where and how to the master project in a fully documented format
  • accept dependency information for any dependency built by the master project in a standardized format
And there's a bunch more. If you go to any build system developer and tell them to add these features to their system they will first laugh at you and tell you that it will happen absolutely never.

This is totally understandable. Pairing together the output of two wildly different unstable interfaces in a reliable way is not fun or often even possible. But it gets worse.

Lucy in the Sky with Diamond Dependency Graphs

Suppose that your dependency graph looks like this.

The main program uses two libraries libbaz and libbob. Each one of them builds with a different build system each of which has its own package manager functionality. They both depend on a common library libfoo. As an example libbob might be a language wrapper for libfoo whereas libbaz only uses it internally. It is crucially important that the combined project has one, and only one, copy of libfoo and it must be shared by both dependents. Duplicate dependencies lead, at best, into link time errors and at worst to ten hour debugging sessions of madness in production.

The question then becomes: who should build libfoo? If it is provided as a system dependency this is not an issue but for build tree dependencies things break horribly. Each package manager will most likely insist on compiling all their own dependencies (in their own special format) and plain refuse to work anything else. What if we want the main program to build libfoo instead (as it is the one in charge)? This quagmire is the main reason why certain language advocates' view of "just call into our build tool [which does not support any way of injecting external dependency information] from your build tool and things will work" ultimately unworkable.

What have we learned?

  1. Everything is terrible and broken.
  2. Every project must provide a completely build system agnostic way of declaring how it is to be used when it is provided as a system dependency.
  3. Every build system must support reading said dependency information.
  4. Mixing multiple build systems in a single build directory is madness.