Communication Between the Compiler, the Build System, and Beyond

Moving packages out of the operating system and effectively into language specific silos is a trade off. Go executables are statically linked:

  - they're larger
  - to fix a security bug you have to recompile them, you can't just update the  relevant library 
  - a program not written in go can't use a go library

We find these tradeoffs useful because distributions don't always stay close enough to the bleeding edge for what we're doing and often we're using machines where we have no permission to install system packages or update them. So we derive great value from effectively having a private packaging system.

On the other hand a build system that only does silos, assumes some packaging system and is monolingual has limited applicability.

I basically disagree with all of that, even with the premise: My C projects are already extremely fast to build. I also don't agree we should give up modularity for whole program optimization, I do not think we should give up dynamic (edit) linking, I do not think everything needs to be entangled to achieve fast builds...

The build times we see affecting C++ and Rust are because the type systems of languages is misdesigned. The supply-chain issues we have with the language-level packaging manager downloading source instead of pre-compiled binaries from curated repositories are also related to this. I think we need leaner, simpler, and better modularized software. Not whole-program optimization which is only necessary because of the monomorphized gigantic intermediate results created for badly designed languages, which then needs to be reduced again in a costly process.

Fwiw on the compiler front, I used to work on the msvc tool chain. It has been doing lto (aka ltcg) for a very long time at reasonably good throughput. That tool chain also has done incremental compilation on a per function basis for about 7 years. Sharing header parsing across translation units has been possible through PCH for decades. Even without PCH, inline header functions only get codegened once in ltcg mode (not counting inline expansion).

In a large multi-dll build like Windows or Office, the import/export information across modules is computed without doing codegen, so ltcg codegen can be done in parallel regardless of the module dependency graph.

I never worked deeply in the build systems for Linux based OSes or other unixes, but I gather that some symbol visibility choices make the build situation worse in Unix systems than Windows.