#include <name>, the library inside the binary

mc carries its own source inside the executable, compressed, and its standard library in a tree beside it — lib/mc/v<version>/, what a release tarball unpacks next to the binary. Both are served through the angle-bracket form of #include, and a program cannot tell which of the two answered: no checkout, no include path, no install step.

Since M52 the blob holds 60 of the manifest's 101 rows — every src/ path, plus prelude and user_default, which are the compiler's own #includes. The other 41 — the libraries, the teaching fixtures, the two #embed test payloads — travel in the tree (§ The catalogue says which is which). The rule is the owner's: a row stays inside the binary iff the compiler itself is built from it.

// expect-exit: 0
// expect-stdout: 42
#include <sys>
#include <prelude>

i64 main() {
    i64 n = 0;
    while (n < 42) { n = n + 1; }
    putnum(n);
    write(1, "\n", 1);
    return 0;
}

Every <mc/*> name column 1 of tools/bundle.list carries — a library or a compiler PART alike — is in the recorded public surface, as the bundle kind (101 entries: § The catalogue); what a PATCH, a MINOR and a MAJOR may do to one is hooks.md § 8.

<name> is served by the bundle, by the library tree, or by a package this project locked — or it is an error — there is no fallback to the working directory, on purpose: <name> means "a library that is not in my tree", and the answer must be a function of (this binary, this project's lock, the installed packages) and of nothing else.

$ mc prog.mc -o prog.o
prog.mc:1: unknown bundled include: no/such/module

A name the blob does not carry and no root answers reports the road instead, because the answer is provisioning and not spelling:

$ mc prog.mc -o prog.o
prog.mc:1: #include <sys>: not in this compiler and mc 0.16.0's library tree was not found: run mc install

#include "path" is unchanged: the includer's own directory first, then each [include].paths root in order. Angle brackets are libraries, quotes are my files.

Since M44 the bundle is the SECOND of three places a <name> can come from. Before it comes a package mc.lock pins, which is why a project may override a bundled name — [deps] float = "1.3.0" makes <float> that tree instead of this blob, pinned by a content hash and re-checked on every build. After it comes the installed copy of the compiler's own package, which a binary that carries the blob never reaches. A project with no [deps] never leaves the blob, and its output is byte for byte what it was. See packages.md § 2.

The catalogue #

The manifest is tools/bundle.list, one NAME<TAB>PATH per line, sorted by name: 101 entries, plus mc/bundle_data, which is regenerated on demand (see below). Those are the names <...> accepts. The manifest keeps every row whichever side of the cut it is on — it is also the NAME<TAB>PATH map an installed tree and the tree beside the binary are read through, so a row that left the blob has to stay in it or the library is reachable by neither road.

Which side a row is on, and it is the path that decides:

Measured on macOS arm64, the same compiler with the 41 rows in the blob and out of it — __text and __cstring do not move, because a bundle row is data:

−132 098 B, −9.4% of the binary and −18.0% of __DATA,__data. The 41 rows are 256 KB of source on disk; the whole tree is what a -slim tarball has always needed and now carries.

So in the tables that follow, everything under lib/ except <prelude> and <user_default>, plus the two tests/mc/bundle/ payloads, comes from the tree — the system layers, <io>, <float> and its machines, <i128>, <u128>, <f16>, and every demonstration — while every <mc/...> name, <lz>, <prelude> and <user_default> are inside the binary. Nothing about the spelling changed, and no source in this repository or outside it had to move: #include <float> is #include <float> before and after.

The system layer — pick exactly one #

O_RDONLY/O_WRONLY/O_CREAT/O_TRUNC live in each system layer, not in <io>, because they are per-system values (O_CREAT is 0x200 on macOS, 0x40 on Linux and 0x100 in the Windows layer, where the flags are not passed to the system at all).

The language layer #

The compiler itself #

<mc/core> is the whole compiler minus exactly one function, void user_init(). Including it and supplying that function is a taught compiler (hooks.md).

<mc/version>, and why the version is in the bundle (M44) #

src/version.mc is one function, mc_version(), returning a string literal: 0.0.0-dev in the repository and the tag in a released binary, rewritten by scripts/set-version.sh in the release job and never committed. It is bundled for the same reason <mc/host> is resolved per binary: a taught compiler must report the version of the binary that built it. mc --exe over <mc/host> + <mc/core> + a module produces a compiler out of the blob, so if the blob's copy said 0.0.0-dev a release binary would build compilers that lie about their version — and, from M44's install step, look in the wrong ~/.mc/libs/mc/v<version>/. That is why scripts/set-version.sh's second act is make bundle, and why scripts/check-bundle.sh guards the sentinel. See ../ci.md § Versioning.

<mc/host>, the one name that is not an entry #

<mc/core> is host-neutral: it says nothing about posix_spawnp, the environment, the O_* values or which (os, arch) pair the binary is. That comes from a host file, and a compiler needs exactly one of them, included before the core.

<mc/host> is how a source asks for "the one that matches whichever mc is compiling me". It is resolved in src/core_bundle.mc (host_bundle_open) to host_include(), which each host file answers for itself — mc/host_macos, mc/host_linux_aarch64, mc/host_linux_x86_64, mc/host_windows_aarch64 or mc/host_windows_x86_64 — and the entry it lands on is what the once-only include list records, so writing <mc/host> and <mc/host_macos> in the same program includes the file once, not twice.

#include <mc/host>
#include <mc/core>

void user_init() { }

That is a complete compiler, and it is exactly what mc build generates for a [compiler] section (../build.md). See ../guide/90-linux-host.md for what the host layer answers.

<float> — f32 and f64 (M24) #

A LIBRARY, not a compiler feature. The stock mc has no floats: nothing here is in lib/user_default.mc, and a float program is built by a taught compiler the way examples/api and examples/lang are.

Both machines carry the two widths in one table and pick the single-precision row of an operation from the double one (fa_w on AArch64, fx_w2 on x86-64): the arithmetic pair, fneg/fabs/ fsqrt/fmov, the compares, the load, the store and the four conversions — scvtf, ucvtf, fcvtzs, fcvtzu. Which of each pair is emitted is decided by type_signed, not by the type id: a signed narrow integer (i32, i16, i8 — M45's TK_SINT) converts through scvtf/fcvtzs and an unsigned one (u8/u16/u32) through ucvtf/fcvtzu, so (f64)(i32)(-5) is -5.0 and not 4294967291.0. An indirect call returns a float when the cast says so, (f64) callp(p, x) (language.md § 7): MTASK_CALLP reads walk_ret_type(), which is the type of the call node.

The generality proofs (M24 step 2) #

Three modules the core has never heard of, each with an empty git diff src/.

What <f16> promises per architecture. It drives ONE machine — the arm64 table <float> registered — and the type it registers says so: TK_FLOAT where a half lives in a v register beside every other float, TK_INT where it does not, which is what a half with no arithmetic unit is: two bytes of storage the core moves. On the second road no intrinsic is registered at all and the module pushes f16_to_f32, f32_to_f16, ldf16 and stf16 as ordinary functions in a second source (p_push_source, the shape docs/specs/M24.md § Generality asks for) — the softfloat conversion, in integer arithmetic, rounding to nearest with ties to even and quietening a NaN exactly as fcvt does. It is pushed rather than left to the program as an include, the way <float_rt> is, because a program cannot include a file on one target and not on another.

The same source compiles and prints the same numbers either way; tests/wide/031-f16.mc is run on macos/aarch64, linux/aarch64 and linux/x86_64 and cross-compiled for both Windows architectures by make check-wide. What the fallback costs is four small functions in every object and four names a program may not define itself.

The road it decides on is [target].arch, which mc build has already read when user_init runs; with no [target], the host. --machine= and --backend= are read after user_init and cannot be seen from there, so mc --backend=elf-obj-x86_64 on an aarch64 host is refused by the machine instead — opcode 303 is not x86-64's: no machine claims it (machine.md § The opcode bands).

<float>, <f16> and <i128>/<u128> coexist in one compiler, in either registration order. Each derived machine claims a band of the opcode space they share -- <float> 100..199, <i128> 200..299, <f16> 300..399 -- and bounds it at both ends, so no module encodes another's instructions (machine.md § 3 is the registry). lib/mc_float_wide.mc and lib/mc_wide_float.mc are the two orders as compiler entry points, and tests/wide/035-coexist.mc is the program: same stdout, same exit code, same object from both.

Casting a narrow integer to i128/u128 sign-extends a signed source and zero-extends anything else, keyed on type_signed(src) and not on the id — so (i128)(i32) -5 and (u128)(i32) -5 both fill the high half with ones (C sign-extends a signed source whatever the wide target's signedness), while (i128)(u32) 0xffffffff leaves it zero. tests/wide/034-cast-narrow.mc checks every case bit-for-bit and runs on macos/aarch64, linux/aarch64, linux/x86_64 and — on the CI legs — windows/aarch64 and windows/x86_64.

Known limits of <i128>/<u128>, all deliberate at M24:

• At most four 16-byte arguments in one call on arm64. They take the eight argument registers in even pairs; a fifth is a clean i128/u128: too many arguments for the register pairs (a diagnostic, never a miscompile), and no spill-to-stack path exists.
• callp with a wide argument or a wide result is unsupported. Neither machine overrides MTASK_CALLP, so an indirect call falls to the base machine, which lays a 16-byte value out as if it were 8. Call a wide value only through a named function (MTASK_CALL, which the module does override).
• A decimal literal ≥ 2^128 wraps silently. iw_muladd accumulates the digits in four 32-bit limbs and drops the carry off the top; there is no overflow diagnostic on the literal.

The demonstrations #

Everything make check-surface wires up is in the manifest too, so the demos can be reproduced from a downloaded tarball with no checkout — out of the library tree beside the binary, since M52 (§ The catalogue):

The parts of the core (M41) #

<mc/core> is the sum of seven parts, each a bundled name of its own. A recreated compiler names the parts it wants and writes its own main(); src/core.mc is literally those seven plus src/main.mc, and scripts/check-parts.sh compiles both spellings and cmps the two objects, so this table cannot drift from the code.

<mc/core_pkg> gained src/install.mc with M44 step 4: mc install, the subcommand that puts the mc package itself under <libs> — which is where a compiler assembled WITHOUT <mc/core_bundle> reads every <name> from (§ The slim flavour below) — and src/upgrade.mc with step 5: mc upgrade, which replaces the BINARY from a release and then spawns the new one to install the tree that matches it (cli.md § 3f).

The two files M41 split out are bundled under their own names too: <mc/objmodel> (src/objmodel.mc, the section/symbol/relocation model every writer reads) and <mc/cli> (src/cli.mc, mc_main()). <mc/macho> is now the Mach-O writer alone.

<mc/core_sandbox> (M43) needs nothing but <mc/core_min> and the host file: every system call it issues goes through host_syscall6() and is named by an SN_* index of <mc/sysno>, whose per-architecture number tables are <mc/sysno_linux_aarch64> and <mc/sysno_linux_x86_64> — the two files that also carry sys6, the raw shim. Those three are bundled because a Linux host file includes one of them, and mc build writes #include <mc/host> for a taught compiler.

Spelled out, the whole compiler is:

#include <mc/host>
#include <mc/core_min>
#include <mc/core_machines>
#include <mc/core_writers>
#include <mc/core_build>
#include <mc/core_bundle>
#include <mc/core_pkg>
#include <mc/core_sandbox>
#include <mc/main>
#include <user_default>

and docs/guide/98-recreating-the-compiler.md is what each omitted line costs, in bytes and in capability.

<mc/core_pkg> (M44) is the one part that names another: it #includes core_build.mc, because resolving a dependency needs the driver that reads mc.toml. The split is deliberate and it is where the milestone's debloat lives — the READ side of packages ([deps], mc.lock, the tree hash, #include <pack/file.mc>, the refusals) is <mc/core_build>'s src/deps.mc, so a compiler WITHOUT <mc/core_pkg> still builds a project from its lock and its deps/ tree. That compiler has no fetcher, no registry, no MVS and no lock writer at all, and it advertises neither mc pkg nor mc update: the usage IS the subcommand table. tests/pkg/nopkg.mc is that compiler, and scripts/check-pkg.sh builds the vendored fixture project with it.

A part stands on <mc/core_min> alone, and scripts/check-parts.sh compiles each of the six optional parts on top of the minimal one to say so. It is not free: four names had to move when M41 landed, because the full assembly hides a cross-part dependency completely. tm_cat and tm_num_str went from src/toml.mc to src/arena.mc (src/cli.mc needs the first for --include= and src/backend_coff.mc the second for a long section name); MODE_755 went from src/backend_exe.mc to src/arena.mc (src/driver.mc uses it for mkdir -p); and R_X86_PC32/R_X86_PLT32 went from src/machine_x86_64.mc to src/objmodel.mc, which is where a relocation kind the machine emits and the writer maps belongs. None of the four changed a byte of generated code — they are a #define and three leaf functions.

The naming rule. A bundle name's last path component is the file's basename, because a relative #include inside a bundled file resolves by joining and then, failing that, by last component (see below), and tools/bundle.mc refuses a manifest where two entries share one. That is why the parts are core_min and not core-min: src/core_min.mc cannot be reached as core-min, so a hyphenated bundle name would be a name nothing inside the bundle could include.

The slim flavour (M44 step 4) #

A release carries two binaries per target: mc, the full one, and mc-slim, the same compiler with the blob left out — mc-<ver>-<target>-slim.tar.gz, where the file inside is still called mc. Measured here (macOS arm64, scripts/check-parts.sh): 534 179 bytes against 1 225 843, 44% of the full binary, and its __DATA,__data is 8 608 bytes against 605 104.

It is an assembly and not a part: src/core_slim.mc, bundled as <mc/core_slim>, is the five parts a compiler needs to compile anything, and src/main_slim.mc (<mc/main_slim>) is the main() that matches. Spelled out:

#include <mc/host>
#include <mc/core_min>
#include <mc/core_machines>
#include <mc/core_writers>
#include <mc/core_build>
#include <mc/core_pkg>
#include <mc/main_slim>
#include <user_default>

Two parts are missing. <mc/core_bundle> is the point of the flavour. <mc/core_sandbox> goes with it because mc sandbox is a Linux supervisor and not part of compiling anything; a build that wants it back adds the include and the mc_sandbox_init() call, and nothing else changes.

Since M52 both flavours ship the library beside the binary as well — lib/mc/v<version>/ in every tarball, which is resolution root 2 (packages.md § 2). For the slim flavour that is the larger change: untarring it gives a working #include <prelude> with no mc install at all, which it never had before. For the full one it is what keeps the promise when a name leaves the blob.

What a slim binary does with nothing installed AND no tree beside it: compile a program with no <...> include, every --dump-*, mc --host, mc --version, mc build of a project whose sources use no bundled name, mc limits, mc sysroot, mc pkg — and mc install itself. What it does after mc install (cli.md § 3e) is everything the full binary does, out of <libs>/mc/v<version>/ instead of out of its own __data: scripts/check-pkg.sh § 32 compiles #include <mc/host> + <mc/core> + <user_default> with it and gets an object byte for byte identical to the full compiler's src/mc.mc.

src/mc_slim.mc is the checked-in entry (host layer + core_slim.mc + user.mc), with one sibling per cross target — src/mc_linux_slim.mc, src/mc_linux_x86_64_slim.mc, src/mc_windows_slim.mc, src/mc_windows_x86_64_slim.mc — and one *-slim-obj.toml config each, exactly as the full flavour has. make mc-slim builds build/mc-slim on the host.

Your own bundle #

A debloated compiler can keep #include <name> with its own, much smaller library, at zero core lines. tools/bundle.mc is four includes — <mc/arena>, <lz>, <mc/bundle_data>, <mc/bundle> — over a manifest of NAME<TAB>PATH lines, and all four are bundled names. So:

1. write the same four-line tool against those names and build it with mc --exe;
2. run it over your own manifest to generate your own bundle_data.mc;
3. have your compiler include that file plus <mc/bundle> instead of <mc/core_bundle>, and call lex_set_bundle(&bundle_open) from its main().

src/bundle.mc needs nothing but BUNDLE_COUNT and the two arrays the generator emits. The one thing you give up is <mc/host>: that name is resolved by host_bundle_open in src/core_bundle.mc, which is four lines you copy if you want it.

Relative includes inside a bundled file #

The bundle is flat: mc/lex is one name, not a directory and a file. But src/core.mc still says #include "arena.mc" and src/driver.mc still says #include "../lib/prelude.mc", and both must keep working when the including file is itself bundled. So the lexer joins and normalises the name the usual way, drops a trailing .mc, and looks the result up; if that misses, it retries with the last path component.

tools/bundle.mc refuses a manifest in which two entries share a last component, so that fallback can never be ambiguous.

A bundled file is pushed onto the same #include stack a real one uses, with the bundled name in place of a path — which is what a diagnostic then shows:

syntax_demo_test:10: type expected at top level

mc/bundle_data, the file the bundle cannot contain #

src/core.mc includes bundle_data.mc, and src/bundle_data.mc is the bundle. It cannot be inside itself: its own bytes would change the bytes it contains. So it is the one name that is regenerated on demand — bundle_find("mc/bundle_data") answers with index BUNDLE_COUNT, and the file is written out again from the blob and index already in memory, by the very same bundle_emit that tools/bundle.mc uses. One definition of the format, so the two cannot drift.

Since M21.5 bundle_emit has two modes, and the difference is arena, not format:

An array initializer costs one AST node per element, so spelling the 180 KB blob out cost ~22 500 nodes — 2.3 MB of arena — in every taught compiler that includes <mc/core>. As #embed it costs exactly one node (N_BLOB). mc/bundle.bin is therefore the second name that is not in the blob (index BUNDLE_BIN = BUNDLE_COUNT + 1): it is bundle_blob, served straight out of the compiler's own data with no inflate and no copy, rounded up to a multiple of 8 so both forms declare a global of the same size.

The disk copy keeps the u64 form because the frozen stage0/lex.c has no embed in its dir_names[] and build/mc0 src/mc.mc is the seed step of make mc1. Both forms produce the same object, which is what makes the split safe — and what check-standalone measures.

scripts/check-standalone.sh proves the consequence in the strongest form available: a compiler built from #include <mc/core> + #include <user_default>, in an empty directory, compiles src/mc.mc into an object byte for byte identical to build/mc2.o.

The format, and regenerating it #

src/bundle_data.mc is generated source, checked in. It holds one blob — the NUL-terminated names first, in manifest order, then each LZ77 stream — and one flat index with four values per entry (name offset, stream offset, compressed size, real size). Names live in the blob rather than as string literals because every literal counts against the C seed's budget, and the index is one array rather than four for the same reason.

make bundle          # regenerate src/bundle_data.mc from tools/bundle.list
make check-bundle    # prove the checked-in copy is exactly what comes out

make bundle is the only way that file is ever written. Run it whenever a lib/*.mc or a core module changes, before make bootstrap — make check runs check-bundle first precisely so that a stale bundle fails with a message naming make bundle, instead of failing later as a mysterious fixed-point difference.

The bundle is inside the fixed point: mc1 and mc2 both carry it, and the objects they produce must still be identical. See ../guide/70-bootstrap.md.

#embed and the bundle #

#embed NAME "path" [lz] (directives.md) reads its payload the same way its includer was read: from disk for a real file, and from the bundle when the directive was written inside a bundled <name> include. <embed_demo> and <embed_demo.txt> are in the manifest exactly so that this path is exercised from a binary with no checkout.

Edit this page