Configuration

The config package (@config) loads configuration for a MoonBit application. It reads ordered sources, parses files with the formats that you register, and merges everything into one value tree. The package is pure: it never reads a file, the process environment, or the working directory itself. A host gives it those inputs, so a load gives the same result on wasm, wasm-gc, js, and native.

Import moonrockz/moon-lander/config as @config and a format package such as moonrockz/moon-lander/config/formats/toml as @toml. Each mbt check block on this page runs as a test with mise run test.

One load, start to finish

An application describes its sources with a builder. The pure load reads only the builder and a file tree. When the tree does not hold a file that a source needs, load asks for it. The host reads the files and calls load again.

The smallest program

Sources that hold values need no files. build loads them over an empty, complete file tree. A later source wins.

///|
test "the smallest program" {
  let config = @config.build([
    @config.in_memory_pairs([("app.name", @config.string("demo"))]),
    @config.in_memory_pairs([("app.name", @config.string("acme"))]),
  ])
  @expect.expect(@config.get_text(config, "app.name")).to_equal("acme")
}

Sources and the builder

Each source has a kind, a display name, a display priority, and an input. Add them to the builder in merge order: low first, high last. Every function returns a new builder, so you can keep a base builder and extend it.

///|
test "describe sources with a builder" {
  let base = @config.builder()
    |> @config.format(@toml.format())
    |> @config.defaults({ "server": { "port": 8080, "host": "localhost" } })
  let builder = base
    |> @config.file("/srv/app.toml")
    |> @config.add(
      @config.in_memory_pairs([("server.port", @config.int(9000))]),
    )
  let tree = @config.FileTree::new(complete=true).with_file(
    "/srv/app.toml",
    (
      #|[server]
      #|host = "example.org"
    ),
  )
  guard @config.load(builder, tree) is Done(config) else {
    fail("a complete tree never asks for files")
  }
  @expect.expect(@config.get_text(config, "server.host")).to_equal(
    "example.org",
  )
  @expect.expect(@config.get_text(config, "server.port")).to_equal("9000")
}

The layer stack

Merge order decides which value wins. Each source also has a priority that reports show. When you give no priority, it comes from the kind and the file role. An in-memory source takes the priority of the source before it, so it fits at any position.

1. Overrides (700)command-line values
2. Environment variables (600)APP_SERVER__PORT=9000
3. User override file (400)app.user.toml
4. Local override file (390)app.local.toml
5. Environment file (375)app.Staging.toml
6. Workspace member file (350)pkgs/api/app.toml
7. Workspace, explicit, or legacy file (300)app.toml
8. Global user file (200)~/.config/app/
9. System file (100)/etc/app/
10. Defaults (0)compiled in

Added in this order, the sources merge from the bottom up, so the top layer wins. A priority that is lower than the one before it is an error, because the list order and the reports would disagree. Equal priorities are allowed.

///|
test "a decreasing priority is an error" {
  let builder = @config.builder()
    |> @config.add(@config.ConfigSource::skipped(Environment))
    |> @config.file("/srv/app.toml", priority=300)
  try @config.load(builder, @config.FileTree::new(complete=true)) catch {
    SourceOrder(previous~, previous_priority~, ..) =>
      @expect.expect((previous, previous_priority)).to_equal((
        "environment", 600,
      ))
    _ => fail("expected a source-order error")
  } noraise {
    _ => fail("expected a source-order error")
  }
}

Files and the request loop

A FileTree is an immutable snapshot of what the host has read: directory listings, file text, and absence records. A path is held, missing, or unknown. load asks only for unknown paths, and it asks for all of them in one step. In a complete tree every path that the tree does not hold is missing, so load never asks.

A host runs this loop. Native and js hosts read the disk; a wasm host gives a complete tree. The loop below answers from a map, so it runs on every target.

///|
/// Answer one request from `disk`, a map from file path to text. A directory
/// exists when a file path in `disk` is below it.
fn answer(
  tree : @config.FileTree,
  request : @config.FileRequest,
  disk : Map[String, String],
) -> @config.FileTree {
  match request {
    ReadFile(path) =>
      match disk.get(path) {
        Some(text) => tree.with_file(path, text)
        None => tree.with_missing(path)
      }
    ListDir(dir) => {
      let entries : Array[@config.DirEntry] = []
      for path, _ in disk {
        guard path.has_prefix(dir + "/") else { continue }
        let rest = path[dir.length() + 1:].to_owned()
        let entry : @config.DirEntry = match rest.find("/") {
          Some(index) => { name: rest[:index].to_owned(), kind: Directory }
          None => { name: rest, kind: File }
        }
        if !entries.contains(entry) {
          entries.push(entry)
        }
      }
      if entries.is_empty() {
        tree.with_missing(dir)
      } else {
        tree.with_listing(dir, entries)
      }
    }
  }
}

///|
/// Load until the configuration is done, answering each request from `disk`.
fn load_from(
  builder : @config.ConfigBuilder,
  disk : Map[String, String],
  tree? : @config.FileTree = @config.FileTree::new(),
) -> @config.Config raise @config.ConfigError {
  match @config.load(builder, tree) {
    Done(config) => config
    Need(requests) =>
      load_from(
        builder,
        disk,
        tree=requests.fold(init=tree, (tree, request) => {
          answer(tree, request, disk)
        }),
      )
  }
}

///|
test "a host answers the requests of a load" {
  let builder = @config.builder()
    |> @config.working_dir("/srv")
    |> @config.format(@toml.format())
    |> @config.file("conf/app.toml")
  let config = load_from(builder, { "/srv/conf/app.toml": "name = \"acme\"" })
  @expect.expect(@config.get_text(config, "name")).to_equal("acme")
  @expect.expect(config.reports()[0].path).to_equal(Some("/srv/conf/app.toml"))
}

A relative path joins the working directory that @config.working_dir sets. A missing explicit file is an error, FileNotFound.

Candidate files

A candidate list names the files that may hold one source, for example app.toml and app.yaml in one directory. A candidate exists when the listing of its directory holds its exact name as a file, also on a file system that ignores case. The load never chooses between two existing candidates.

///|
test "one existing candidate loads" {
  let builder = @config.builder()
    |> @config.format(@toml.format())
    |> @config.add(
      @config.ConfigSource::candidates(
        ["/srv/app.toml", "/srv/.config/app.toml"],
        display="App",
      ),
    )
  let config = load_from(builder, { "/srv/.config/app.toml": "mode = \"dev\"" })
  let report = config.reports()[0]
  @expect.expect((report.status, report.path)).to_equal((
    Loaded,
    Some("/srv/.config/app.toml"),
  ))
}

///|
test "two existing candidates are ambiguous" {
  let builder = @config.builder()
    |> @config.format(@toml.format())
    |> @config.add(
      @config.ConfigSource::candidates(
        ["/srv/app.toml", "/srv/.config/app.toml"],
        display="App",
      ),
    )
  let disk = { "/srv/app.toml": "", "/srv/.config/app.toml": "" }
  try load_from(builder, disk) catch {
    Ambiguous(location~, candidates~, ..) as error => {
      @expect.expect(location).to_equal("/srv")
      @expect.expect(candidates.length()).to_equal(2)
      @expect.expect(error.message()).to_start_with(
        "Ambiguous App configuration; found:",
      )
    }
    _ => fail("expected an ambiguity error")
  } noraise {
    _ => fail("expected an ambiguity error")
  }
}

Formats

A format turns file text into a value tree with a struct root. Register the formats that the application reads; the file extension selects one, compared in lowercase. parse_file gives other packages the same selection.

///|
test "the extension selects the format" {
  let formats = @config.builder()
    |> @config.format(@toml.format())
    |> @config.ConfigBuilder::formats
  @expect.expect(@config.parse_file(formats, "/srv/APP.TOML", "a = 1")).to_equal(
    @config.fields([("a", @config.int(1))]),
  )
  try @config.parse_file(formats, "/srv/app.ion", "{}") catch {
    UnsupportedFormat(extensions~, ..) =>
      @expect.expect(extensions).to_equal(["toml"])
    _ => fail("expected an unsupported-format error")
  } noraise {
    _ => fail("expected an unsupported-format error")
  }
}

The formats virtual package supplies all four format packages by default. An executable-package override selects TOML only or an application-defined set without changing the caller's import or registration loop. This selection happens at build time and can remove unused parsers from the executable. The guide explains MoonBit virtual packages, their pkg.mbti contract, and the difference between selecting linked code and registering formats in a builder.

Merge rules

A merged configuration never holds a null struct field: a null with no base value adds nothing. Nulls inside lists stay. The merge never changes its inputs.

Where each value came from

Every leaf of the merged tree has a non-empty list of origins: the kind, the display name, the priority, and the file path of the source that set it. A list counts as one leaf. A struct has no origin of its own, and a value that replaces a struct removes the records below it. Reports name each kind as Morphir does, for example project for a workspace file and user-override for a user override.

///|
test "the origin of each value" {
  let config = @config.build([
    @config.ConfigSource::defaults({ "ir": { "layout": "tree", "mode": "safe" } }),
    @config.in_memory_pairs(
      [("ir.mode", @config.string("fast"))],
      name="command line",
    ),
  ])
  let origins = config.provenance().map(record => {
    (record.0.to_string(), record.1.map(origin => origin.source))
  })
  @expect.expect(origins).to_equal([
    ("ir.layout", ["defaults"]),
    ("ir.mode", ["command line"]),
  ])
  @expect.expect(config.reports()[0].kind.report_name()).to_equal("defaults")
}

Excluding a subtree

A source can drop key paths after the parse and before the merge. A layer that a project inherits can drop the identity of another project without a warning. A member layer can drop a key that only the workspace may set, with a warning.

///|
test "an excluded key warns" {
  let source = @config.in_memory_pairs([
      ("workspace.out_dir", @config.string("out")),
      ("ir.layout", @config.string("tree")),
    ])
    .exclude([@config.KeyPath::parse("workspace.out_dir")], warn=true)
  let config = @config.build([source])
  @expect.expect(config.get(@config.KeyPath::parse("workspace.out_dir"))).to_equal(
    None,
  )
  @expect.expect(config.warnings().length()).to_equal(1)
}

Source providers

A source provider gives sources during the load, at its position in the list. Discovery uses this hook. A provider gets a LoadContext with the file tree, the platform, the variable set, the working directory, and the formats. It gives Need(requests) until the tree holds what it reads, then Done with its sources and warnings. A provider reads only the context, never the process.

///|
test "a provider adds the file of its directory" {
  let local_file : @config.SourceProvider = {
    name: "local",
    expand: context => {
      guard context.working_dir is Some(dir) else {
        return Done({ sources: [], warnings: [] })
      }
      Done({
        sources: [
          @config.ConfigSource::candidates(
            ["\{dir}/app.local.toml"],
            display="local",
            role=LocalOverride,
          ),
        ],
        warnings: [],
      })
    },
  }
  let builder = @config.builder()
    |> @config.working_dir("/srv")
    |> @config.format(@toml.format())
    |> @config.provider(local_file)
  let config = load_from(builder, { "/srv/app.local.toml": "debug = true" })
  @expect.expect(@config.get_text(config, "debug")).to_equal("true")
  @expect.expect(config.reports()[0].priority).to_equal(390)
}

Bind typed values

Read a section into a typed value with bind, or the full configuration with bind_root. Any type that implements FromJson works, usually through derive(FromJson). A section that no source sets decodes as null, so an Option target gives None.

///|
struct ServerOptions {
  host : String
  port : Int
} derive(FromJson, Eq, Debug)

///|
test "bind a section to a record" {
  let config = @config.build([
    @config.ConfigSource::defaults({
      "server": { "host": "localhost", "port": 8080 },
    }),
  ])
  let server : ServerOptions = config |> @config.bind("server")
  @expect.expect(server).to_equal({ host: "localhost", port: 8080 })
  let metrics : ServerOptions? = config.bind("metrics")
  @expect.expect(metrics).to_equal(None)
}

A decode failure raises Decode. Its text names the decode path, the decoder detail, and the sources that set the bad value, as <path>: <detail> (<blame text>). A path inside a list shows the index, for example targets[1].level. The text never shows the configured value.

///|
test "a decode error names the file that set the value" {
  let builder = @config.builder()
    |> @config.format(@toml.format())
    |> @config.file("/srv/app.toml")
  let tree = @config.FileTree::new(complete=true).with_file(
    "/srv/app.toml",
    "server.host = \"example.org\"\nserver.port = \"eighty\"",
  )
  guard @config.load(builder, tree) is Done(config) else { fail("no config") }
  try {
    let _ : ServerOptions = config |> @config.bind("server")
  } catch {
    error =>
      @expect.expect(error.message()).to_equal(
        "server.port: Int::from_json: expected number (set in project /srv/app.toml)",
      )
  } noraise {
    _ => fail("expected a decode error")
  }
}

Bind over a default value

bind_or merges the configured section over a typed default, with the same rules as the configuration merge: tables merge, a configured list or scalar replaces the default, and an absent section keeps the default. The type needs ToJson as well as FromJson. A failure in a value that the default gave says from the bind default.

///|
struct ServerDefaults {
  host : String
  port : Int
  tags : Array[String]
} derive(FromJson, ToJson, Eq, Debug)

///|
test "configured keys win over the default" {
  let config = @config.build([
    @config.in_memory_pairs([
      ("server.port", @config.int(9000)),
      ("server.tags", @config.list([@config.string("eu")])),
    ]),
  ])
  let server = config
    |> @config.bind_or("server", ({
      host: "localhost",
      port: 8080,
      tags: ["default"],
    } : ServerDefaults))
  @expect.expect(server).to_equal({ host: "localhost", port: 9000, tags: ["eu"] })
  @expect.expect(config.bind_or("mode", "safe")).to_equal("safe")
}

Binding decodes through FromJson, so its rules apply:

The configuration never holds a null field, so a null in a source does not reach an Option field: the field is absent and decodes as None.

Secret references in a bound section

Binding never resolves a secret reference such as { env = "API_TOKEN" }. FromJson would read a reference as an ordinary table, and binding can not tell which keys a type reads. So bind, bind_root and bind_or fail with Decode at the first reference in the bound section, with the detail is a secret reference; bind it as SecretRef with bind_with. Decode the section with FromConfig and bind_with, and give the field the type SecretRef (see "Exact decoding with FromConfig"). Or bind a narrower section that holds no reference. Read the secret value with Config::secret.

///|
struct ApiClient {
  url : String
} derive(FromJson)

///|
test "a secret reference stops binding" {
  let builder = @config.builder()
    |> @config.format(@toml.format())
    |> @config.file("/srv/app.toml")
  let tree = @config.FileTree::new(complete=true).with_file(
    "/srv/app.toml",
    "api.url = \"https://example.org\"\napi.token = { env = \"API_TOKEN\" }",
  )
  guard @config.load(builder, tree) is Done(config) else { fail("no config") }
  try {
    let client : ApiClient = config |> @config.bind("api")
    ignore(client.url)
  } catch {
    error =>
      @expect.expect(error.message()).to_equal(
        "api.token: is a secret reference; bind it as SecretRef with bind_with (set in project /srv/app.toml)",
      )
  } noraise {
    _ => fail("expected a decode error")
  }
  let url : String = config |> @config.bind("api.url")
  @expect.expect(url).to_equal("https://example.org")
}

Validate bound values

A type declares its rules with the Validate trait. validate is pure: it returns every broken rule as a Violation and never raises. violation(fields, detail) names fields of the section; a rule over two fields names both, so the error shows the source of each value. nested(field, found) moves the violations of a nested value below its field. Map, Array and Option values check each value, below its key or list index. A type with no rules writes impl Validate for T and uses the default, which finds nothing.

validated(value, config, key) checks a bound value and gives an Options[T]: the value, the key path of its section, and an optional instance name. All violations raise one Validation error, with one line for each path: <path>: <detail> (<blame text>). No options value exists after a failure. options(config, key) is the short form of bind, then validated.

///|
struct IrSettings {
  layout : String
  strict_mode : Bool
} derive(FromJson, ToJson)

///|
impl @config.Validate for IrSettings with validate(self) {
  if self.layout == "split" && !self.strict_mode {
    [
      @config.violation(
        ["layout", "strict_mode"],
        "split layout needs strict_mode = true",
      ),
    ]
  } else {
    []
  }
}

///|
test "validation names each value and its source" {
  let builder = @config.builder()
    |> @config.format(@toml.format())
    |> @config.file("/srv/app.toml")
  let tree = @config.FileTree::new(complete=true).with_file(
    "/srv/app.toml",
    "ir.layout = \"split\"",
  )
  guard @config.load(builder, tree) is Done(config) else { fail("no config") }
  let default : IrSettings = { layout: "tree", strict_mode: false }
  try {
    let ir = config
      |> @config.bind_or("ir", default)
      |> @config.validated(config, "ir")
    ignore(ir.value)
  } catch {
    error =>
      @expect.expect(error.message()).to_equal(
        (
          #|ir.layout: split layout needs strict_mode = true (set in project /srv/app.toml)
          #|ir.strict_mode: split layout needs strict_mode = true (not set by any source)
        ),
      )
  } noraise {
    _ => fail("expected a validation error")
  }
}

Named options

named(config, key, bind_one) binds each child of a section as one named instance of the same type. It calls bind_one with the configuration and the key text of each child, for example backends.ts, so bind_or, bind and validated all fit in it. Each options value gets the child name as its instance name. The Decode and Validation failures of all instances raise one Validation error. A section that no source sets gives no instances.

///|
struct BackendSettings {
  out_dir : String
  level : Int
} derive(FromJson, ToJson)

///|
impl @config.Validate for BackendSettings with validate(self) {
  if self.level > 0 {
    []
  } else {
    [@config.violation(["level"], "must be positive")]
  }
}

///|
test "bind each backend as a named instance" {
  let config = @config.build([
    @config.in_memory_pairs([
      ("backends.ts.out_dir", @config.string("dist/ts")),
      ("backends.scala.level", @config.int(2)),
    ]),
  ])
  let backends : @config.NamedOptions[BackendSettings] = config
    |> @config.named("backends", (config, key) => {
      config
      |> @config.bind_or(key, ({ out_dir: "out", level: 1 } : BackendSettings))
      |> @config.validated(config, key)
    })
  @expect.expect(backends.names()).to_equal(["ts", "scala"])
  guard backends.get("ts") is Some(ts) else { fail("no ts instance") }
  @expect.expect(ts.name).to_equal(Some("ts"))
  @expect.expect(ts.key.to_string()).to_equal("backends.ts")
  @expect.expect(ts.value.out_dir).to_equal("dist/ts")
}

Unknown keys

FromJson ignores keys that a type does not read, so a misspelled key has no effect. bind_or can find these keys: with unknown=Reject, it writes the bound value back with ToJson and compares the keys. Each configured key that the written value does not hold is a failure with its sources, and all of them raise one Validation error. The check also looks in list items. The default is Allow, which ignores unknown keys.

Reject is correct only when ToJson writes every key that FromJson accepts, under the same name. A derived FromJson and ToJson pair with the same arguments does this. A hand-written decoder can accept a key that its encoder does not write, for example an alias. For such a type, use RejectExcept(keys). It accepts the listed key paths, relative to the section, and the keys below them.

///|
test "a misspelled key is an error with its source" {
  let builder = @config.builder()
    |> @config.format(@toml.format())
    |> @config.file("/srv/app.toml")
  let tree = @config.FileTree::new(complete=true).with_file(
    "/srv/app.toml",
    "ir.layout = \"split\"\nir.strict_mod = true",
  )
  guard @config.load(builder, tree) is Done(config) else { fail("no config") }
  let default : IrSettings = { layout: "tree", strict_mode: false }
  let ir = config |> @config.bind_or("ir", default)
  @expect.expect(ir.strict_mode).to_equal(false)
  try {
    config |> @config.bind_or("ir", default, unknown=Reject) |> ignore
  } catch {
    error =>
      @expect.expect(error.message()).to_equal(
        "ir.strict_mod: is an unknown key (set in project /srv/app.toml)",
      )
  } noraise {
    _ => fail("expected a validation error")
  }
  let legacy = config
    |> @config.bind_or("ir", default, unknown=RejectExcept(["strict_mod"]))
  @expect.expect(legacy.layout).to_equal("split")
}

Exact decoding with FromConfig

FromJson reads numbers as Double, so it loses the scale of a decimal, the digits of a large integer, and the fraction of an Int. A type that implements FromConfig decodes straight from the configuration tree instead, and bind_with(config, key) binds it. Each decoder gets a ConfigNode: its decode path, its value, and the sources to blame.

The package supplies FromConfig for String, Bool, Int, Int64, UInt64, Double, @bigint.BigInt, @ion.Decimal, @ion.Timestamp, SecretRef, T?, Array[T], Map[String, T] and Sourced[T]. The integer types accept only an integer value in their range, so 80.5 fails for Int. @ion.Decimal accepts a decimal or an integer. Double accepts a float, an integer or a decimal. A node with no value gives None for T? and fails with no value for the other types.

Sourced[T] holds a value and the sources to blame for it, so the application can resolve a relative path against the file that set it. SecretRef accepts a secret reference and gives it unresolved; every other type fails at a reference.

///|
struct Invoice {
  rate : @ion.Decimal
  max_items : Int64
  lock : @config.Sourced[String]
  server : ServerOptions
}

///|
impl @config.FromConfig for Invoice with from_config(node) {
  {
    rate: node.get("rate"),
    max_items: node.get("max_items"),
    lock: node.get("lock"),
    server: node.child("server").json(),
  }
}

///|
test "exact decoding keeps the decimal scale and the file" {
  let builder = @config.builder()
    |> @config.format(@ion_format.format())
    |> @config.file("/srv/app.ion")
  let tree = @config.FileTree::new(complete=true).with_file(
    "/srv/app.ion",
    (
      #|{
      #|  billing: {
      #|    rate: 1.50,
      #|    max_items: 9007199254740993,
      #|    lock: "deps.lock",
      #|    server: { host: "localhost", port: 8080 },
      #|  },
      #|}
    ),
  )
  guard @config.load(builder, tree) is Done(config) else { fail("no config") }
  let billing : Invoice = config |> @config.bind_with("billing")
  @expect.expect(billing.rate.coefficient().to_string()).to_equal("150")
  @expect.expect(billing.rate.exponent()).to_equal(-2)
  @expect.expect(billing.max_items).to_equal(9007199254740993L)
  guard billing.lock.blame is Set([origin]) else { fail("one origin") }
  @expect.expect(origin.location).to_equal(Some("/srv/app.ion"))
  @expect.expect(billing.server).to_equal({ host: "localhost", port: 8080 })
}

Environment variables and overrides

An environment source reads the variables that start with its prefix. __ separates names, and the names are written in lower case, so ACME_SERVER__PORT=9000 sets server.port. Values get a type: true and false, integers, and JSON lists or tables; other text stays a string. The origin of each value names its variable. An overrides source, for example from the command line, has the highest default priority, 700.

///|
test "an environment variable overrides a file" {
  let builder = @config.builder()
    |> @config.env_vars({ "ACME_SERVER__PORT": "9000", "HOME": "/home/u" })
    |> @config.format(@toml.format())
    |> @config.file("/srv/acme.toml")
    |> @config.env(prefix="ACME")
  let tree = @config.FileTree::new(complete=true).with_file(
    "/srv/acme.toml",
    "server.port = 8080\nserver.host = \"example.org\"",
  )
  guard @config.load(builder, tree) is Done(config) else { fail("no config") }
  @expect.expect(@config.get_text(config, "server.port")).to_equal("9000")
  let origin = config.origins(@config.KeyPath::parse("server.port"))[0]
  @expect.expect(origin.location).to_equal(Some("ACME_SERVER__PORT"))
}

Hosts

The core never reads the disk or the process. A host gives it those inputs.

///|
test "a wasm host gives every input" {
  let builder = @config.builder()
    |> @config.env_vars({ "ACME_MODE": "fast" })
    |> @config.working_dir("/app")
    |> @config.format(@toml.format())
    |> @config.file("acme.toml")
  let tree = @config.FileTree::new(complete=true).with_file(
    "/app/acme.toml",
    "mode = \"safe\"",
  )
  guard @config.load(builder, tree) is Done(config) else {
    fail("a complete tree never asks for files")
  }
  @expect.expect(@config.get_text(config, "mode")).to_equal("safe")
}

Errors

Every failure is one ConfigError case with labelled fields, so a caller matches the case and does not compare text. message() gives text for people.

Later changes raise the other cases: environment keys, merge keys, secrets, discovery, and binding.