Secrets

Current implementation

Secrets are currently implemented using sops-nix . The key used to decrypt them is stored at /var/lib/sops-nix/key.txt, and when a new system is installed using the ./install.sh script that key is copied over to the newly provisioned device. At system activation the key is used to decrypt the encrypted secrets that were placed in the /nix/store when the system was built.

The encrypted secrets are stored in a repository separate from this one to provide some resistance against harvest now, decrypt later attacks. To edit them, make a copy of the secrets repository and follow the instructions listed there to update, commit, merge, and push your changes. From there you can update the pin for that repository in this one.

As of right now beacon is unable to have any deployed secrets on it due to the single shared global decryption key. If beacon was provided that key and was compromised it would be very hard to recover from that scenario. This is an issue for beacon and not other devices because it is our most public facing device.

User passwords

Human account passwords are managed via Kerberos. Client hosts authenticate them against the realm and pull passwords from there.

See the device identity documentation for how the realm itself is laid out

Some legacy users are still managed via sops and need to be migrated to kerberos when the owners of these accounts are available to set up their passwords. Legacy users are currently commented out in the config but can be reenabled when needed.

Updating passwords

From the kdc root

sudo kadmin.local -q "cpw username@JAN-LEILA.COM"

From client device

# change your own password
kpasswd
# change a specific users password
kpasswd <user>

Future plans

Design goals

• secrets should be deterministic.
• access should be granted per secret.
• secret backend should be unable to read secrets.
• a build server should be able to create a derivation using encrypted secrets without being able to decrypt those secrets itself.
• first time system activation should not be require network. Network should only be required for build time.
• secrets should be able to have placeholder values that can be used.
• secrets should have a isProduction metadata flag on them so we can refuse to evaluate a derivation for production with placeholder/development values.
• secret values should be a semi-immutable ledger of all values that have been used for a particular identifier. (we can delete things if we really really need to but its best practice if things are never deleted)
• secrets should not be able to be read out of the /nix/store/ directly and must be decrypted at activation time and placed in /run/secrets/.
• secrets in /run/secrets/ should have ownership set to specific users. (should not fail when the user does not exist and just skip and warn)
• pluggable backend that can be replaced with multiple options such as a server or a static file. We should be able to set the backend per secret.
• access to encrypted secrets should be restricted to authorized identities as much as possible to protect against harvest now decrypt later attacks.
• secret writer identities vs read identities vs access only identities
• secrets must be signed by writer so that we don’t need to trust the server to not forge secrets

Notes

Static file security

A static file backend is able to restrict access to authorized identities by splitting it into partial files per client. Each client gets its own file that only contains the ledger entries and grants that client is authorized for, so only that partial file ends up in the clients nix store instead of the full set of secrets.

Authorization extension

When we add a new identity as an authorized user of a secret we need a way of giving it its e2e tokens.

Determinism

All identities need to have the same key at the end of the day so the hash is deterministic.

ie. there needs to be one canonical key for every single secret

I think this means that we can’t use any KDF tricks because that would prevent us from revoking access to a secret once it is generated.

Secrets are irrevocable revoke

Secrets are not able to be revoked from an identity because they could just have a cached value that they are using or just not listen to us if we want them to delete something.

Provisioning identities

The public key of a client identity has to exist before its system can be built, so the private key has to be made before the first build.

Long term the private key should be generated on the device during install so it never leaves the device:

1. Boot the target into the installer.
2. Generate the identity on the target in memory and print only the public key.
3. Set that public key in the hosts config, add the host as a reader to the secrets it needs, and update the pins.
4. Build the system, partition the disks, move the private key from memory to privateKeyFile on the target disk, and install.

This requires a bunch of custom tooling. Until that tooling exists the identity will be generated off device by an admin and shipped to the device at install time along with the system.

Proposed API

{ lib, config, secrets, ... }: {
  options = {
    secretValue = secrets.mkSecretOption {
      # arbitrary hash to identify which secret this refers to
      identifierHash = lib.fakeHash;
      # value to use if the pinHash is not set.
      # If this value is used isProduction will always be false
      placeholder = "value here";
      description = "xyz";
      owner = "xyz";
      group = "xyz";
    };
  };

  secrets = {
    # if this is set to true then all used secrets need to have a production flag set.
    production = false;
    identity = {
      # public private key pair used to identify a particular host so that we grant access to secrets to each host individually.
      publicKey = "public key here";
      # on system path to the private key matching publicKey.
      privateKeyFile = "/path/here";
    };
    keystore = {
      # used to store e2e encryption keys per secret.
      path = "/path/here";
    };
  };

  config = {
    secretValue = {
      # overwrite the pin to be a specific value in the ledger.
      # If this value is set to fakeHash then the building should inform you what the latest pinHash is for every known audience.
      # This pin is based on a hash of the metadata attached to the secret combined with the e2e encrypted secret. The secret is only pulled when any of its properties are evaluated.
      pinHash = lib.fakeHash;
      # force a particular secret to be a production key. If the metadata of the pinned secret does not match this will cause the derivation to fail to evaluate 
      forceProduction = true;
    };

    # once a secret is pulled it will be stored encrypted in the nix store. Then at runtime it will be decrypted and stored under /run/secrets
    passwordFile = config.secretValue.filePath;
    # true if the pinned value is flagged as production by the backends metadata
    passwordIsProduction = config.secretValue.isProduction;
  };
}

Protocol

Primitives

Algorithms are left as placeholders. Any algorithm with the listed properties can fill a placeholder.

• H: hash function. Collision resistant and second preimage resistant. Must be a hash nix accepts as a fixed output hash so a pinHash can be used directly as the outputHash of the derivation that fetches an entry.
• SIG: signature scheme (KeyGen, Sign, Verify). Strongly unforgeable under chosen message attack. Must be post quantum secure (or a hybrid) because ledger entries are long lived and a forged entry could be pinned years later.
• KEM: key encapsulation mechanism (KeyGen, Encaps, Decaps). IND-CCA2 secure. Must be post quantum secure, because wrapped keys are readable by anyone holding an entry.
• AEAD: authenticated encryption with associated data (Seal, Open). IND-CCA2 confidentiality and ciphertext integrity. Must be post quantum secure. Key committing, so a ciphertext only opens under one key (a writer can’t give different readers different plaintexts for the same entry). Nonces large enough to be generated randomly without risk of collision.
• KDF: key derivation function. Output is indistinguishable from random when the input has enough entropy, and it takes a context string for domain separation. Outputs for the same input under different context strings must be independent, so learning one reveals nothing about another.
• RNG: cryptographically secure random number generator.
• ENC: canonical encoding. Deterministic and injective: every value has exactly one encoding and every encoding parses unambiguously. Fields are encoded in the order listed.

|| is concatenation. Every input to H, SIG, and KDF starts with a unique tag string (for example "secrets/entry") so values from one context can never be confused with values from another.

Identities

• Writer: a SIG key pair and a KEM key pair created by the secrets cli or gui.

  • writerPublic = ENC(sigPublic, kemPublic)
  • identifierHash = H("secrets/writer" || writerPublic)

  The identifier of a secret is the hash of its writer key, meaning “the secret that this key is able to write to”. Holding the writer private keys lets you append entries, read every version, and grant read access. Writer keys are never rotated. Rotating a writer key means creating a new secret identity.

• Reader: a KEM key pair for a client. The public key is set in secrets.identity.publicKey and the private key lives on the client at secrets.identity.privateKeyFile.

  • readerPublic = ENC(kemPublic)
  • readerId = H("secrets/reader" || readerPublic)

  The public key is uploaded to the backend. Anyone who fetches a readerPublic checks it against the readerId they asked for, so the backend can’t substitute its own key.

• Access only: a KEM key pair for an identity that can fetch entries but never appears as a recipient, such as a build server. The key is only used to authenticate with a backend.

  • accessPublic = ENC(kemPublic)
  • accessId = H("secrets/access-id" || accessPublic)

  Access only identities are listed in accessIds and never get a recipient record, so there is no wrappedDEK for their key to open. The only thing their key is ever used for is the authentication challenge, which only returns a value under the "secrets/auth" context string. A key must never be used as both a reader and an access only identity, otherwise the identity could read through its reader record. Tooling refuses to add a reader whose kemPublic matches a known access only identity and the other way around.

Ledger entries

EntryBody = ENC(
  formatVersion,
  identifierHash,
  writerPublic,
  seq,
  audience,
  isProduction,
  nonce,
  ciphertext,
  recipients,
  accessIds,
)
Entry   = ENC(EntryBody, SIG.Sign(writerSigPrivate, "secrets/entry" || EntryBody))
pinHash = H(Entry)

• formatVersion: version of this protocol.
• seq: version of the secret value. It only increments when the value changes. An entry that only adds recipients or access ids keeps the seq of the value it re-grants, so giving more access to an old value doesn’t make it look like the newest value.
• audience: a name chosen by the writer for who the value is meant for, such as "production", "production-demo-server", "staging", or "development". It is only a name and is not tied to isProduction.
• isProduction: set by the writer and covered by the signature, so the backend can’t change it.
• nonce: from RNG.
• ciphertext = AEAD.Seal(DEK, nonce, value, ad = H("secrets/value" || ENC(identifierHash, seq, audience, isProduction))) where DEK is a data encryption key from RNG.
• recipients: list of recipient records sorted by id. It always contains the writer (with id = identifierHash) so the writer can re-grant later.
• accessIds: sorted list of accessIds allowed to fetch this entry.

A recipient record is ENC(id, encapsulation, wrappedDEK):

(encapsulation, sharedSecret) = KEM.Encaps(recipientKemPublic)
wrapKey    = KDF(sharedSecret, "secrets/wrap" || ENC(identifierHash, seq, id, encapsulation))
wrappedDEK = AEAD.Seal(wrapKey, zeroNonce, DEK, ad = H("secrets/wrap-ad" || ciphertext))

A zero nonce is safe here because each wrapKey comes from a fresh encapsulation and is only used once.

All randomness (DEK, nonce, encapsulations, signature randomness) is used once, when the writer creates the entry. After that the entry is fixed bytes fetched by pinHash as a fixed output derivation, which is what keeps builds deterministic. Because the recipients and access ids are part of the entry, giving anyone new access always means a new entry and a new pin.

Writing

A new value:

1. seq is one more than the highest seq in the ledger, or 0 for the first entry. Set audience and isProduction.
2. Generate a fresh DEK and nonce, and encrypt the value.
3. Wrap the DEK for the writer and each reader. Reader public keys are fetched from the backend and checked against their readerId.
4. Set accessIds, sign, and append.

A re-grant (adding readers or access ids to an existing seq):

1. Take the latest entry with that seq and unwrap its DEK with the writer KEM key.
2. Copy seq, audience, isProduction, nonce, and ciphertext from that entry. The recipients are the old recipient records (copied as is) plus new records for the added readers, and the access ids are the old ones plus the added ones.
3. Sign and append.

A re-grant can’t remove readers because a removed reader already has the DEK. Removing a reader means writing a new value.

Any entry can be deleted except the latest entry for each audience. The entry with the highest seq is always the latest for its audience, so it is never deleted and a seq is never reused for a different value.

Verification

Ledger rules, checked by the backend when appending and by tooling when walking the ledger:

• H("secrets/writer" || writerPublic) == identifierHash, and the signature verifies under writerPublic.
• seq is either higher than every seq in the ledger (a new value), or equal to an existing seq (a re-grant). For a re-grant, audience, nonce, ciphertext, and isProduction must match the existing entries with that seq, and recipients and accessIds must be a strict superset of those in every existing entry with that seq.
• recipients and accessIds are sorted and unique, and recipients contains identifierHash.

Because every re-grant is a strict superset of all entries before it with the same seq, entries sharing a seq are always ordered by what they contain. The latest entry for an audience is the one with the highest seq in that audience, and among entries with that seq, the one whose recipients and access ids contain all the others. The latest entry for each audience is what the build reports when pinHash is set to fakeHash.

At evaluation, where only the pinned entry is available:

1. Fetch the entry by pinHash.
2. Check that H("secrets/writer" || writerPublic) equals the option’s identifierHash and that the signature verifies.
3. If secrets.production or forceProduction is set, require that isProduction is true and that no placeholder is used.
4. Check that H("secrets/reader" || secrets.identity.publicKey) is in recipients so a missing grant fails at evaluation instead of activation.

At activation, with no network:

1. Verify the signature again and find the recipient record whose id is the host’s readerId.
2. sharedSecret = KEM.Decaps(identityKemPrivate, encapsulation), derive wrapKey, and open wrappedDEK.
3. Open ciphertext with the same associated data used when sealing. Any failure is fatal for that secret.
4. Write the value to /run/secrets/ with the configured owner and group, and discard the DEK.

Placeholders never touch the backend. They are written to /run/secrets/ through the same activation path and always have isProduction set to false.

Backends

Every backend provides:

• getEntry(pinHash): returns the entry.
• getLedger(identifierHash): returns every entry the requester is allowed to see in seq order, used by tooling to find the latest entries.
• appendEntry(entry): the backend checks the ledger rules and rejects entries that break them. Clients still verify everything themselves.
• putIdentity(public) and getIdentity(id): store and look up reader and access only public keys. The caller checks the hash.

A backend only serves an entry to the writer and the identities in its recipients and accessIds.

Authentication

Before serving anything a backend challenges the requesting identity to prove it holds the private key for its id. The challenge uses the identity’s existing KEM key so no extra key pair is needed:

client  -> backend : id, request
backend            : (encapsulation, sharedSecret) = KEM.Encaps(kemPublic of id)
backend -> client  : encapsulation
client             : sharedSecret = KEM.Decaps(kemPrivate, encapsulation)
client  -> backend : proof = KDF(sharedSecret, "secrets/auth" || ENC(backendId, encapsulation, channelBinding, request))
backend            : check proof in constant time, then check id is allowed for request

• A fresh encapsulation for every request stops old proofs from being replayed.
• channelBinding is a value unique to the connection (such as a TLS exporter), so a proof relayed from the real client can’t be used on another connection.
• backendId stops a proof made for one backend being used against another.
• request stops the requested entries being swapped after the proof is made.

A malicious backend can send an encapsulation copied from a recipient record and get back a proof made from that record’s sharedSecret. This is safe because the proof and the wrapKey use different KDF context strings, so the proof reveals nothing about the wrapKey. Clients must never return anything derived from a sharedSecret except under the "secrets/auth" context string.

A static file backend stores the same data as files and is written by the tooling, which enforces the ledger rules on its behalf. It generates a partial file per client containing only the entries that client is authorized for. It can’t run the challenge, so access to a partial file is whatever protects where it is hosted. The build server fetches its full partial file and checks each entry against its pinHash and writer signature at build time instead.

The backend never sees plaintext or private keys. It can see identifier hashes, the readerIds and accessIds on each entry, seqs, isProduction, sizes, and when entries are appended.