fleetlock

fleetlock is a reboot coordinator for Fedora CoreOS nodes in Kubernetes clusters. It implements the FleetLock protocol for use as a Zincati lock strategy backend.

Usage

Zincati runs on-host (zincati.service). Declare a Zincati fleet_lock strategy when provisioning Fedora CoreOS nodes. Set base_url for host nodes to access the in-cluster fleetlock Service (e.g. known ClusterIP).

variant: fcos
version: 1.4.0
storage:
  files:
    - path: /etc/zincati/config.d/55-update-strategy.toml
      contents:
        inline: |
          [updates]
          strategy = "fleet_lock"
          [updates.fleet_lock]
          base_url = "http://10.3.0.15/"

Apply the fleetlock Deployment, Service (with ClusterIP), and ServiceAccount.

kubectl apply -f examples/k8s

Inspect the fleetlock Lease object.

$ kubectl get leases -n default
NAME                HOLDER                             AGE
fleetlock-default   049ad0f57ade4723a48692b7b692c318   4m50s

Configuration

Configure the server via flags.

flag	description	default
-address	HTTP listen address	0.0.0.0:8080
-log-level	Logger level	info
-version	Show version	NA
-help	Show help	NA

Or via environment variables.

variable	description	default
NAMESPACE	Kubernetes Namespace	"default"
KUBECONFIG	Development Kubeconfig	NA

Typhoon

For Typhoon clusters, add the Zincati config a snippet.

module "nemo" {
  ...
  controller_snippets = [
    file("./snippets/zincati-strategy.yaml"),
  ]
  worker_snippets = [
    file("./snippets/zincati-strategy.yaml"),
  ]
}

Manual Intervention

fleetlock coordinates OS auto-updates to avoid concurrent node updates or a potential bad auto-update continuing. Zincati obtains a reboot lease lock before finalization (i.e reboot).

If an auto-update fails, the lease continues to be held by design. An admin should investigate the node failure and decide whether it is safe to remove the lease.

$ kubectl get leases
$ kubectl delete lease fleetlock-default

Metrics

fleetlock serves Prometheus /metrics from Go, process, and custom collectors.

name	description
fleetlock_lock_state	State of the fleetlock lease (0 unlocked, 1 locked)
fleetlock_lock_transition_count	Number of fleetlock lease transitions
fleetlock_lock_request_count	Number of lock requests
fleetlock_unlock_request_count	Number of unlock requests

Development

To develop locally, build and run the executable.

Static Binary

Build the static binary.

make build

Container Image

Build the container image.

make image

Run

Run the executable.

export KUBECONFIG=some-dev-kubeconfig
./bin/fleetlock

Use curl to emulate a Zincati FleetLock client.

{
  "client_params": {
    "id": "c988d2509fdf5cdcbed39037c56406fb",
    "group": "default"
  }
}

Request a reboot lock.

curl -H "fleet-lock-protocol: true" -d @examples/body.json http://127.0.0.1:8080/v1/pre-reboot

Release a reboot lock.

curl -H "fleet-lock-protocol: true" -d @examples/body.json http://127.0.0.1:8080/v1/steady-state

Related

• Zincati Guide
• Zincati Docs
• FleetLock Protocol