Collecting Agent Logs

Earlier versions of the LinearB On-Premise Agent could optionally deploy a bundled Fluent Bit DaemonSet (global.installFluentBit: true) that tailed container log files and wrote them to a local volume. That bundled DaemonSet has been removed. This guide explains how to point your own log collector at the agent instead.

Overview

Every LinearB service (agent-api, agent-cli, agent-poller, scheduler, webhook-receiver, jobs-suite-dispatcher, jobs-suite-worker, job-lifecycle-manager, pods-cleaner, and the rest) logs structured JSON to stdout/stderr through a shared internal logger. None of them write log files to disk, and there was never anything Fluent-Bit-specific baked into the services themselves — the bundled DaemonSet was just one particular way of reading the same stdout streams that Kubernetes already captures for every pod.

That means:

Log collection is a cluster concern, not an agent concern. Any standard Kubernetes log collector (Fluent Bit, Fluentd, an OpenTelemetry Collector, Vector, Datadog's own log collection, your cloud provider's managed logging, etc.) picks up agent logs with no changes to the agent.
If you already run a cluster-wide log collector, you are already collecting the agent's logs. The only thing you may want to add is a selector that isolates just the agent's pods — see The selector contract below.
If you don't run one yet, Example 1 and Example 2 below are complete, deployable starting points — the first is the direct replacement for the bundled DaemonSet's behavior (including writing to a local volume for air-gapped troubleshooting).

Removed values can block your upgrade

global.installFluentBit and the chart's bundled fluent-bit dependency no longer exist. If your values file sets global.installFluentBit: true, the chart fails the install or upgrade with an error naming this page — it does not silently ignore it.

That is deliberate: a removed flag that was quietly dropped would leave you believing a log collector was still running when it was not. installFluentBit: false and a fluent-bit: block on their own never installed the collector, so they do not block the upgrade; Helm prints a warning instead. Delete global.installFluentBit and any fluent-bit: block from your values file, then deploy one of the collectors below if you want agent logs collected.

global.installFluentBit: false (as the old local-values.yaml template shipped) asks for nothing, so it does not fail: the install continues and Helm prints a warning to delete it.

Error: execution error at (on-prem-agent/templates/removed-values-guard.yaml): global.installFluentBit
has been removed: the agent no longer bundles a Fluent Bit collector. ...

The selector contract

Every agent service pod carries a stable set of app.kubernetes.io/* labels:

Label	Example value	Stable across releases?
`app.kubernetes.io/name`	`agent-api`, `webhook-receiver`, … (per service)	Yes, per service
`app.kubernetes.io/instance`	`on-prem-agent` (your Helm release name)	Only if you never rename the release
`app.kubernetes.io/part-of`	`on-prem-agent`	Yes — recommended
`app.kubernetes.io/managed-by`	`Helm`	Yes

Use app.kubernetes.io/part-of: on-prem-agent as your collector's selector. Unlike app.kubernetes.io/instance — which is derived from whatever you named your Helm release, and changes if you ever rename or re-release it — part-of is a fixed value that identifies "this pod belongs to the on-prem agent" regardless of release name or which individual service it is. It is also the one label on the Linta, sensors, PM and ops Job pods the scheduler creates, so the same selector covers them.

Third-party components are not covered

The bundled third-party charts use their own labels: minio, rabbitmq and redis pods have no part-of, ingress-nginx sets part-of: ingress, and datadog sets its own value. To collect their logs too, add a second selector, e.g. on app.kubernetes.io/name.

Verify it immediately, before setting up any collector, with a plain kubectl logs:

kubectl -n linearb logs -l app.kubernetes.io/part-of=on-prem-agent --all-containers --prefix --tail=50

You should see structured JSON lines like:

{"asctime": "2026-08-21 10:15:32,101", "levelname": "INFO", "message": "job admitted", "ctx": {"service": "scheduler"}}

Illustrative, not a schema guarantee

The exact field names come from the agent's internal JSON log formatter and are not a public contract — treat asctime/levelname/message/ctx as representative, not something to build a brittle parser against. Every line is valid single-line JSON, which is what matters for a collector's JSON parser.

If that command returns nothing, check the namespace (linearb by default) and your Helm release name before assuming the label is missing — part-of should be present on every agent pod regardless of release name.

Adding your own pod labels

If your collector selects on a label of your own, add it with global.commonLabels (every agent service pod) or <service>.podLabels (one service; wins on conflict):

global:
  commonLabels:
    team: platform
agent-api:
  podLabels:
    tier: api

app.kubernetes.io/name, instance and part-of are set by the chart; setting them fails the install. These labels are not applied to third-party components or to scheduler-created Job pods.

Example 1: Fluent Bit

This is the direct replacement for the agent's old bundled DaemonSet — same tool, same /var/log/containers/*.log tailing approach, but deployed and owned by you, outside the agent's Helm release, so it survives upgrades and uninstalls of the agent chart untouched.

The manifests below deploy:

A ServiceAccount + ClusterRole + ClusterRoleBinding so Fluent Bit's kubernetes filter can query the Kubernetes API for pod metadata (labels, namespace, container name).
A ConfigMap holding the Fluent Bit configuration: tail /var/log/containers/*.log, enrich with Kubernetes metadata, then grep-filter down to only pods labeled app.kubernetes.io/part-of=on-prem-agent.
A DaemonSet that mounts the node's log directories via hostPath — see the privileges note below before you deploy this in a locked-down cluster.

Apply the shared RBAC and config first:

# onprem-log-collector-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: onprem-log-collector
  namespace: linearb
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: onprem-log-collector
rules:
  - apiGroups: [""]
    resources: ["namespaces", "pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: onprem-log-collector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: onprem-log-collector
subjects:
  - kind: ServiceAccount
    name: onprem-log-collector
    namespace: linearb

# onprem-fluent-bit-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: onprem-fluent-bit-config
  namespace: linearb
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush             1
        Log_Level         info
        Daemon            off
        HTTP_Server       Off

    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        multiline.parser  docker, cri
        Tag               kube.*
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Merge_Log           On
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

    [FILTER]
        Name    grep
        Match   kube.*
        Regex   $kubernetes['labels']['app.kubernetes.io/part-of'] ^on-prem-agent$

    [OUTPUT]
        Name    file
        Match   kube.*
        Path    /output
        File    on-prem-agent.log

That last [OUTPUT] block is option (a) — write to a local PVC, equivalent to the old bundled behavior, useful for air-gapped troubleshooting where there is no external log sink to forward to. For option (b) — forward to an external endpoint instead — replace that one [OUTPUT] block with:

[OUTPUT]
    Name         http
    Match        kube.*
    Host         logs.example.com
    Port         443
    URI          /v1/logs
    Format       json_lines
    tls          On
    tls.verify   On

(Swap Name http for Name forward if your backend speaks the Fluentd forward protocol instead of plain HTTP ingestion.) The rest of the ConfigMap — the [INPUT] and both [FILTER] blocks — stays identical either way. Only option (a) needs the output volume/PVC in the DaemonSet below; if you use option (b), drop the PersistentVolumeClaim, the output volume, and its volumeMounts entry.

Once you've picked an output, deploy the DaemonSet:

# onprem-log-collector-daemonset.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: onprem-agent-logs-pvc     # only needed for output (a)
  namespace: linearb
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: onprem-log-collector
  namespace: linearb
  labels:
    app.kubernetes.io/name: onprem-log-collector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: onprem-log-collector
  template:
    metadata:
      labels:
        app.kubernetes.io/name: onprem-log-collector
    spec:
      serviceAccountName: onprem-log-collector
      tolerations:
        - operator: Exists
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:2.2.3
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
            limits:
              memory: 200Mi
          volumeMounts:
            - name: config
              mountPath: /fluent-bit/etc/fluent-bit.conf
              subPath: fluent-bit.conf
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: varlibcontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: output
              mountPath: /output       # only needed for output (a)
      volumes:
        - name: config
          configMap:
            name: onprem-fluent-bit-config
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibcontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: output                # only needed for output (a)
          persistentVolumeClaim:
            claimName: onprem-agent-logs-pvc

kubectl apply -f onprem-log-collector-rbac.yaml
kubectl apply -f onprem-fluent-bit-config.yaml
kubectl apply -f onprem-log-collector-daemonset.yaml

This DaemonSet needs host-level read access

Reading /var/log/containers and /var/lib/docker/containers from every node means the pod needs hostPath volumes and, on most container runtimes, permission to read files that are root-owned on the node. Depending on your cluster's Pod Security Admission level or (on OpenShift) your SecurityContextConstraints, you may need to grant this DaemonSet's ServiceAccount a permissive SCC/PSA exemption, or run the container with an explicit securityContext that matches your policy. This is a standard requirement for any file-tailing log collector — it is not specific to Fluent Bit or to this agent — but a security reviewer should expect to see it called out, so it's called out here.

Image source

fluent/fluent-bit:2.2.3 above is the public Docker Hub image. In an air-gapped environment, mirror it into your own registry first, the same way you already mirror the agent's own images.

Example 2: OpenTelemetry Collector

Equivalent setup using the upstream opentelemetry-collector-contrib distribution: a filelog receiver in place of Fluent Bit's tail input, a k8sattributes processor in place of the kubernetes filter, and an OTLP exporter in place of Fluent Bit's forward/HTTP output.

# onprem-otelcol-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: onprem-otelcol
  namespace: linearb
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: onprem-otelcol
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces", "nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: onprem-otelcol
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: onprem-otelcol
subjects:
  - kind: ServiceAccount
    name: onprem-otelcol
    namespace: linearb

# onprem-otelcol-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: onprem-otelcol-config
  namespace: linearb
data:
  config.yaml: |
    receivers:
      filelog:
        include:
          - /var/log/pods/*/*/*.log
        include_file_path: true
        include_file_name: false
        operators:
          - type: container

    processors:
      k8sattributes:
        auth_type: "serviceAccount"
        passthrough: false
        extract:
          metadata:
            - k8s.pod.name
            - k8s.namespace.name
            - k8s.pod.uid
          labels:
            - tag_name: part_of
              key: app.kubernetes.io/part-of
              from: pod
      filter/onprem_agent:
        error_mode: ignore
        logs:
          log_record:
            - 'attributes["part_of"] != "on-prem-agent"'
      batch: {}

    exporters:
      otlp:
        endpoint: logs-collector.example.com:4317
        tls:
          insecure: false

    service:
      pipelines:
        logs:
          receivers: [filelog]
          processors: [k8sattributes, filter/onprem_agent, batch]
          exporters: [otlp]

# onprem-otelcol-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: onprem-otelcol
  namespace: linearb
  labels:
    app.kubernetes.io/name: onprem-otelcol
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: onprem-otelcol
  template:
    metadata:
      labels:
        app.kubernetes.io/name: onprem-otelcol
    spec:
      serviceAccountName: onprem-otelcol
      tolerations:
        - operator: Exists
      containers:
        - name: otelcol
          image: otel/opentelemetry-collector-contrib:0.111.0
          args: ["--config=/etc/otelcol/config.yaml"]
          resources:
            requests:
              cpu: 50m
              memory: 150Mi
            limits:
              memory: 300Mi
          volumeMounts:
            - name: config
              mountPath: /etc/otelcol
            - name: varlogpods
              mountPath: /var/log/pods
              readOnly: true
            - name: varlibcontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: onprem-otelcol-config
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibcontainers
          hostPath:
            path: /var/lib/docker/containers

kubectl apply -f onprem-otelcol-rbac.yaml
kubectl apply -f onprem-otelcol-config.yaml
kubectl apply -f onprem-otelcol-daemonset.yaml

The filter processor above drops every log record whose k8sattributes-derived part_of attribute isn't on-prem-agent, so only the agent's own logs reach the otlp exporter — the same narrowing that Fluent Bit's grep filter does in Example 1. As with Fluent Bit, mirror the otel/opentelemetry-collector-contrib image into your own registry if you're air-gapped, and expect the same host-level read access / Pod Security Admission considerations called out above.

Fresh k3d from scratch

This walks through the whole path on a bare machine: create a k3d cluster sized for the agent, deploy the agent, install a collector, and see agent logs flow end to end. Commands below assume you are in the public/ directory of this repository (where set-up-k3d.sh and deploy.sh live) and have already installed the prerequisite tooling (kubectl, helm, Docker, jq, oras, yq).

Create local-values.yaml from the template, filling in at least global.ORG_ID, global.JFROG_USER, global.JFROG_KEY, and global.JFROG_HOST (see the Deployment Guide). set-up-k3d.sh reads global.ORG_ID to name the cluster, and deploy.sh reads the JFrog values to pull the chart and images.

cp local-values.yaml.template local-values.yaml
# edit local-values.yaml with your ORG_ID and JFrog credentials

Create the k3d cluster. This installs k3d v5.8.3, generates k3d-cluster-config.yaml (1 server node + 2 agent nodes, Traefik disabled), and switches your kubectl context to it:

./set-up-k3d.sh

The cluster is named linearb-onprem-<ORG_ID> (from the ORG_ID you set in step 1), and kubectl config current-context should now read k3d-linearb-onprem-<ORG_ID>.

Deploy the agent:

./deploy.sh --latest
# or a pinned version: ./deploy.sh --version v5.x.x

This creates the linearb namespace if it doesn't exist and runs helm upgrade --install with your local-values.yaml.

Confirm the agent is up and already logging to stdout — no collector needed yet for this check:

kubectl -n linearb get pods
kubectl -n linearb logs -l app.kubernetes.io/part-of=on-prem-agent --all-containers --tail=20

Install a collector — apply the manifests from Example 1 or Example 2 above verbatim; nothing in them needs to change for k3d.
Verify logs are flowing through the collector. For Fluent Bit with output (a) (local PVC):

kubectl -n linearb exec -it daemonset/onprem-log-collector -- tail -f /output/on-prem-agent.log

For output (b) or the OpenTelemetry example, check your external endpoint's ingestion side, or temporarily swap in a stdout/debug exporter and read the collector pod's own logs:

kubectl -n linearb logs -l app.kubernetes.io/name=onprem-log-collector --tail=50

k3d specifics

k3d runs each Kubernetes node as a Docker container on your machine, not as a separate host. The cluster created above has one server node container and two agent node containers, named k3d-linearb-onprem-<ORG_ID>-server-0, k3d-linearb-onprem-<ORG_ID>-agent-0, and k3d-linearb-onprem-<ORG_ID>-agent-1.

This matters for the hostPath volumes in the manifests above (/var/log, /var/log/pods, /var/lib/docker/containers): those paths are resolved by the kubelet running inside each node container, which is exactly where the log collector's DaemonSet pods are scheduled — so the manifests above work unmodified, the same as on any other cluster. The one thing that's different from a bare-metal install is if you (not the collector) want to look at those files directly: they are not at /var/log/containers on your workstation. You'd need to shell into the node container itself, for example:

docker exec -it k3d-linearb-onprem-<ORG_ID>-agent-0 ls /var/log/containers

Not verified against this repo

The default StorageClass k3d/k3s provisions out of the box (commonly local-path, backed by Rancher's local-path-provisioner) is standard k3d/k3s behavior, but it is not something set-up-k3d.sh or any script in this repo configures or asserts — the script only wires up a hostPath-backed PV for the MinIO PVC specifically (see the next section). If you rely on dynamic provisioning for the collector's own PVC (Example 1, output (a)) on k3d, confirm your cluster's default StorageClass with kubectl get storageclass rather than assuming a name.

Retaining logs on disk in an air-gapped install

If you want the old bundled-Fluent-Bit-to-disk behavior back — for example, so support can inspect recent logs on a cluster with no outbound log-forwarding path — use output (a) in Example 1: a Fluent Bit file output writing to its own PersistentVolumeClaim.

Do not write logs onto the MinIO PVC

The chart's minio-pvc claim backs the agent's own object storage (integration data, job artifacts) — it is not free disk space. The old bundled design intentionally avoided this too (its file output wrote to a dedicated volume, not MinIO's), but it is worth stating explicitly now that log retention is your own DaemonSet's configuration rather than something the chart manages for you: point log output at a separate PVC, sized and monitored on its own, as shown in Example 1. Filling MinIO's volume breaks the agent's object storage, not just logging.

Log rotation is also now your responsibility. Fluent Bit's file output does not rotate or cap the files it writes — the chart used to ship this DaemonSet, but never actually managed rotation for you either. If you use output (a), plan for it yourself: a logrotate sidecar, a bounded/rolling output plugin, or simply monitoring the PVC's free space and alerting before it fills. The safer default for anything beyond short-term troubleshooting is output (b) — forward to an external log store that already handles retention.

Using `extraObjects`

If you'd rather manage the collector as part of the agent's own Helm release instead of applying separate manifests, the umbrella chart accepts an extraObjects list in values: raw Kubernetes manifests that get rendered (via Helm's tpl, so {{ .Release.Namespace }} and similar work) and installed alongside the rest of the chart.

Add the same objects from Example 1 (or Example 2) under extraObjects in local-values.yaml instead of applying them as separate files:

extraObjects:
  - apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: onprem-log-collector
      namespace: '{{ .Release.Namespace }}'
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: onprem-log-collector
    rules:
      - apiGroups: [""]
        resources: ["namespaces", "pods"]
        verbs: ["get", "list", "watch"]
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: onprem-log-collector
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: onprem-log-collector
    subjects:
      - kind: ServiceAccount
        name: onprem-log-collector
        namespace: '{{ .Release.Namespace }}'
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: onprem-fluent-bit-config
      namespace: '{{ .Release.Namespace }}'
    data:
      fluent-bit.conf: |
        [SERVICE]
            Flush             1
            Log_Level         info
            Daemon            off
            HTTP_Server       Off

        [INPUT]
            Name              tail
            Path              /var/log/containers/*.log
            multiline.parser  docker, cri
            Tag               kube.*
            Mem_Buf_Limit     5MB
            Skip_Long_Lines   On

        [FILTER]
            Name                kubernetes
            Match               kube.*
            Kube_URL            https://kubernetes.default.svc:443
            Merge_Log           On
            Keep_Log            Off
            K8S-Logging.Parser  On
            K8S-Logging.Exclude On

        [FILTER]
            Name    grep
            Match   kube.*
            Regex   $kubernetes['labels']['app.kubernetes.io/part-of'] ^on-prem-agent$

        [OUTPUT]
            Name    file
            Match   kube.*
            Path    /output
            File    on-prem-agent.log
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: onprem-agent-logs-pvc
      namespace: '{{ .Release.Namespace }}'
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
  - apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: onprem-log-collector
      namespace: '{{ .Release.Namespace }}'
      labels:
        app.kubernetes.io/name: onprem-log-collector
    spec:
      selector:
        matchLabels:
          app.kubernetes.io/name: onprem-log-collector
      template:
        metadata:
          labels:
            app.kubernetes.io/name: onprem-log-collector
        spec:
          serviceAccountName: onprem-log-collector
          tolerations:
            - operator: Exists
          containers:
            - name: fluent-bit
              image: fluent/fluent-bit:2.2.3
              resources:
                requests:
                  cpu: 50m
                  memory: 100Mi
                limits:
                  memory: 200Mi
              volumeMounts:
                - name: config
                  mountPath: /fluent-bit/etc/fluent-bit.conf
                  subPath: fluent-bit.conf
                - name: varlog
                  mountPath: /var/log
                  readOnly: true
                - name: varlibcontainers
                  mountPath: /var/lib/docker/containers
                  readOnly: true
                - name: output
                  mountPath: /output
          volumes:
            - name: config
              configMap:
                name: onprem-fluent-bit-config
            - name: varlog
              hostPath:
                path: /var/log
            - name: varlibcontainers
              hostPath:
                path: /var/lib/docker/containers
            - name: output
              persistentVolumeClaim:
                claimName: onprem-agent-logs-pvc

This is the complete set of objects from Example 1's option (a) (local PVC), just re-indented one level to sit under extraObjects and with {{ .Release.Namespace }} in place of the hardcoded linearb namespace so it tracks whatever namespace you actually install the chart into. Swap in Example 1's option (b) [OUTPUT] block (and drop the PVC/output volume) the same way if you'd rather forward externally, or swap in Example 2's objects instead for the OpenTelemetry Collector.

Then deploy as usual:

./deploy.sh --latest

The trade-off is coupling: the collector now upgrades, and uninstalls, together with the agent release. If you'd rather manage its lifecycle independently — e.g. so helm uninstall on the agent doesn't also tear down your log collector — keep it as separate manifests (Examples 1 and 2 above) instead.