Agent fleet URL not taking what is in the yaml config

[clockard (Chase)]

Has anyone seen an issue with the fleet url that the agent is using does not correspond to the url in the yaml file ( elastic-agent-managed-kubernetes.yml) or the corresponding fleet configuration.
evn vars for elsatic-agent in kube-system

Environment:
      FLEET_ENROLL:            1
      FLEET_INSECURE:          true
      FLEET_URL:               https://fleet-server-agent-http.elasticsystem.svc:8220
      ELASTICSEARCH_HOST:      https://elasticsearch-es-http.elasticsystem.svc:9200
      FLEET_ENROLLMENT_TOKEN:  xxxxxxxxxxxx
      KIBANA_HOST:             http://kibana:5601
      KIBANA_FLEET_USERNAME:   elastic
      KIBANA_FLEET_PASSWORD:   xxxxxxxxxx
      NODE_NAME:                (v1:spec.nodeName)
      POD_NAME:                elastic-agent-qk6m2 (v1:metadata.name)
      ELASTIC_NETINFO:         false

In the logs for elastic-agent pod im seeing this.
message":"Error dialing lookup elasticsearch-es-http.elastic-system.svc on 10.96.0.10:53: no such host","component

Why is it using the wrong URL (elastic-system) and not (elasticsystem)
fyi The reason I'm using a non dash namespace is when you deploy the standard template and get an error saying you can't have a dash in the name space for the X pack configuration.
Also I'm using the correct URL and the fleet configuration.

[strawgate (William Easton)]

That error is actually Filebeat saying it can't access Elasticsearch at the specified url and is not related to fleet enrollment.

ELASTICSEARCH_HOST isn't defined as an environmental variable in the agent manifest I'm looking at, it should be getting the Elasticsearch host from output settings as defined in the agent policy in kibana.

Can you share the full modified manifest and the full log from the elastic agent pod? Can you check the output you have defined under "settings" in fleet and make sure the url is correct there?

[clockard (Chase)]

here is the eck manifest and Kubernetes agent manifest

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
  namespace: eck
spec:
  version: 8.15.3
  kibanaRef:
    name: kibana
  elasticsearchRefs:
  - name: elasticsearch
  mode: fleet
  fleetServerEnabled: true
  policyID: eck-fleet-server
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
            runAsUser: 0
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: eck
spec:
  version: 8.15.3
  kibanaRef:
    name: kibana
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        volumes:
        - name: agent-data
          emptyDir: {}
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: eck
spec:
  version: 8.15.3
  count: 1
  elasticsearchRef:
    name: elasticsearch
  http:
    service:
      spec:
        type: LoadBalancer
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.eck.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.eck.svc:8220"]
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        namespace: eck
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: eck
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
          - name: system-1
            id: system-1
            package:
              name: system
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: eck
spec:
  version: 8.15.3
  nodeSets:
  - name: default
    count: 2
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 2048Gi
  http:
    service:
      spec:
        type: LoadBalancer
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fleet-server
rules:
- apiGroups: [""]
  resources:
  - pods
  - nodes
  - namespaces
  - events
  - services
  - configmaps
  verbs:
  - get
  - watch
  - list
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - create
  - update
- nonResourceURLs:
  - "/metrics"
  verbs:
  - get
- apiGroups: ["extensions"]
  resources:
    - replicasets
  verbs:
  - "get"
  - "list"
  - "watch"
- apiGroups:
  - "apps"
  resources:
  - statefulsets
  - deployments
  - replicasets
  verbs:
  - "get"
  - "list"
  - "watch"
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
- apiGroups:
  - "batch"
  resources:
  - jobs
  verbs:
  - "get"
  - "list"
  - "watch"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: eck
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent
subjects:
- kind: ServiceAccount
  name: elastic-agent
  namespace: eck
roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io

AGENT

---
# For more information https://www.elastic.co/guide/en/fleet/current/running-on-kubernetes-managed-by-fleet.html
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: elastic-agent
  namespace: kube-system
  labels:
    app: elastic-agent
spec:
  selector:
    matchLabels:
      app: elastic-agent
  template:
    metadata:
      labels:
        app: elastic-agent
    spec:
      # Tolerations are needed to run Elastic Agent on Kubernetes control-plane nodes.
      # Agents running on control-plane nodes collect metrics from the control plane components (scheduler, controller manager) of Kubernetes
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      serviceAccountName: elastic-agent
      hostNetwork: true
      # 'hostPID: true' enables the Elastic Security integration to observe all process exec events on the host.
      # Sharing the host process ID namespace gives visibility of all processes running on the same host.
      hostPID: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: elastic-agent
          image: docker.elastic.co/beats/elastic-agent:8.15.4
          env:
            # Set to 1 for enrollment into Fleet server. If not set, Elastic Agent is run in standalone mode
            - name: FLEET_ENROLL
              value: "1"
            # Set to true to communicate with Fleet with either insecure HTTP or unverified HTTPS
            - name: FLEET_INSECURE
              value: "true"
            # Fleet Server URL to enroll the Elastic Agent into
            # FLEET_URL can be found in Kibana, go to Management > Fleet > Settings
            - name: FLEET_URL
              value: "https://fleet-server-agent-http.eck.svc:8220"
            # Elasticsearch API key used to enroll Elastic Agents in Fleet (https://www.elastic.co/guide/en/fleet/current/fleet-enrollment-tokens.html#fleet-enrollment-tokens)
            # If FLEET_ENROLLMENT_TOKEN is empty then KIBANA_HOST, KIBANA_FLEET_USERNAME, KIBANA_FLEET_PASSWORD are needed
            - name: FLEET_ENROLLMENT_TOKEN
              value: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=="
            - name: KIBANA_HOST
              value: "http://kibana:5601"
            # The basic authentication username used to connect to Kibana and retrieve a service_token to enable Fleet
            - name: KIBANA_FLEET_USERNAME
              value: "elastic"
            # The basic authentication password used to connect to Kibana and retrieve a service_token to enable Fleet
            - name: KIBANA_FLEET_PASSWORD
              value: "changeme"
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            # The following ELASTIC_NETINFO:false variable will disable the netinfo.enabled option of add-host-metadata processor. This will remove fields host.ip and host.mac.
            # For more info: https://www.elastic.co/guide/en/beats/metricbeat/current/add-host-metadata.html
            - name: ELASTIC_NETINFO
              value: "false"
          securityContext:
            runAsUser: 0
            # The following capabilities are needed for 'Defend for containers' integration (cloud-defend)
            # If you are using this integration, please uncomment these lines before applying.
            #capabilities:
            #  add:
            #    - BPF # (since Linux 5.8) allows loading of BPF programs, create most map types, load BTF, iterate programs and maps.
            #    - PERFMON # (since Linux 5.8) allows attaching of BPF programs used for performance metrics and observability operations.
            #    - SYS_RESOURCE # Allow use of special resources or raising of resource limits. Used by 'Defend for Containers' to modify 'rlimit_memlock'
            ########################################################################################
            # The following capabilities are needed for Universal Profiling.
            # More fine graded capabilities are only available for newer Linux kernels.
            # If you are using the Universal Profiling integration, please uncomment these lines before applying.
            #procMount: "Unmasked"
            #privileged: true
            #capabilities:
            #  add:
            #    - SYS_ADMIN
          resources:
            limits:
              memory: 700Mi
            requests:
              cpu: 100m
              memory: 400Mi
          volumeMounts:
            - name: proc
              mountPath: /hostfs/proc
              readOnly: true
            - name: cgroup
              mountPath: /hostfs/sys/fs/cgroup
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: etc-full
              mountPath: /hostfs/etc
              readOnly: true
            - name: var-lib
              mountPath: /hostfs/var/lib
              readOnly: true
            - name: etc-mid
              mountPath: /etc/machine-id
              readOnly: true
            - name: sys-kernel-debug
              mountPath: /sys/kernel/debug
            - name: elastic-agent-state
              mountPath: /usr/share/elastic-agent/state
            # If you are using the Universal Profiling integration, please uncomment these lines before applying.
            #- name: universal-profiling-cache
            #  mountPath: /var/cache/Elastic
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: cgroup
          hostPath:
            path: /sys/fs/cgroup
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        # The following volumes are needed for Cloud Security Posture integration (cloudbeat)
        # If you are not using this integration, then these volumes and the corresponding
        # mounts can be removed.
        - name: etc-full
          hostPath:
            path: /etc
        - name: var-lib
          hostPath:
            path: /var/lib
        # Mount /etc/machine-id from the host to determine host ID
        # Needed for Elastic Security integration
        - name: etc-mid
          hostPath:
            path: /etc/machine-id
            type: File
        # Needed for 'Defend for containers' integration (cloud-defend) and Universal Profiling
        # If you are not using one of these integrations, then these volumes and the corresponding
        # mounts can be removed.
        - name: sys-kernel-debug
          hostPath:
            path: /sys/kernel/debug
        # Mount /var/lib/elastic-agent-managed/kube-system/state to store elastic-agent state
        # Update 'kube-system' with the namespace of your agent installation
        - name: elastic-agent-state
          hostPath:
            path: /var/lib/elastic-agent-managed/kube-system/state
            type: DirectoryOrCreate
        # Mount required for Universal Profiling.
        # If you are using the Universal Profiling integration, please uncomment these lines before applying.
        #- name: universal-profiling-cache
        #  hostPath:
        #    path: /var/cache/Elastic
        #    type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent
subjects:
  - kind: ServiceAccount
    name: elastic-agent
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: kube-system
  name: elastic-agent
subjects:
  - kind: ServiceAccount
    name: elastic-agent
    namespace: kube-system
roleRef:
  kind: Role
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: elastic-agent-kubeadm-config
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: elastic-agent
    namespace: kube-system
roleRef:
  kind: Role
  name: elastic-agent-kubeadm-config
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
  labels:
    k8s-app: elastic-agent
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - namespaces
      - events
      - pods
      - services
      - configmaps
      # Needed for cloudbeat
      - serviceaccounts
      - persistentvolumes
      - persistentvolumeclaims
    verbs: ["get", "list", "watch"]
  # Enable this rule only if planing to use kubernetes_secrets provider
  #- apiGroups: [""]
  #  resources:
  #  - secrets
  #  verbs: ["get"]
  - apiGroups: ["extensions"]
    resources:
      - replicasets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - statefulsets
      - deployments
      - replicasets
      - daemonsets
    verbs: ["get", "list", "watch"]
  - apiGroups:
      - ""
    resources:
      - nodes/stats
    verbs:
      - get
  - apiGroups: [ "batch" ]
    resources:
      - jobs
      - cronjobs
    verbs: [ "get", "list", "watch" ]
  # Needed for apiserver
  - nonResourceURLs:
      - "/metrics"
    verbs:
      - get
  # Needed for cloudbeat
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources:
      - clusterrolebindings
      - clusterroles
      - rolebindings
      - roles
    verbs: ["get", "list", "watch"]
  # Needed for cloudbeat
  - apiGroups: ["policy"]
    resources:
      - podsecuritypolicies
    verbs: ["get", "list", "watch"]
  - apiGroups: [ "storage.k8s.io" ]
    resources:
      - storageclasses
    verbs: [ "get", "list", "watch" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: elastic-agent
  # Should be the namespace where elastic-agent is running
  namespace: kube-system
  labels:
    k8s-app: elastic-agent
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: elastic-agent-kubeadm-config
  namespace: kube-system
  labels:
    k8s-app: elastic-agent
rules:
  - apiGroups: [""]
    resources:
      - configmaps
    resourceNames:
      - kubeadm-config
    verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: kube-system
  labels:
    k8s-app: elastic-agent
---

Kub integration fleet API call

      "enabled": true,
      "streams": {
        "kubernetes.controllermanager": {
          "enabled": true,
          "vars": {
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token",
            "hosts": [
              "https://0.0.0.0:10257"
            ],
            "period": "10s",
            "ssl.verification_mode": "none",
            "controller_manager_label_key": "component",
            "controller_manager_label_value": "kube-controller-manager"
          }
        }
      }
    },
    "events-kubernetes/metrics": {
      "enabled": true,
      "streams": {
        "kubernetes.event": {
          "enabled": true,
          "vars": {
            "period": "10s",
            "add_metadata": true,
            "skip_older": true,
            "leaderelection": true
          }
        }
      }
    },
    "container-logs-filestream": {
      "enabled": true,
      "streams": {
        "kubernetes.container_logs": {
          "enabled": true,
          "vars": {
            "paths": [
              "/var/log/containers/*${kubernetes.container.id}.log"
            ],
            "symlinks": true,
            "data_stream.dataset": "kubernetes.container_logs",
            "containerParserStream": "all",
            "containerParserFormat": "auto",
            "additionalParsersConfig": "# - ndjson:\n#     target: json\n#     ignore_decoding_error: true\n# - multiline:\n#     type: pattern\n#     pattern: '^\\['\n#     negate: true\n#     match: after\n",
            "custom": ""
          }
        }
      }
    },
    "audit-logs-filestream": {
      "enabled": true,
      "streams": {
        "kubernetes.audit_logs": {
          "enabled": true,
          "vars": {
            "paths": [
              "/var/log/kubernetes/kube-apiserver-audit.log"
            ]
          }
        }
      }
    }
  }
}

Daemonset

kubectl --namespace kube-system describe daemonset elastic-agent
Name:           elastic-agent
Selector:       app=elastic-agent
Node-Selector:  <none>
Labels:         app=elastic-agent
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 3
Current Number of Nodes Scheduled: 3
Number of Nodes Scheduled with Up-to-date Pods: 3
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=elastic-agent
  Service Account:  elastic-agent
  Containers:
   elastic-agent:
    Image:      docker.elastic.co/beats/elastic-agent:8.15.4
    Port:       <none>
    Host Port:  <none>
    Limits:
      memory:  700Mi
    Requests:
      cpu:     100m
      memory:  400Mi
    Environment:
      FLEET_ENROLL:            1
      FLEET_INSECURE:          true
      FLEET_URL:               https://fleet-server-agent-http.eck.svc:8220
      FLEET_ENROLLMENT_TOKEN:  ZkdlSkxwTUJ4TVIxYWNpSTB2QzQ6U21sTXJ6S3FRRkdRY0NLUDRLdlRvQQ==
      KIBANA_HOST:             http://kibana:5601
      KIBANA_FLEET_USERNAME:   elastic
      KIBANA_FLEET_PASSWORD:   changeme
      NODE_NAME:                (v1:spec.nodeName)
      POD_NAME:                 (v1:metadata.name)
      ELASTIC_NETINFO:         false
    Mounts:
      /etc/machine-id from etc-mid (ro)
      /hostfs/etc from etc-full (ro)
      /hostfs/proc from proc (ro)
      /hostfs/sys/fs/cgroup from cgroup (ro)
      /hostfs/var/lib from var-lib (ro)
      /sys/kernel/debug from sys-kernel-debug (rw)
      /usr/share/elastic-agent/state from elastic-agent-state (rw)
      /var/lib/docker/containers from varlibdockercontainers (ro)
      /var/log from varlog (ro)
  Volumes:
   proc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:
   cgroup:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/cgroup
    HostPathType:
   varlibdockercontainers:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker/containers
    HostPathType:
   varlog:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:
   etc-full:
    Type:          HostPath (bare host directory volume)
    Path:          /etc
    HostPathType:
   var-lib:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib
    HostPathType:
   etc-mid:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/machine-id
    HostPathType:  File
   sys-kernel-debug:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/kernel/debug
    HostPathType:
   elastic-agent-state:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/elastic-agent-managed/kube-system/state
    HostPathType:  DirectoryOrCreate
Events:            <none>

[clockard (Chase)]

You'll have to forgive me but I posted what I could from the logs I can't see way to attach a file but I can get you anything you might need and again I really appreciate any time or effort that would put into this.

[WARN  tini (230323)] Tini is not running as PID 1 and isn't registered as a child subreaper.
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1.
{"log.level":"info","@timestamp":"2024-11-16T05:51:49.947Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":195},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":230335,"agent.version":"8.15.4","agent.unprivileged":false,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.164Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":144},"message":"agent is not upgradable, not starting watcher","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.166Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":284},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.167Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":65},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.190Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":71},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["containerd/metrics","http/metrics","gcp/metrics","jolokia/metrics","entity-analytics","httpjson","awsfargate/metrics","cloudfoundry/metrics","rabbitmq/metrics","cloudbeat/cis_k8s","meraki/metrics","sql/metrics","cloudbeat/cis_azure","cloudbeat/vuln_mgmt_aws","aws-cloudwatch","journald","netflow","activemq/metrics","pf-elastic-symbolizer","synthetics/browser","elasticsearch/metrics","cloud_defend/control","nats/metrics","pf-elastic-collector","cel","mysql/metrics","mssql/metrics","syncgateway/metrics","haproxy/metrics","kafka/metrics","memcached/metrics","osquery","cometd","unix","system/metrics","uwsgi/metrics","apache/metrics","docker","http_endpoint","synthetics/http","redis/metrics","syslog","enterprisesearch/metrics","mongodb/metrics","iis/metrics","audit/file_integrity","benchmark","container","gcp-pubsub","panw/metrics","pf-host-agent","cloudbeat","mqtt","stan/metrics","vsphere/metrics","fleet-server","prometheus/metrics","cloudbeat/cis_aws","cloudbeat/asset_inventory_aws","filestream","logstash/metrics","postgresql/metrics","oracle/metrics","statsd/metrics","zookeeper/metrics","cloudbeat/cis_eks","audit/system","kafka","salesforce","linux/metrics","log","synthetics/icmp","packet","cloudbeat/cis_gcp","aws/metrics","apm","endpoint","aws-s3","udp","beat/metrics","kubernetes/metrics","azure-blob-storage","redis","tcp","etw","winlog","etcd/metrics","audit/auditd","synthetics/tcp","nginx/metrics","traefik/metrics","o365audit","websocket","docker/metrics","kibana/metrics","azure-eventhub","cloudfoundry","gcs","lumberjack","windows/metrics","azure/metrics"],"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.190Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/capabilities.LoadFile","file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.190Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":77},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.190Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":92},"message":"Loading baseline config from /usr/share/elastic-agent/state/elastic-agent.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.393Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.NewManager","file.name":"runtime/manager.go","file.line":180},"message":"GRPC comms socket listening at localhost:6789","log":{"source":"elastic-agent"},"address":"localhost:6789","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.411Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":172},"message":"Parsed configuration and determined agent is managed by Fleet","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-11-16T05:51:50.412Z","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig","file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.442Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/control/v2/server.(*Server).Start","file.name":"server/server.go","file.line":88},"message":"GRPC control socket listening at unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","log":{"source":"elastic-agent"},"address":"unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.443Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).Run","file.name":"runtime/manager.go","file.line":246},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.443Z","log.logger":"composable.providers.docker","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker.(*dynamicProvider).Run","file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.443Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes.(*dynamicProvider).watchResource","file.name":"kubernetes/kubernetes.go","file.line":112},"message":"Kubernetes provider started for resource pod with node scope","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.449Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/kubernetes.DiscoverKubernetesNode","file.name":"kubernetes/util.go","file.line":162},"message":"kubernetes: Node talos-cn03 discovered by NODE_NAME environment variable","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.661Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.(*managedConfigManager).Run","file.name":"application/managed_mode.go","file.line":167},"message":"restoring current policy from disk","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.675Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/actions/handlers.(*PolicyChangeHandler).applyLoggingConfig","file.name":"handlers/handler_action_policy_change.go","file.line":405},"message":"Setting fallback log level <nil> from policy","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.688Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/gateway/fleet.(*FleetGateway).Run","file.name":"fleet/fleet_gateway.go","file.line":146},"message":"Fleet gateway started","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.690Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":123},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.691Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:\"\", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc0020e5650), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:\"\", APIKey:\"\", SecretToken:\"\", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.691Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":96},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.691Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.691Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.748Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1272},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-11-16T05:51:50.748Z","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig","file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.758Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes.(*dynamicProvider).watchResource","file.name":"kubernetes/kubernetes.go","file.line":112},"message":"Kubernetes provider started for resource node with node scope","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:50.758Z","log.logger":"composable.providers.kubernetes","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/kubernetes.DiscoverKubernetesNode","file.name":"kubernetes/util.go","file.line":146},"message":"kubernetes: Using node talos-cn03 provided in the config","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.089Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":627},"message":"Spawned new component kubernetes/metrics-default: Starting: spawned pid '230361'","log":{"source":"elastic-agent"},"component":{"id":"kubernetes/metrics-default","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.089Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit kubernetes/metrics-default-kubernetes/metrics-kubelet-a37785e2-579f-4843-9e21-f2c638fff820: Starting: spawned pid '230361'","log":{"source":"elastic-agent"},"component":{"id":"kubernetes/metrics-default","state":"STARTING"},"unit":{"id":"kubernetes/metrics-default-kubernetes/metrics-kubelet-a37785e2-579f-4843-9e21-f2c638fff820","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.089Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit kubernetes/metrics-default: Starting: spawned pid '230361'","log":{"source":"elastic-agent"},"component":{"id":"kubernetes/metrics-default","state":"STARTING"},"unit":{"id":"kubernetes/metrics-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.230Z","message":"Home path: [/usr/share/elastic-agent/data/elastic-agent-0ab98c/components] Config path: [/usr/share/elastic-agent/data/elastic-agent-0ab98c/components] Data path: [/usr/share/elastic-agent/state/data/run/kubernetes/metrics-default] Logs path: [/usr/share/elastic-agent/data/elastic-agent-0ab98c/components/logs]","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.origin":{"file.line":828,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.231Z","message":"Beat ID: a7423221-ca99-4665-bf34-36c6c2596a5e","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.origin":{"file.line":836,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.233Z","message":"Output reload is enabled, the beat will restart as needed on change of output config","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.logger":"centralmgmt","log.origin":{"file.line":204,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.NewV2AgentManagerWithClient"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.233Z","message":"Set gc percentage to: 100","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.origin":{"file.line":890,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.235Z","message":"running under elastic-agent, per-beat lockfiles disabled","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"ecs.version":"1.6.0","log.origin":{"file.line":443,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.235Z","message":"Starting stats endpoint","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.origin":{"file.line":69,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"api","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.236Z","message":"Syscall filter successfully installed","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"ecs.version":"1.6.0","log.logger":"seccomp","log.origin":{"file.line":125,"file.name":"seccomp/seccomp.go","function":"github.com/elastic/beats/v7/libbeat/common/seccomp.loadFilter"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.236Z","message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/wyZuDAx8vsqXD4cvx932213NPR-EMC2J.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/wyZuDAx8vsqXD4cvx932213NPR-EMC2J.sock)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.logger":"api","log.origin":{"file.line":71,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.236Z","message":"Beat info","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.logger":"beat","log.origin":{"file.line":1385,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"metricbeat","system_info":{"beat":{"path":{"config":"/usr/share/elastic-agent/data/elastic-agent-0ab98c/components","data":"/usr/share/elastic-agent/state/data/run/kubernetes/metrics-default","home":"/usr/share/elastic-agent/data/elastic-agent-0ab98c/components","logs":"/usr/share/elastic-agent/data/elastic-agent-0ab98c/components/logs"},"type":"metricbeat","uuid":"a7423221-ca99-4665-bf34-36c6c2596a5e"},"ecs.version":"1.6.0"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.236Z","message":"Build info","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"service.name":"metricbeat","system_info":{"build":{"commit":"bb01339f5a453be79bda545a40c863353134e88a","libbeat":"8.15.4","time":"2024-11-07T08:50:08.000Z","version":"8.15.4"},"ecs.version":"1.6.0"},"log.logger":"beat","log.origin":{"file.line":1394,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.236Z","message":"Go runtime info","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"system_info":{"ecs.version":"1.6.0","go":{"arch":"amd64","max_procs":4,"os":"linux","version":"go1.22.8"}},"log.logger":"beat","log.origin":{"file.line":1397,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.237Z","message":"Host info","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"service.name":"metricbeat","system_info":{"ecs.version":"1.6.0","host":{"architecture":"x86_64","boot_time":"2024-11-07T10:18:47Z","containerized":false,"id":"71a5d88a83f7ac62ed94c85876a4642c","ip":["127.0.0.1","::1","192.168.0.87","fe80::be24:11ff:fe67:caa1","fe80::80bf:c8ff:fefd:1ef3","10.244.1.37","fe80::7887:61ff:fe6b:cadd","fe80::b467:39ff:fe48:bd75","fe80::98b7:3bff:fe74:ca47","fe80::e414:71ff:fe0b:8b64","fe80::401f:2ff:fef6:ac6","fe80::d855:ebff:febd:f370","fe80::e0af:2eff:fe5f:75cc","fe80::e06b:a2ff:fe52:1295","fe80::5801:7cff:feca:d89e","fe80::40b6:46ff:fe17:c239"],"kernel_version":"6.6.32-talos","mac":["e6:4a:6a:7b:4c:47","ae:06:90:aa:59:5e","bc:24:11:67:ca:a1","82:bf:c8:fd:1e:f3","7a:87:61:6b:ca:dd","b6:67:39:48:bd:75","9a:b7:3b:74:ca:47","e6:14:71:0b:8b:64","42:1f:02:f6:0a:c6","da:55:eb:bd:f3:70","e2:af:2e:5f:75:cc","e2:6b:a2:52:12:95","5a:01:7c:ca:d8:9e","42:b6:46:17:c2:39"],"name":"talos-cn03","native_architecture":"x86_64\n","os":{"codename":"focal","family":"debian","major":20,"minor":4,"name":"Ubuntu","patch":6,"platform":"ubuntu","type":"linux","version":"20.04.6 LTS (Focal Fossa)"},"timezone":"UTC","timezone_offset_sec":0}},"log.logger":"beat","log.origin":{"file.line":1403,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.238Z","message":"Process info","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"service.name":"metricbeat","system_info":{"ecs.version":"1.6.0","process":{"capabilities":{"ambient":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"]},"cwd":"/usr/share/elastic-agent/state/data/run/kubernetes/metrics-default","exe":"/usr/share/elastic-agent/data/elastic-agent-0ab98c/components/agentbeat","name":"agentbeat","pid":230361,"ppid":230335,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2024-11-16T05:51:50.350Z"}},"log.logger":"beat","log.origin":{"file.line":1432,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.238Z","message":"Setup Beat: metricbeat; Version: 8.15.4","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.origin":{"file.line":341,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.239Z","message":"Output is configured through Central Management","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.origin":{"file.line":373,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.241Z","message":"Beat name: Talos-CN03","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.logger":"publisher","log.origin":{"file.line":105,"file.name":"pipeline/module.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.241Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.30.1","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.origin":{"file.line":79,"file.name":"add_kubernetes_metadata/kubernetes.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata.isKubernetesAvailable"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.245Z","message":"kubernetes: Node talos-cn03 discovered by NODE_NAME environment variable","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.origin":{"file.line":162,"file.name":"kubernetes/util.go","function":"github.com/elastic/elastic-agent-autodiscover/kubernetes.DiscoverKubernetesNode"},"service.name":"metricbeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","log.logger":"kubernetes","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.268Z","message":"Starting metrics logging every 30s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.logger":"monitoring","log.origin":{"file.line":150,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.268Z","message":"metricbeat start running.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.origin":{"file.line":540,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.276Z","log.logger":"component.runtime.kubernetes/metrics-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":715},"message":"control checkin v2 protocol has chunking enabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.278Z","message":"BeatV2Manager.unitListen UnitChanged.ID(kubernetes/metrics-default-kubernetes/metrics-kubelet-a37785e2-579f-4843-9e21-f2c638fff820), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":506,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.278Z","message":"BeatV2Manager.unitListen UnitChanged.ID(kubernetes/metrics-default), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.origin":{"file.line":506,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-11-16T05:51:51.336Z","log.logger":"transport","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.NetDialer.TestNetDialer.func3","file.name":"transport/tcp.go","file.line":53},"message":"DNS lookup failure \"fleet-server-agent-http.elastic-system.svc\": lookup fleet-server-agent-http.elastic-system.svc on 10.96.0.10:53: no such host","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-11-16T05:51:51.336Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/gateway/fleet.(*FleetGateway).doExecute","file.name":"fleet/fleet_gateway.go","file.line":191},"message":"Possible transient error during checkin with fleet-server, retrying","log":{"source":"elastic-agent"},"error":{"message":"fail to checkin to fleet-server: all hosts failed: 1 error occurred:\n\t* requester 0/1 to host https://fleet-server-agent-http.elastic-system.svc:8220/ errored: Post \"https://fleet-server-agent-http.elastic-system.svc:8220/api/fleet/agents/3006e4b5-3a09-494c-ae91-3ca1d2595f89/checkin?\": lookup fleet-server-agent-http.elastic-system.svc on 10.96.0.10:53: no such host\n\n"},"request_duration_ns":3468166,"failed_checkins":1,"retry_after_ns":66181389794,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.655Z","message":"Home path: [/usr/share/elastic-agent/data/elastic-agent-0ab98c/components] Config path: [/usr/share/elastic-agent/data/elastic-agent-0ab98c/components] Data path: [/usr/share/elastic-agent/state/data/run/filestream-default] Logs path: [/usr/share/elastic-agent/data/elastic-agent-0ab98c/components/logs]","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":828,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.656Z","message":"Beat ID: 7bd8d664-a063-4b9c-815c-020893c76add","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":836,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.658Z","message":"Output reload is enabled, the beat will restart as needed on change of output config","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"centralmgmt","log.origin":{"file.line":204,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.NewV2AgentManagerWithClient"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.658Z","message":"Set gc percentage to: 100","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":890,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.660Z","message":"running under elastic-agent, per-beat lockfiles disabled","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.origin":{"file.line":443,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.660Z","message":"Starting stats endpoint","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"api","log.origin":{"file.line":69,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.661Z","message":"Syscall filter successfully installed","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"seccomp","log.origin":{"file.line":125,"file.name":"seccomp/seccomp.go","function":"github.com/elastic/beats/v7/libbeat/common/seccomp.loadFilter"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.661Z","message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/7uyxrneW50ZMk_CZbrE0Hjg3idmd7AzZ.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/7uyxrneW50ZMk_CZbrE0Hjg3idmd7AzZ.sock)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"api","log.origin":{"file.line":71,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.661Z","message":"Beat info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"beat","log.origin":{"file.line":1385,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"beat":{"path":{"config":"/usr/share/elastic-agent/data/elastic-agent-0ab98c/components","data":"/usr/share/elastic-agent/state/data/run/filestream-default","home":"/usr/share/elastic-agent/data/elastic-agent-0ab98c/components","logs":"/usr/share/elastic-agent/data/elastic-agent-0ab98c/components/logs"},"type":"filebeat","uuid":"7bd8d664-a063-4b9c-815c-020893c76add"},"ecs.version":"1.6.0"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.661Z","message":"Build info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"beat","log.origin":{"file.line":1394,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"build":{"commit":"bb01339f5a453be79bda545a40c863353134e88a","libbeat":"8.15.4","time":"2024-11-07T08:50:08.000Z","version":"8.15.4"},"ecs.version":"1.6.0"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.661Z","message":"Go runtime info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"beat","log.origin":{"file.line":1397,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"ecs.version":"1.6.0","go":{"arch":"amd64","max_procs":4,"os":"linux","version":"go1.22.8"}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.662Z","message":"Host info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"beat","log.origin":{"file.line":1403,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"ecs.version":"1.6.0","host":{"architecture":"x86_64","boot_time":"2024-11-07T10:18:47Z","containerized":false,"id":"71a5d88a83f7ac62ed94c85876a4642c","ip":["127.0.0.1","::1","192.168.0.87","fe80::be24:11ff:fe67:caa1","fe80::80bf:c8ff:fefd:1ef3","10.244.1.37","fe80::7887:61ff:fe6b:cadd","fe80::b467:39ff:fe48:bd75","fe80::98b7:3bff:fe74:ca47","fe80::e414:71ff:fe0b:8b64","fe80::401f:2ff:fef6:ac6","fe80::d855:ebff:febd:f370","fe80::e0af:2eff:fe5f:75cc","fe80::e06b:a2ff:fe52:1295","fe80::5801:7cff:feca:d89e","fe80::40b6:46ff:fe17:c239"],"kernel_version":"6.6.32-talos","mac":["e6:4a:6a:7b:4c:47","ae:06:90:aa:59:5e","bc:24:11:67:ca:a1","82:bf:c8:fd:1e:f3","7a:87:61:6b:ca:dd","b6:67:39:48:bd:75","9a:b7:3b:74:ca:47","e6:14:71:0b:8b:64","42:1f:02:f6:0a:c6","da:55:eb:bd:f3:70","e2:af:2e:5f:75:cc","e2:6b:a2:52:12:95","5a:01:7c:ca:d8:9e","42:b6:46:17:c2:39"],"name":"talos-cn03","native_architecture":"x86_64\n","os":{"codename":"focal","family":"debian","major":20,"minor":4,"name":"Ubuntu","patch":6,"platform":"ubuntu","type":"linux","version":"20.04.6 LTS (Focal Fossa)"},"timezone":"UTC","timezone_offset_sec":0}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.663Z","message":"Process info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"beat","log.origin":{"file.line":1432,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"ecs.version":"1.6.0","process":{"capabilities":{"ambient":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"]},"cwd":"/usr/share/elastic-agent/state/data/run/filestream-default","exe":"/usr/share/elastic-agent/data/elastic-agent-0ab98c/components/agentbeat","name":"agentbeat","pid":230371,"ppid":230335,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2024-11-16T05:51:50.770Z"}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.663Z","message":"Setup Beat: filebeat; Version: 8.15.4","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.origin":{"file.line":341,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.664Z","message":"Output is configured through Central Management","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.origin":{"file.line":373,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.665Z","message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.30.1","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":79,"file.name":"add_kubernetes_metadata/kubernetes.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata.isKubernetesAvailable"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.666Z","message":"Beat name: Talos-CN03","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"publisher","log.origin":{"file.line":105,"file.name":"pipeline/module.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.667Z","message":"Enabled modules/filesets: ","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"modules","log.origin":{"file.line":136,"file.name":"fileset/modules.go","function":"github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.667Z","message":"Starting metrics logging every 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":150,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"monitoring","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.667Z","message":"filebeat start running.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":540,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.669Z","message":"kubernetes: Node talos-cn03 discovered by NODE_NAME environment variable","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0","log.logger":"kubernetes","log.origin":{"file.line":162,"file.name":"kubernetes/util.go","function":"github.com/elastic/elastic-agent-autodiscover/kubernetes.DiscoverKubernetesNode"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.680Z","message":"Loading data file of '/usr/share/elastic-agent/state/data/run/filestream-default/registry/filebeat' succeeded. Active transaction id=23874","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":127,"file.name":"memlog/store.go","function":"github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.682Z","message":"Finished loading transaction log file for '/usr/share/elastic-agent/state/data/run/filestream-default/registry/filebeat'. Active transaction id=24024","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":134,"file.name":"memlog/store.go","function":"github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-11-16T05:51:51.682Z","message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":331,"file.name":"beater/filebeat.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.686Z","message":"States Loaded from registrar: 0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"registrar","log.origin":{"file.line":107,"file.name":"registrar/registrar.go","function":"github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).loadStates"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.686Z","message":"Loading Inputs: 0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":71,"file.name":"beater/crawler.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"crawler","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.686Z","message":"Loading and starting Inputs completed. Enabled inputs: 0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"crawler","log.origin":{"file.line":106,"file.name":"beater/crawler.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.695Z","log.logger":"component.runtime.filestream-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":715},"message":"control checkin v2 protocol has chunking enabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.695Z","message":"BeatV2Manager.unitListen UnitChanged.ID(filestream-default-filestream-audit-logs-a37785e2-579f-4843-9e21-f2c638fff820), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":506,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-16T05:51:51.696Z","message":"BeatV2Manager.unitListen UnitChanged.ID(filestream-default), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":506,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"ecs.version":"1.6.0"}
I1116 05:51:51.864599  230335 leaderelection.go:250] attempting to acquire leader lease kube-system/elastic-agent-cluster-leader...
{"log.level":"info","@timestamp":"2024-11-16T05:51:52.030Z","message":"Home path: [/usr/share/elastic-agent/data/elastic-agent-0ab98c/components] Config path: [/usr/share/elastic-agent/data/elastic-agent-0ab98c/components] Data path: [/usr/share/elastic-agent/state/data/run/log-default] Logs path: [/usr/share/elastic-agent/data/elastic-agent-0ab98c/components/logs]","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":828,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"ecs.version":"1.6.0"}

[strawgate (William Easton)]

I used the two manifests you provided as-is (no changes) in a brand new K8s cluster with an eck namespace and everything immediately worked.

Can you upload the full agent log to a GitHub gist, pastebin, or something similar?

[clockard (Chase)]

@strawgate

pastebin

I posted the Operator logs and agent logs. I ran the whole EKC diagnostic so if you want any more information that I'm happy to provide it. again thank you so much for your time If there is a way I can buy you a cup of coffee let me know.

https://gist.github.com/Thor192/4c87af3d1d5e93ac4cb8136346523494

[strawgate (William Easton)]

@clockard

https://gist.github.com/Thor192/4c87af3d1d5e93ac4cb8136346523494

That agent log is partial and is missing the beginning. Is there anyway you can restart the agent pod and share the new log?

It also looks like you might be deploying Elastic Agents via the ECK operator:

---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: eck
spec:
  version: 8.15.3
  kibanaRef:
    name: kibana
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        volumes:
        - name: agent-data
          emptyDir: {}

and also via the Agent Manifest you've shared. It looks like the logs you shared may have been from the ECK-operator and not from the standalone manifest?

[strawgate (William Easton)]

The log from the ECK-operator deployed Elastic Agent (the log that was shared) actually shows data being successfully published to Elasticsearch:
{"log.level":"info","@timestamp":"2024-11-17T03:19:40.297Z","message":"Non-zero metrics in the last 30s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":291,"throttled":{"ns":105589,"periods":2}}},"memory":{"mem":{"usage":{"bytes":503275520}}}},"cpu":{"system":{"ticks":304360,"time":{"ms":50}},"total":{"ticks":648510,"time":{"ms":120},"value":648510},"user":{"ticks":344150,"time":{"ms":70}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":17},"info":{"ephemeral_id":"8041ede7-f82b-4633-afc8-9fbfcf7aba4c","uptime":{"ms":161341001},"version":"8.15.3"},"memstats":{"gc_next":91282808,"memory_alloc":49331528,"memory_total":26018998584,"rss":204865536},"runtime":{"goroutines":83}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":11}},"output":{"events":{"acked":92,"active":0,"batches":3,"total":92},"read":{"bytes":2845,"errors":3},"write":{"bytes":22772,"latency":{"histogram":{"count":15253,"max":1259,"mean":143.4794921875,"median":92,"min":32,"p75":166,"p95":484.75,"p99":814.25,"p999":1258.5500000000004,"stddev":157.0281111916896}}}},"pipeline":{"clients":11,"events":{"active":6,"published":90,"total":90},"queue":{"acked":92,"added":{"bytes":139741,"events":90},"consumed":{"bytes":141800,"events":92},"filled":{"bytes":20293,"events":6,"pct":0.001875},"max_bytes":0,"max_events":3200,"removed":{"bytes":141800,"events":92}}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"diskio":{"events":36,"success":36},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":18,"success":18},"process":{"events":18,"success":18},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.12,"15":0.19,"5":0.19,"norm":{"1":0.03,"15":0.0475,"5":0.0475}}}}},"log.logger":"monitoring","log.origin":{"file.line":192,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"ecs.version":"1.6.0"}

"output":{"events":{"acked":92,"active":0,"batches":3,"total":92}

[clockard (Chase)]

So then why don't I show the fleet agents in the server fleet integration? The agents never start.

[clockard (Chase)]

And to be clear, I have the standard agents deployed in the namespace for the system integration. The agents that aren't coming up are in the kube-system named space for integration for kubernetes logs

[strawgate (William Easton)]

The logs you shared in the gist were just for the agent in the eck namespace.

If you can share logs from an elastic agent pod in the kube-system namespace I can take a look at them

[clockard (Chase)]

I apologize profusely now that I think about it that makes a lot of sense but I have updated GIST with the appropriate logs. Thanks again for real.

[strawgate (William Easton)]

Okay yep

{"log.level":"warn","@timestamp":"2024-11-18T05:17:53.792Z","log.logger":"transport","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.NetDialer.TestNetDialer.func3","file.name":"transport/tcp.go","file.line":53},"message":"DNS lookup failure \"fleet-server-agent-http.elastic-system.svc\": lookup fleet-server-agent-http.elastic-system.svc on 10.96.0.10:53: no such host","ecs.version":"1.6.0"}

There's the fleet enrollment error you were pointing to before.

My top guess at this point is that maybe the Elastic Agent was previously deployed with this as the fleet enrollment URL, it didn't work, and then you redeployed it with a correct URL?

Once Agent starts, it translates the env vars into the local agent configuration and then the env vars no longer impact the Agent configuration.

As a result, you may need to wipe the agent configuration from the volume (in this case):

            - name: elastic-agent-state
              mountPath: /usr/share/elastic-agent/state
...
        - name: elastic-agent-state
          hostPath:
            path: /var/lib/elastic-agent-managed/kube-system/state
            type: DirectoryOrCreate

You should try to remove the /var/lib/elastic-agent-managed/kube-system/state folder from the node as its mounted from the host or by execing into the pod and removing the /usr/share/elastic-agent/state folder and then restarting the agent.

There's an open issue here that tracks a request for allowing env var changes to cause agent re-enrollment.

[clockard (Chase)]

Thank you so much I suspected something like that but I kind of wrapped my head around it.

If anyone finds it in the future I will give you the information.

So I'm using Talos and it doesn't have an SSH component so you have to spin up a debug container.

kubectl debug -n kube-system -it --image alpine node/$NODE

once in the container I ran the following

su
rm -r /host/var/lib/elastic-agent-managed/kube-system/state 

once you exit the container you can remove it with

kubectl delete pod --field-selector=status.phase==Succeeded -n kube-system

I am facing a cert issue but my problem is resolved

 x509: certificate signed by unknown authority

Thank you!!!!

[clockard (Chase)]

can I get some help on this? i'm just slow i think

Error dialing x509: certificate signed by unknown authority Kubernetes integration

I know there is allot on info about this, but I'm not grokking what needs to be done. The agents for the integration with kubernetes deploys in the kube-system namespace. i am using the quickstart fyi. What do i need to do to get the cert that was generated in the eck name space to the agents in kube-system? Error dialing x509: certificate signed by unknown authority this is the cert i want yes? elasticsearch-es-http-ca-internal