Automating Elastic Agent Policies

Elastic Stack Elastic Agent
1

I have been experimenting with attempting to automate the ECK elastic agent policy in fleet. Currently, my fleet server set up is this with these two policies defined:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: ${kibana_name}
spec:
  version: 8.4.1
  count: 1
  elasticsearchRef:
    name: eck-elasticsearch-cluster
  config:
    xpack.fleet.agents.fleet_server.hosts: "${fleet_server_urls}"
    xpack.fleet.agents.elasticsearch.hosts: "${fleet_elasticsearch_urls}"
    xpack.fleet.agents.pollingRequestTimeout: 30000
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
      - name: kubernetes
        version: 1.19.1
      - name: journald
        version: 1.1.0
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        namespace: default
        is_managed: false
        is_default: false
        is_default_fleet_server: true
        unenroll_timeout: 200
        package_policies:
        - name: fleet_server_1
          id: fleet_server_1
          package:
            name: fleet_server
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 200
        is_default: true
        package_policies:
          - name: journald-1
            id: journald-1
            package:
              name: journald
          - name: kubernetes-1
            id: kubernetes-1
            package:
              name: kubernetes

Now, once this is created. We go into the UI, and update the kubernetes integration under this ECK Elastic Policy to only pull k8s container logs:

Capture
Capture1232×1059 72.4 KB

Now, I want to automate this by adding it into my yaml. so I thought I could view the yaml for the Elasticsearch policy, and add the inputs under the package field in my original yaml aboce. Here is the elastic agent policy yaml I can pull from the UI once I manually edited it to my liking:

id: eck-agent
revision: 4
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://fleet.es.test.domain.com:443'
fleet:
  hosts:
    - >-
      https://eck-fleet-server.test.domain.com:443
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices:
        - names:
            - logs-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.cloud_defend-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.cloudbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.cloudbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat_input-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat_input-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_elastic_collector-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_elastic_symbolizer-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_host_agent-default
          privileges:
            - auto_configure
            - create_doc
    _elastic_agent_checks:
      cluster:
        - monitor
    journald-1:
      indices:
        - names:
            - logs-*-*
          privileges:
            - auto_configure
            - create_doc
    kubernetes-1:
      indices:
        - names:
            - logs-kubernetes.container_logs-default
          privileges:
            - auto_configure
            - create_doc
agent:
  download:
    sourceURI: 'https://artifacts.elastic.co/downloads/'
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: true
  features: {}
inputs:
  - id: journald-logs-journald-1
    name: journald-1
    revision: 1
    type: journald
    use_output: default
    meta:
      package:
        name: journald
        version: 1.1.0
    data_stream:
      namespace: default
    package_policy_id: journald-1
    streams:
      - id: journald-journald.logs-journald-1
        data_stream:
          dataset: logs
        condition: '${host.platform} == ''linux'''
        tags:
          - journald-log
        processors:
          - convert:
              tag: journald-to-ecs
              mode: rename
              ignore_missing: true
              fields:
                - from: message_id
                  to: event.code
                - from: journald.code.file
                  to: log.origin.file.name
                - from: journald.code.line
                  to: log.origin.file.line
                - from: journald.code.func
                  to: log.origin.function
                - from: syslog.pid
                  to: log.syslog.procid
                - from: syslog.identifier
                  to: log.syslog.appname
          - drop_fields:
              ignore_missing: true
              fields:
                - syslog
                - container.id_truncated
  - id: filestream-container-logs-kubernetes-1
    name: kubernetes-1
    revision: 2
    type: filestream
    use_output: default
    meta:
      package:
        name: kubernetes
        version: 1.19.1
    data_stream:
      namespace: default
    package_policy_id: kubernetes-1
    streams:
      - id: filestream-kubernetes.container_logs-kubernetes-1
        data_stream:
          dataset: kubernetes.container_logs
          type: logs
        paths:
          - '/var/log/containers/*${kubernetes.container.id}.log'
        prospector.scanner.symlinks: true
        parsers:
          - container:
              stream: all
              format: auto
signed:
  data: >-

and here is where I put it in the original (search filestream-container-logs-kubernetes-1):

    xpack.fleet.agentPolicies:
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 200
        is_default: true
        package_policies:
          - name: journald-1
            id: journald-1
            package:
              name: journald
          - name: kubernetes-1
            id: kubernetes-1
            package:
              name: kubernetes
              inputs:
              - id: filestream-container-logs-kubernetes-1
                  name: kubernetes-1
                  revision: 2
                  type: filestream
                  use_output: default
                  meta:
                    package:
                      name: kubernetes
                      version: 1.19.1
                  data_stream:
                    namespace: default
                  package_policy_id: kubernetes-1
                  streams:
                    - id: filestream-kubernetes.container_logs-kubernetes-1
                      data_stream:
                        dataset: kubernetes.container_logs
                        type: logs
                      paths:
                        - '/var/log/containers/*${kubernetes.container.id}.log'
                      prospector.scanner.symlinks: true
                      parsers:
                        - container:
                            stream: all
                            format: auto

Unfortunately, this does not work when pushing the changes through the operator. What am I doing wrong?

2

currently the error I am receiving is:

FATAL Error: [config validation of [xpack.fleet].agentPolicies.2.package_policies.1.inputs.0.streams.0.id]: definition for this key is missing

this is the current section I am employing under my policy:

          - name: kubernetes-1
            id: kubernetes-1
            package:
              name: kubernetes
            inputs:
              - id: filestream-container-logs-kubernetes-1
                name: kubernetes-1
                revision: 2
                type: filestream
                use_output: default
                meta:
                  package:
                    name: kubernetes
                    version: 1.19.1
                data_stream:
                  namespace: default
                package_policy_id: kubernetes-1
                streams:
                  - id: filestream-kubernetes.container_logs-kubernetes-1
                    data_stream:
                      dataset: kubernetes.container_logs
                      type: logs
                    paths:
                      - '/var/log/containers/*$${kubernetes.container.id}.log'
                    prospector.scanner.symlinks: true
                    parsers:
                      - container:
                          stream: all
                          format: auto
3

@Sheperd403 Did you managed to solve this issue? I'm facing the same problem.