# Automating Elastic Agent Policies

**URL:** <https://discuss.elastic.co/t/automating-elastic-agent-policies/357246>\
**Category:** Elastic Agent\
**Created:** [April 11, 2024, 10:22pm UTC](https://discuss.elastic.co/t/automating-elastic-agent-policies/357246 "2024-04-11T22:22:19Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sheperd403](https://avatars.discourse-cdn.com/v4/letter/s/6bbea6/32.png) [@Sheperd403](https://discuss.elastic.co/u/Sheperd403)\
**Post date:** [April 11, 2024, 10:22pm UTC](https://discuss.elastic.co/t/automating-elastic-agent-policies/357246/1 "2024-04-11T22:22:19Z")

</div>

I have been experimenting with attempting to automate the ECK elastic agent policy in fleet. Currently, my fleet server set up is this with these two policies defined:

```auto
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: ${kibana_name}
spec:
  version: 8.4.1
  count: 1
  elasticsearchRef:
    name: eck-elasticsearch-cluster
  config:
    xpack.fleet.agents.fleet_server.hosts: "${fleet_server_urls}"
    xpack.fleet.agents.elasticsearch.hosts: "${fleet_elasticsearch_urls}"
    xpack.fleet.agents.pollingRequestTimeout: 30000
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
      - name: kubernetes
        version: 1.19.1
      - name: journald
        version: 1.1.0
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        namespace: default
        is_managed: false
        is_default: false
        is_default_fleet_server: true
        unenroll_timeout: 200
        package_policies:
        - name: fleet_server_1
          id: fleet_server_1
          package:
            name: fleet_server
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 200
        is_default: true
        package_policies:
          - name: journald-1
            id: journald-1
            package:
              name: journald
          - name: kubernetes-1
            id: kubernetes-1
            package:
              name: kubernetes

```

Now, once this is created. We go into the UI, and update the kubernetes integration under this ECK Elastic Policy to only pull k8s container logs:

 ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/4/0/40b828699de3b1242012a437691604a10f7e525c.png)

Now, I want to automate this by adding it into my yaml. so I thought I could view the yaml for the Elasticsearch policy, and add the inputs under the package field in my original yaml aboce. Here is the elastic agent policy yaml I can pull from the UI once I manually edited it to my liking:

```auto
id: eck-agent
revision: 4
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://fleet.es.test.domain.com:443'
fleet:
  hosts:
    - >-
      https://eck-fleet-server.test.domain.com:443
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices:
        - names:
            - logs-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.cloud_defend-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.cloudbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.cloudbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat_input-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat_input-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_elastic_collector-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_elastic_symbolizer-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_host_agent-default
          privileges:
            - auto_configure
            - create_doc
    _elastic_agent_checks:
      cluster:
        - monitor
    journald-1:
      indices:
        - names:
            - logs-*-*
          privileges:
            - auto_configure
            - create_doc
    kubernetes-1:
      indices:
        - names:
            - logs-kubernetes.container_logs-default
          privileges:
            - auto_configure
            - create_doc
agent:
  download:
    sourceURI: 'https://artifacts.elastic.co/downloads/'
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: true
  features: {}
inputs:
  - id: journald-logs-journald-1
    name: journald-1
    revision: 1
    type: journald
    use_output: default
    meta:
      package:
        name: journald
        version: 1.1.0
    data_stream:
      namespace: default
    package_policy_id: journald-1
    streams:
      - id: journald-journald.logs-journald-1
        data_stream:
          dataset: logs
        condition: '${host.platform} == ''linux'''
        tags:
          - journald-log
        processors:
          - convert:
              tag: journald-to-ecs
              mode: rename
              ignore_missing: true
              fields:
                - from: message_id
                  to: event.code
                - from: journald.code.file
                  to: log.origin.file.name
                - from: journald.code.line
                  to: log.origin.file.line
                - from: journald.code.func
                  to: log.origin.function
                - from: syslog.pid
                  to: log.syslog.procid
                - from: syslog.identifier
                  to: log.syslog.appname
          - drop_fields:
              ignore_missing: true
              fields:
                - syslog
                - container.id_truncated
  - id: filestream-container-logs-kubernetes-1
    name: kubernetes-1
    revision: 2
    type: filestream
    use_output: default
    meta:
      package:
        name: kubernetes
        version: 1.19.1
    data_stream:
      namespace: default
    package_policy_id: kubernetes-1
    streams:
      - id: filestream-kubernetes.container_logs-kubernetes-1
        data_stream:
          dataset: kubernetes.container_logs
          type: logs
        paths:
          - '/var/log/containers/*${kubernetes.container.id}.log'
        prospector.scanner.symlinks: true
        parsers:
          - container:
              stream: all
              format: auto
signed:
  data: >-

```

and here is where I put it in the original (search filestream-container-logs-kubernetes-1):

```auto
    xpack.fleet.agentPolicies:
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 200
        is_default: true
        package_policies:
          - name: journald-1
            id: journald-1
            package:
              name: journald
          - name: kubernetes-1
            id: kubernetes-1
            package:
              name: kubernetes
              inputs:
              - id: filestream-container-logs-kubernetes-1
                  name: kubernetes-1
                  revision: 2
                  type: filestream
                  use_output: default
                  meta:
                    package:
                      name: kubernetes
                      version: 1.19.1
                  data_stream:
                    namespace: default
                  package_policy_id: kubernetes-1
                  streams:
                    - id: filestream-kubernetes.container_logs-kubernetes-1
                      data_stream:
                        dataset: kubernetes.container_logs
                        type: logs
                      paths:
                        - '/var/log/containers/*${kubernetes.container.id}.log'
                      prospector.scanner.symlinks: true
                      parsers:
                        - container:
                            stream: all
                            format: auto

```

Unfortunately, this does not work when pushing the changes through the operator. What am I doing wrong?

---

<div class="post-metadata">

**Author:** ![Sheperd403](https://avatars.discourse-cdn.com/v4/letter/s/6bbea6/32.png) [@Sheperd403](https://discuss.elastic.co/u/Sheperd403)\
**Post date:** [April 13, 2024, 5:41am UTC](https://discuss.elastic.co/t/automating-elastic-agent-policies/357246/2 "2024-04-13T05:41:52Z")

</div>

currently the error I am receiving is:

> FATAL Error: [config validation of [xpack.fleet].agentPolicies.2.package\_policies.1.inputs.0.streams.0.id]: definition for this key is missing

this is the current section I am employing under my policy:

```auto
          - name: kubernetes-1
            id: kubernetes-1
            package:
              name: kubernetes
            inputs:
              - id: filestream-container-logs-kubernetes-1
                name: kubernetes-1
                revision: 2
                type: filestream
                use_output: default
                meta:
                  package:
                    name: kubernetes
                    version: 1.19.1
                data_stream:
                  namespace: default
                package_policy_id: kubernetes-1
                streams:
                  - id: filestream-kubernetes.container_logs-kubernetes-1
                    data_stream:
                      dataset: kubernetes.container_logs
                      type: logs
                    paths:
                      - '/var/log/containers/*$${kubernetes.container.id}.log'
                    prospector.scanner.symlinks: true
                    parsers:
                      - container:
                          stream: all
                          format: auto

```

---

<div class="post-metadata">

**Author:** ![Bornegrim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bornegrim/32/137311_2.png) [@Bornegrim](https://discuss.elastic.co/u/Bornegrim)\
**Post date:** [September 4, 2024, 1:39pm UTC](https://discuss.elastic.co/t/automating-elastic-agent-policies/357246/3 "2024-09-04T13:39:07Z")

</div>

@Sheperd403 Did you managed to solve this issue? I'm facing the same problem.