Struggling to set up multiple namespaces for a single Beats data stream

Elastic Stack Beats
, ,
1

Dear community

In Elastic Fleet you can set a namespace for an integration so you can have multiple data streams for a single type of index template without having to duplicate settings (ILM policy, index template, components, mappings). With Beats I don't see how it's possible do such thing.

So I'm here sharing my frustation and confusion over the fact that it's not easy to setup multiple namespaces for a single type of data stream. I eventually got it working by running the setup command twice, one for each of the namespaces. But I was hoping to avoid this!

For example, you want to setup Packetbeat to ingest DNS data and separate the datasets given which network the data arrives from. So you configure something in packetbeat.yml like this to add fields to the doc to separate them:

fields:
  data_stream.type: packetbeat
  data_stream.dataset: dns
  event.dataset: dns

processors:
  - add_host_metadata: ~
  - add_fields:
      when.or:
        - network.source.ip: "192.168.0.0/24"
        - network.destination.ip: "192.168.0.0/24"
      target: ""
      fields:
        data_stream.namespace: "clients"

  - add_fields:
      when.not.has_fields: ["data_stream.namespace"]
      target: ""
      fields:
        data_stream.namespace: "default"

And for output to Elasticsearch you set output.elasticsearch.index: "%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}"

This should make the data go into the data stream packetbeat-dns-clients if the processor condition for network matches or to the default packetbeat-dns-default

Now comes the issue, how would you setup the packetbeat to create the necessary datastream and index templates? There is only one way, you have to run the setup command twice, one for each of the namespaces:

  
./packetbeat setup -e -c packetbeat.yml \
  --index-management \
  -E setup.ilm.enabled=true \
  -E setup.template.name="packetbeat-dns-clients" \
  -E setup.template.pattern="packetbeat-dns-clients" \
  -E setup.ilm.policy_name="packetbeat-dns-clients" \
  -E output.elasticsearch.index="packetbeat-dns-clients"


./packetbeat setup -e -c packetbeat.yml \
  --index-management \
  -E setup.ilm.enabled=true \
  -E setup.template.name="packetbeat-dns-default" \
  -E setup.template.pattern="packetbeat-dns-default" \
  -E setup.ilm.policy_name="packetbeat-dns-default" \
  -E output.elasticsearch.index="packetbeat-dns-default"

If you try to create a more generic index template that supports namespaces, you’ll get an error that the data stream can’t be found:

$ ./packetbeat setup -e -c packetbeat.yml \
  --index-management \
  -E setup.ilm.enabled=true \
  -E setup.template.name="packetbeat-dns" \
  -E setup.template.pattern="packetbeat-dns-*" \
  -E setup.ilm.policy_name="packetbeat-dns"


{"log.level":"error","@timestamp":"2025-11-04T11:58:30.899Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError","file.name":"instance/beat.go","file.line":1594},"message":"Exiting: error loading template: failed to put data stream: could not put data stream: 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [packetbeat-dns]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [packetbeat-dns]\"},\"status\":400}. Response body: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [packetbeat-dns]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [packetbeat-dns]\"},\"status\":400}","service.name":"packetbeat","ecs.version":"1.6.0"} Exiting: error loading template: failed to put data stream: could not put data stream: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [packetbeat-dns]"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [packetbeat-dns]"},"status":400}. Response body: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [packetbeat-dns]"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [packetbeat-dns]"},"status":400}

Sharing my full working config below:

packetbeat.interfaces.device: eth0
packetbeat.interfaces.internal_networks:
  - private

# =================================== Flows ====================================

packetbeat.flows:
  enabled: true
  timeout: 30s
  period: 10s

# =========================== Transaction protocols ============================

packetbeat.protocols:
- type: icmp
  enabled: true

- type: dns
  enabled: true
  ports: [53]
  include_authorities: true
  include_additionals: true

# ================================== Template ==================================
setup.ilm.enabled: false
setup.template.enabled: false
packetbeat.overwrite_pipelines: false

# Elasticsearch template settings
setup.template.settings:
  index:
    number_of_shards: 1

# ================================== General ===================================

name: <hostname>
tags: [packetbeat]

packetbeat.ignore_outgoing: true

fields_under_root: true
fields:
  data_stream.type: packetbeat
  data_stream.dataset: dns
  event.dataset: dns

# =================================== Kibana ===================================
setup.kibana:
  host: "https://<kibana_host>:443"
  username: "packetbeat_dns"
  password: "<hidden>"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:
  
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  hosts: ["<elastic lb>"]
  protocol: "https"
  ssl:
    enabled: true
    ca_trusted_fingerprint: ""

  username: "packetbeat_dns"
  password: "<hidden>"

  index: "%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}"

  # Pipeline to route events to protocol pipelines.
  pipeline: "packetbeat-%{[agent.version]}-routing"

# ================================= Processors =================================

processors:
  - add_host_metadata: ~
  - add_fields:
      when.or:
        - network.source.ip: "192.168.0.0/24"
        - network.destination.ip: "192.168.0.0/24"
      target: ""
      fields:
        data_stream.namespace: "client"

  - add_fields:
      when.not.has_fields: ["data_stream.namespace"]
      target: ""
      fields:
        data_stream.namespace: "default"
2

Hi @dot-mike ... Nicely Done... and sooo close try this

What version?

I think you will find this will work ...

As to why this works and the other does not, I do not recall at the moment (and a little rushed this am to go digging) it has something to do when you have -* it does not know what initial data stream to create as part of the process named. It will create an initial data stream named packetbeat-dns (whcih does match the -* but does match the just * pattern) as the initial but will / should also work for the different namespaces you want.

I tested with filebeat seems to work.

1 Like
3

Hi and many thanks for the quick reply @stephenb ! Always nice to see you around. I will try this and see if it works

1 Like