# Struggling to set up multiple namespaces for a single Beats data stream

**URL:** <https://discuss.elastic.co/t/struggling-to-set-up-multiple-namespaces-for-a-single-beats-data-stream/383212>\
**Category:** Beats\
**Tags:** packetbeat, user-experience, datastreams\
**Created:** [November 4, 2025, 1:14pm UTC](https://discuss.elastic.co/t/struggling-to-set-up-multiple-namespaces-for-a-single-beats-data-stream/383212 "2025-11-04T13:14:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dot-mike](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dot-mike/32/143339_2.png) [@dot-mike](https://discuss.elastic.co/u/dot-mike)\
**Post date:** [November 4, 2025, 1:14pm UTC](https://discuss.elastic.co/t/struggling-to-set-up-multiple-namespaces-for-a-single-beats-data-stream/383212/1 "2025-11-04T13:14:33Z")

</div>

Dear community

In Elastic Fleet you can set a namespace for an integration so you can have multiple data streams for a single type of index template without having to duplicate settings (ILM policy, index template, components, mappings). With Beats I don't see how it's possible do such thing.

So I'm here sharing my frustation and confusion over the fact that it's not easy to setup multiple namespaces for a single type of data stream. **I eventually got it working by running the `setup` command twice, one for each of the namespaces. But I was hoping to avoid this!**

For example, you want to setup **Packetbeat** to ingest DNS data and separate the datasets given which network the data arrives from. So you configure something in `packetbeat.yml` like this to add fields to the doc to separate them:

```auto
fields:
  data_stream.type: packetbeat
  data_stream.dataset: dns
  event.dataset: dns

processors:
  - add_host_metadata: ~
  - add_fields:
      when.or:
        - network.source.ip: "192.168.0.0/24"
        - network.destination.ip: "192.168.0.0/24"
      target: ""
      fields:
        data_stream.namespace: "clients"

  - add_fields:
      when.not.has_fields: ["data_stream.namespace"]
      target: ""
      fields:
        data_stream.namespace: "default"

```

And for output to Elasticsearch you set `output.elasticsearch.index: "%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}"`

This should make the data go into the data stream `packetbeat-dns-clients` if the processor condition for network matches or to the default `packetbeat-dns-default`

Now comes the issue, how would you setup the packetbeat to create the necessary datastream and index templates? There is only one way, you have to run the setup command twice, one for each of the namespaces:

```auto
  
./packetbeat setup -e -c packetbeat.yml \
  --index-management \
  -E setup.ilm.enabled=true \
  -E setup.template.name="packetbeat-dns-clients" \
  -E setup.template.pattern="packetbeat-dns-clients" \
  -E setup.ilm.policy_name="packetbeat-dns-clients" \
  -E output.elasticsearch.index="packetbeat-dns-clients"

./packetbeat setup -e -c packetbeat.yml \
  --index-management \
  -E setup.ilm.enabled=true \
  -E setup.template.name="packetbeat-dns-default" \
  -E setup.template.pattern="packetbeat-dns-default" \
  -E setup.ilm.policy_name="packetbeat-dns-default" \
  -E output.elasticsearch.index="packetbeat-dns-default"

```

If you try to create a more generic index template that supports namespaces, you’ll get an error that the data stream can’t be found:

```auto
$ ./packetbeat setup -e -c packetbeat.yml \
  --index-management \
  -E setup.ilm.enabled=true \
  -E setup.template.name="packetbeat-dns" \
  -E setup.template.pattern="packetbeat-dns-*" \
  -E setup.ilm.policy_name="packetbeat-dns"

{"log.level":"error","@timestamp":"2025-11-04T11:58:30.899Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError","file.name":"instance/beat.go","file.line":1594},"message":"Exiting: error loading template: failed to put data stream: could not put data stream: 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [packetbeat-dns]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [packetbeat-dns]\"},\"status\":400}. Response body: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [packetbeat-dns]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [packetbeat-dns]\"},\"status\":400}","service.name":"packetbeat","ecs.version":"1.6.0"} Exiting: error loading template: failed to put data stream: could not put data stream: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [packetbeat-dns]"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [packetbeat-dns]"},"status":400}. Response body: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [packetbeat-dns]"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [packetbeat-dns]"},"status":400}

```

Sharing my full working config below:

```auto
packetbeat.interfaces.device: eth0
packetbeat.interfaces.internal_networks:
  - private

# =================================== Flows ====================================

packetbeat.flows:
  enabled: true
  timeout: 30s
  period: 10s

# =========================== Transaction protocols ============================

packetbeat.protocols:
- type: icmp
  enabled: true

- type: dns
  enabled: true
  ports: [53]
  include_authorities: true
  include_additionals: true

# ================================== Template ==================================
setup.ilm.enabled: false
setup.template.enabled: false
packetbeat.overwrite_pipelines: false

# Elasticsearch template settings
setup.template.settings:
  index:
    number_of_shards: 1

# ================================== General ===================================

name: <hostname>
tags: [packetbeat]

packetbeat.ignore_outgoing: true

fields_under_root: true
fields:
  data_stream.type: packetbeat
  data_stream.dataset: dns
  event.dataset: dns

# =================================== Kibana ===================================
setup.kibana:
  host: "https://<kibana_host>:443"
  username: "packetbeat_dns"
  password: "<hidden>"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:
  
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  hosts: ["<elastic lb>"]
  protocol: "https"
  ssl:
    enabled: true
    ca_trusted_fingerprint: ""

  username: "packetbeat_dns"
  password: "<hidden>"

  index: "%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}"

  # Pipeline to route events to protocol pipelines.
  pipeline: "packetbeat-%{[agent.version]}-routing"

# ================================= Processors =================================

processors:
  - add_host_metadata: ~
  - add_fields:
      when.or:
        - network.source.ip: "192.168.0.0/24"
        - network.destination.ip: "192.168.0.0/24"
      target: ""
      fields:
        data_stream.namespace: "client"

  - add_fields:
      when.not.has_fields: ["data_stream.namespace"]
      target: ""
      fields:
        data_stream.namespace: "default"

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 4, 2025, 2:42pm UTC](https://discuss.elastic.co/t/struggling-to-set-up-multiple-namespaces-for-a-single-beats-data-stream/383212/2 "2025-11-04T14:42:21Z")

</div>

Hi @dot-mike ... Nicely Done... and sooo close try this

What version?

I think you will find this will work ...

> [@dot-mike](#):
>
> ```auto
> ./packetbeat setup -e -c packetbeat.yml \
> --index-management \
> -E setup.ilm.enabled=true \
> -E setup.template.name="packetbeat-dns" \
> -E setup.template.pattern="packetbeat-dns*" \. <<< Take out the -
> -E setup.ilm.policy_name="packetbeat-dns"
> 
> ```

As to why this works and the other does not, I do not recall at the moment (and a little rushed this am to go digging) it has something to do when you have `-*` it does not know what initial data stream to create as part of the process named. It will create an initial data stream named `packetbeat-dns` (whcih does match the `-*` but does match the just `*` pattern) as the initial but will / should also work for the different namespaces you want.

I tested with filebeat seems to work.

---

<div class="post-metadata">

**Author:** ![dot-mike](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dot-mike/32/143339_2.png) [@dot-mike](https://discuss.elastic.co/u/dot-mike)\
**Post date:** [November 4, 2025, 4:27pm UTC](https://discuss.elastic.co/t/struggling-to-set-up-multiple-namespaces-for-a-single-beats-data-stream/383212/3 "2025-11-04T16:27:54Z")

</div>

Hi and many thanks for the quick reply @stephenb ! Always nice to see you around. I will try this and see if it works