# Auditbeat \>=8, logstash, and elasticsearch data stream

**URL:** <https://discuss.elastic.co/t/auditbeat-8-logstash-and-elasticsearch-data-stream/335357>\
**Category:** Logstash\
**Created:** [June 6, 2023, 1:51pm UTC](https://discuss.elastic.co/t/auditbeat-8-logstash-and-elasticsearch-data-stream/335357 "2023-06-06T13:51:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mike\_Williams](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_williams/32/81420_2.png) [@Mike\_Williams](https://discuss.elastic.co/u/Mike_Williams)\
**Post date:** [June 6, 2023, 1:51pm UTC](https://discuss.elastic.co/t/auditbeat-8-logstash-and-elasticsearch-data-stream/335357/1 "2023-06-06T13:51:32Z")

</div>

Hey,

I'm preparing to upgrade a set of auditbeat agents from 7.17 to 8.something.  
Clients are not allowed to talk directly to elasticsearch, all messages go through logstash.

More than happy with the requirement to use data streams in auditbeat version 8.  
My problem is that the messages from auditbeat don't appear to have fields set to enable the elasticsearch output plugin for logstash to actually output to a data stream, or more precisely output them to the correctly named data stream.

The logstash output for data streams is extremely simple.

```auto
output {
  elasticsearch {
... ssl stuff, auth stuff, hosts ...
    data_stream => true
  }

```

The messages from filebeats have the data\_stream\_dataset/data\_stream\_namespace/data\_stream\_type fields set, so pass right though and go to the right place.

By just outputting the messages from auditbeat to an elasticsearch data stream with the above logstash they appear in logs-generic-default, not auditbeat-version.

What field(s) do I need to set for the data\_stream output to output to the data stream name expected by the auditbeat index template(s)? auditbeat-8.6.2 for example.  
I don't know how to map `auditbeat-8.6.2` into `type-namespace-dataset`.

Thanks

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 7, 2023, 3:02am UTC](https://discuss.elastic.co/t/auditbeat-8-logstash-and-elasticsearch-data-stream/335357/2 "2023-06-07T03:02:45Z")

</div>

Hi @Mike_Williams

Perhaps a little confusion on my part.

If you run auditbeat 8.x against elasticsearch 8.x it will create a data stream

So in my case I am running auditbeat and elasticsearch 8.8 when I run auditbeat directly to elasticsearch it creates a Data Stream name `auditbeat-8.8.0` This is the proper name for the data stream with beats and I would recommend leaving it that way.

The data stream will show up in Kibana -\> Stack Management -\> Index Management Data Streams  
`auditbeat-8.8.0`  
There will be a backing index named something like.  
`.ds-auditbeat-8.8.0-2023.06.07-000001`

Use this logstash config to make Auditbeat -\> Logstash -\> Elasticsearch  
the same as Auditbeat -\> Elasticsearch  
[This](https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html) is the equivalent of passthrough

```auto
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create" 
      pipeline => "%{[@metadata][pipeline]}" 
    }
  } else {
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create"
    }
  }
}

```

Hopefully that is what you are looking for....oh a don't forget to run setup first while auditbeat is pointing at elasticsearch, then switch the output to logstash and run.

---

<div class="post-metadata">

**Author:** ![Mike\_Williams](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_williams/32/81420_2.png) [@Mike\_Williams](https://discuss.elastic.co/u/Mike_Williams)\
**Post date:** [June 7, 2023, 9:37am UTC](https://discuss.elastic.co/t/auditbeat-8-logstash-and-elasticsearch-data-stream/335357/3 "2023-06-07T09:37:50Z")

</div>

Hey @stephenb ,

Thanks for the response, you actually hit the nail on the head with `action`.  
Later on yesterday, once I fixed some unrelated issues with my test environment, I stumbled across the requirement for op\_type=create for indexing into indices with elasticsearch. After adding the option within the logstash pipeline to set `action => index` but override it to `action => create` for these auditbeat 8 messages everything worked as expected.

Cheers

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2023, 9:37am UTC](https://discuss.elastic.co/t/auditbeat-8-logstash-and-elasticsearch-data-stream/335357/4 "2023-07-05T09:37:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.