Auditbeat >=8, logstash, and elasticsearch data stream

[Mike_Williams]

Hey,

I'm preparing to upgrade a set of auditbeat agents from 7.17 to 8.something.
Clients are not allowed to talk directly to elasticsearch, all messages go through logstash.

More than happy with the requirement to use data streams in auditbeat version 8.
My problem is that the messages from auditbeat don't appear to have fields set to enable the elasticsearch output plugin for logstash to actually output to a data stream, or more precisely output them to the correctly named data stream.

The logstash output for data streams is extremely simple.

output {
  elasticsearch {
... ssl stuff, auth stuff, hosts ...
    data_stream => true
  }

The messages from filebeats have the data_stream_dataset/data_stream_namespace/data_stream_type fields set, so pass right though and go to the right place.

By just outputting the messages from auditbeat to an elasticsearch data stream with the above logstash they appear in logs-generic-default, not auditbeat-version.

What field(s) do I need to set for the data_stream output to output to the data stream name expected by the auditbeat index template(s)? auditbeat-8.6.2 for example.
I don't know how to map auditbeat-8.6.2 into type-namespace-dataset.

Thanks

[stephenb (Stephen Brown)]

Hi @Mike_Williams

Perhaps a little confusion on my part.

If you run auditbeat 8.x against elasticsearch 8.x it will create a data stream

So in my case I am running auditbeat and elasticsearch 8.8 when I run auditbeat directly to elasticsearch it creates a Data Stream name auditbeat-8.8.0 This is the proper name for the data stream with beats and I would recommend leaving it that way.

The data stream will show up in Kibana -> Stack Management -> Index Management Data Streams
auditbeat-8.8.0
There will be a backing index named something like.
.ds-auditbeat-8.8.0-2023.06.07-000001

Use this logstash config to make Auditbeat -> Logstash -> Elasticsearch
the same as Auditbeat -> Elasticsearch
This is the equivalent of passthrough

input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create" 
      pipeline => "%{[@metadata][pipeline]}" 
    }
  } else {
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create"
    }
  }
}

Hopefully that is what you are looking for....oh a don't forget to run setup first while auditbeat is pointing at elasticsearch, then switch the output to logstash and run.

[Mike_Williams]

Hey @stephenb ,

Thanks for the response, you actually hit the nail on the head with action.
Later on yesterday, once I fixed some unrelated issues with my test environment, I stumbled across the requirement for op_type=create for indexing into indices with elasticsearch. After adding the option within the logstash pipeline to set action => index but override it to action => create for these auditbeat 8 messages everything worked as expected.

Cheers