Problem inserting Logstash process chain after setting up Filebeat->Elasticsearch->Kibana

Elastic Stack Logstash
1

Hi all,
This is my first time using Elasticsearch stack, I initially set up Filebeat, Elasticsearch, and Kibana to ingest the original free OSSEC alert.json file. However, the information was showing up in the message field (under data view Filebeat-*, the only data view I have) rather than with it's own indexed fields which made it hard to search and display.
From what I read, I needed Logstash to apply a template, so I sourced and modified a template to use and edited the output of Filebeat to point at Logstash and the output of Logstash to point at Elasticsearch.

output {
  # Option 1: Output to Elasticsearch
  elasticsearch {
    hosts => ["https://localhost:9200"] # Adjust to your Elasticsearch host
    # index => "ossec-alerts-%{+YYYY.MM.dd}"
    user => "logstash_writer"
    password => "logstash_writer"
    ssl_enabled => true
    #ssl_verification_mode => none
    ssl_certificate_authorities => ['/etc/logstash/certs/http_ca.crt']
    # truststore => "/path/to/truststore.jks"
    # truststore_password => "password"
    data_stream => true
    data_stream_type => "logs"
    data_stream_dataset => "ossec-alerts-logs"
    # ecs_compatibility => v8
    # ecs_compatibility => v1
  }

I also added a user in Elasticsearch called 'logstash_writer' and gave 'all' to cluster privileges, and as the only indices that showed up where the previous 'Filebeat-*', I gave that 'all' also.

Then I started Logstash and got the following:

"fields"=>{"type"=>"ossec_alert"}, "timestamp"=>"2025 Jul 29 01:15:20", "data_stream"=>{"type"=>"logs", "dataset"=>"ossec-alerts-logs", "namespace"=>"default"}}], :error=>{"type"=>"security_exception", "reason"=>"action [indices:data/write/bulk[s]] is unauthorized for user [logstash_writer] with effective roles [logstash_writer] on indices [logs-ossec-alerts-logs-default], this action is granted by the index privileges [create_doc,create,delete,index,write,all]"}}

it's long so I have pasted it below also:

"fields"=>{"type"=>"ossec_alert"}, "timestamp"=>"2025 Jul 29 01:15:20", "data_stream"=>{"type"=>"logs", "dataset"=>"ossec-alerts-logs", "namespace"=>"default"}}], :error=>{"type"=>"security_exception", "reason"=>"action [indices:data/write/bulk[s]] is unauthorized for user [logstash_writer] with effective roles [logstash_writer] on indices [logs-ossec-alerts-logs-default], this action is granted by the index privileges [create_doc,create,delete,index,write,all]"}}

I'm unsure how to continue as I don't have this indices to add privileges against.

I tried:

PUT _data_stream/logs-ossec-alerts-logs-default

in the dev console and then trying to add privileges to it but it doesn't show up as an index (also adding it as a dataview).

After a few days of trying things in desperation I'm a little confused how I progress, any help would be appreciate and I'll also provide what ever configuration data and log data that is needed - I think I have given enough to get this started, but I don't want to overload the initial top post.

Thankyou!

2

What are the permissions for the logstash user?

Also, your data stream name is not valid, the dataset name cannot have the character -.

Try to change to something like ossec_alerts_logs.

3

Hi leandrojmp,

Thankyou for the reply,

I replaced the - for a _.

When you say the logstash user - do you mean logstash_writer? :

role: logstash_writer
cluster privileges: 'all'
(i only have 1 indices showing in roles)
indices: Filebeat-* Privileges: 'all'

I can't seem to find the indices [logs-ossec_alerts_logs-default] in the error message. The error is the same except with the _'s now in the string. The -'s after 'log and before 'default' are added automatically by logstash it seems

I feel like I need to create a data_stream or some sort of template in Elasticsearch and tell it about the things it should be expecting from the logstash configuration, I'm just not sure exactly what that is ...

Regards.

4

You need to add permission for this user on logs-* or else it will not be able to create the data stream you want.

It needs at least the permissions create, creat_index and write on logs-*.

5

Thank you very much, I'd didn't realise I could type indices that didn't exist / don't show up in the drop down yet.

After adding that I was able to create the data view :slight_smile: