Logstash csv export => export of unwanted documents multiple times

Elastic Stack Logstash
1

Hello,

I'm currently experiencing weird behavior of my logstash pipeline. Maybe someone has an idea what I'm currently doing wrong.

What I'm trying to do
I use a logstash pipeline to extract data from elasticsearch and want to save it in an CSV file. For this, I'm using the elasticsearch input like this:

elasticsearch {
  docinfo => true
  docinfo_target => "[@metadata][doc]"
  hosts => ...
  index => "my_indices_*"
  query => ...  // long, long query to extract specific EVTX events, with hour of day condition
}

There is no filter condition, but an output like this:

csv {
  fields => ["@timestamp", "[event][code]", ... , "[@metadata][doc][_index]", "message"]
  csv_options => {"col_sep" => ";"}
  path => ...
}

What I experience
As I start logstash, the first thing I see is, that my pipeline is loaded and the exports begins, but(!) the export is started three times. I have no idea what exactly is happening. Might it be because of my three nodes? Is there some configuration I might miss?

After I've finished my export I open my CSV file, but it has far too many lines. Some lines are matching my desired format (mentioned in the output) and some don't. Some just have the timestamp and message field, but the message field is then in the second column, rather than in the last column as defined. Weird is also, because I also want my index to be shown ("[@metadata][doc][_index]") that the correct formatted lines have the correct index, and the ones which have only timestamp/message fields don't have only a blank value in this column. As I filter all the data out, which has no index, I get my correct number of data. which I can also see in Kibana. For me this is totally weird and confusing. Does someone have any explanation for this behavior?

Specification
Kibana: 8.4.1
Elasticsearch: 8.4.1
Logstash: 8.4.1

Thank you very much.

Best regards,
Dan.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.