Export to CSV with all fields - Logstash 7.0.1

Elastic Stack Logstash
1

Hello,

Is there a way to configure the output to csv file to include all fields in the index by default so that each one does not have to be manually typed out?

Thanks,
Eric

2

Fields is a required option. It does not support %{[fieldname]} references but it will support environment variable substitution. However, it will treat a substituted environment variable as a string, even if it looks like an array. So the answer is no.

Except you can have logstash tell you what to type, which reduces it to copy and paste. Run a line of your data through this...

filter {
    ruby {
        code => '
            csvFields = []
            event.to_hash.each { |k, v|
                csvFields << k
            }
            event.set("[csvFields]", csvFields.to_s)
        '
    }
}
output { stdout { codec => plain { format => "%{csvFields}
" } } }
1 Like
3

Hi Badger,

Thanks for the tip! I tried running what you gave me, but I receive a result I was not expecting. The following is my conf file and output screenshot:

input{
  elasticsearch{
    hosts => ["10.100.1.2:9200"]
    index => "elastiflow-*"
    query => '{"query": { "bool": { "must": {"match_all": {}}}}}'
  }
} 

filter{
  ruby{
    code => '
      csvFields = []
      event.to_hash.each {|k,v|
        csvFields << k
      }
      even.set("[csvFields]", csvFields.to_s)
    '
  }
}

output{
  stdout {
    codec => plain { 
      format => "%{csvFields}"
    }
  }
}

Result:

Csv_header_extraction_errors
Csv_header_extraction_errors.png1584×218 6.47 KB

Do you know what went wrong?

Thanks,
Eric

4

That should be event.set. And you do want to use the newline that I embedded in the format of the codec.

5

Hi Badger,

I just made those corrections and it worked! Thanks for all your help.

Eric

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.