Output to json saving only specific fields

Elastic Stack Logstash
1

Hello everyone

I am new logstash and I am sorry if the question is so basic.

I have been saving my syslog into a csv file, as follow:

 csv {
    path => "C:\path\Desktop\Adib\adib-syslogs-%{+yyyy.MM.dd}.csv"
    csv_options => {
        "write_headers" => true
        "headers" => ["@timestamp", "message", "name","userID", "deviceAction"]

}
    fields => ["@timestamp", "message", "name", "userID", "deviceAction"]
 }

but the output is frankly something I can't take, because for each entry, it save the headers. I have been reading here that there is no option to save the header only once, so I want to start saving the output to a json file.

following the logstash documentation, I found that I can use this command:

output {
 file {
   path => ...
   codec => json
 }
}

but I am a bit confused about the next step.

as you can see in my csv configuration, I am saving specific fields, as all the other I don't need them and I want to save space on my VM

is there a way how I can output my logstash logs to a json and specify only the fields I want to write to the file?

thank you very much for any help

2

You could try using a prune filter with a whitelist_names option.

3

thank you so much for your reply. I am using grafana to visualise the data, and with the white names I won't see all the fields in grafana as I will see only the whitelisted. Is there any option to just filter the fields to save in json while I am still able to visualise the logs fully in grafana?

4

Use pipeline-to-pipeline communication with a forked path pattern.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.