File output plugin add some additional field in output file


(ADG) #1

i m having a issue with logstash
i m getting logs from multiple remote server via
filebeat (INPUT)
use csv filter (FILTER)
and
put it in
elasticsearch & file

the issue is logs which is stored in file having two extra entries before the line
i want same format of logs to be stored in file as in remote server

Input

input {
beats {
port => 5044
}
}

Filter

csv{
separator => ","
columns => ["DateTime" ,"hname" ,"uname" ,"cwd","fname","line"]
}

Output

file{ path => "/var/log/filename"
codec => line
}


(Magnus Bäck) #2

Use the json_lines codec in your file output.


(ADG) #3

i m not get it how to use it
i go through the various link
https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json_lines.html
but there is no example

the way i use this is like

codec => json_line

but logs say
message=>"Couldn't find any codec plugin named 'json_line


(Magnus Bäck) #4

json_lines, not json_line.


(ADG) #5

sorry my bad
i wrote wrong

but it is not give me solution i want

my input is
2018-04-17T18:40:17+0530,ip-10-0-1-14,root,loyal,quotes.txt,Delhi
2018-04-17T18:40:17+0530,ip-10-0-1-14,root,honest,quotes.txt,Delhi 06
2018-04-17T18:40:17+0530,ip-10-0-1-14,root,satisfied,quotes.txt,Delhi 53
2018-04-17T18:40:17+0530,ip-10-0-1-14,root,confident,quotes.txt,Mumbai 02
2018-04-17T18:40:17+0530,ip-10-0-1-14,root,success,quotes.txt,Mumbai

my output is like this
2018-04-17T18:41:09.227Z ip-10-0-1-14 2018-04-17T18:40:17+0530,ip-10-0-1-14,root,loyal,quotes.txt,Delhi
2018-04-17T18:41:09.227Z ip-10-0-1-14 2018-04-17T18:40:17+0530,ip-10-0-1-14,root,honest,quotes.txt,Delhi 06
2018-04-17T18:41:09.227Z ip-10-0-1-14 2018-04-17T18:40:17+0530,ip-10-0-1-14,root,satisfied,quotes.txt,Delhi 53
2018-04-17T18:41:09.227Z ip-10-0-1-14 2018-04-17T18:40:17+0530,ip-10-0-1-14,root,confident,quotes.txt,Mumbai 02
2018-04-17T18:41:09.227Z ip-10-0-1-14 2018-04-17T18:40:17+0530,ip-10-0-1-14,root,success,quotes.txt,Mumbai

i got these two field extra which i mention in bold character

file{ path => "/var/log/filename"
codec => line
}

from this

and
json_lines is not give me that


(Magnus Bäck) #6

Okay, rereading your original post I see what you mean. Use the line codec but configure the message format:

codec => line {
  format => "%{message}"
}

(ADG) #7

Thanks
its working
Thanks a lot


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.