Get message content without message tag in logstash

Elastic Stack Logstash
1

Hello everyone,
This is the config file i used.

input {
    syslog {
	   port => 514
	}
}
filter{
mutate { 
	remove_field => ["facility_label","facility","@version","priority","tags","host","timestamp","severity_label","severity","logsource"]
	split => { "message" => "," }
 }
 }
output {
    stdout {}
    file {
        path => "E:/sample.txt"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
	}
}

This is the output i got

{"message":["1","2019/01/06 22:26:23","001606074152","TRAFFIC","end","1","2019/01/06 22:26:23","103.62.17.6","111.93.8.19","103.62.17.6","192.168.11.16","WAN-LAN","","","ms-rdp","vsys1","untrust","trust","ethernet1/3","ethernet1/4","Kiwi Syslog","2019/01/06 22:26:23","53938","1","62225","3389","62225","3389","0x400050","tcp","allow","3637","1644","1993","17","2019/01/06 22:26:05","3","any","0","361598785","0x0","IN","IN","0","9","8","tcp-rst-from-client","0","0","0","0","","PA-200","from-policy "]}

but i want output as

"1","2019/01/06 22:26:23","001606074152","TRAFFIC","end","1","2019/01/06 22:26:23","103.62.17.6","111.93.8.19","103.62.17.6","192.168.11.16","WAN-LAN","","","ms-rdp","vsys1","untrust","trust","ethernet1/3","ethernet1/4","Kiwi Syslog","2019/01/06 22:26:23","53938","1","62225","3389","62225","3389","0x400050","tcp","allow","3637","1644","1993","17","2019/01/06 22:26:05","3","any","0","361598785","0x0","IN","IN","0","9","8","tcp-rst-from-client","0","0","0","0","","PA-200","from-policy "

still i can't declare field names n pattern of output.because that may vary.output can be db or csv.help me

2

This is a guess as I have never had to do this myself...

Maybe use the CSV output

3

can i use csv output without mentioning fields

4

Right, looks like you need to define them. I though I had read somewhere that fields could be auto generated. Must have been some other CSV tool...

5

i need it in log stash without using any other tool

6

More guess work, sorry...

You might be able to achieve that by changing the codec of the file output

Default value is "json_lines"

7

which value i should place for codec to get that output. i tried with json
codec => json

8

The default codec for a file output is json_lines. You could try using a plain codec with a message format. Use a literal embedded newline to tell it to append a newline to the message.

output { file { path => "/path/to/file.txt" codec => plain { format => "%{message}
" } } }
9

how can we add one more field along with message

codec => line { format => "%{[message]}","%{[@timestamp]}"} i tried this i got error

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 16, column 43 (byte 357) after output {\n stdout {}\n file {\n path => "E:/testpaloaltomessage.txt" \n\t\tcodec => line { format => "%{[message]}"", :backtrace=>["E:/softwares/logstash-6.5.4/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "E:/softwares/logstash-6.5.4/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "E:/softwares/logstash-6.5.4/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "E:/softwares/logstash-6.5.4/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in `initialize'", "E:/softwares/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "E:/softwares/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "E:/softwares/logstash-6.5.4/logstash-core/lib/logstash/pipeline_action/create.rb:42:in `block in execute'", "E:/softwares/logstash-6.5.4/logstash-core/lib/logstash/agent.rb:92:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "E:/softwares/logstash-6.5.4/logstash-core/lib/logstash/agent.rb:92:in `exclusive'", "E:/softwares/logstash-6.5.4/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'", "E:/softwares/logstash-6.5.4/logstash-core/lib/logstash

10

If you want to prepend the timestamp use

output { stdout { codec => plain { format => "%{@timestamp} %{message}
" } } }
1 Like
11

Thank u Mr.Badger

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.