My issue - I cant figure out how to make multiple output for logstash.conf
output {
#if "%beat%" in [type] # not working currently need to check...
if [type] =~ /^.*beat.*$/ # not working currently need to check...
{
elasticsearch {
hosts => "http://${HOSTNAME}:${PORT}"
user => "******"
password => "******"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} else {
elasticsearch {
hosts => "http://${HOSTNAME}:${PORT}"
user => "******"
password => "*******"
index => "%{type}-%{+YYYY.MM.dd}"
document_id => "%{sessionnumber}-%{seqnum}"
doc_as_upsert => "true"
}
}
# display to the console any of the output stream, this can be commented for performance reason
stdout { codec => rubydebug }
}
if I put only the beat part of the output, it works fine.
I have tried to do that, but same issue. I am thinking the only options would be pipelines.yml. There is not much documents regarding pipelines.yml for 5.6.1 version. for other latest versions, there are documents related to that but might not be relevant for 5.6.1 version.
When you say event, do you mean the input type section.
Input part consists 1) using jdbc getting details from Database (can't put here for security reason)
and 2) for beat part.
The input beat part is followed in the tutorials
beats{
port => 5044
}
The jdbc part works fine as long as the output part below as
output {
elasticsearch {
hosts => "http://${HOSTNAME}:${PORT}"
user => "******"
password => "********"
index => "%{type}-%{+YYYY.MM.dd}"
document_id => "%{sessionnumber}-%{seqnum}"
doc_as_upsert => "true"
}
# display to the console any of the output stream, this can be commented for performance reason
stdout { codec => rubydebug }
}
Similarly Beats works fine when the output is (with no changes to the input section which includes jdbc input part)
output {
elasticsearch {
hosts => "http://${HOSTNAME}:${PORT}"
user => "******"
password => "******"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
# display to the console any of the output stream, this can be commented for performance reason
stdout { codec => rubydebug }
}
input {
jdbc {
// hidden and correct
}
beats{
port => 5044
tags => "beats" # tried with and without this line.
}
}
output {
if [tags] =~ /beat/
{
# similar to above code for output beats
} else {
# similar to above code for output for jdbc
}
# display to the console any of the output stream, this can be commented for performance reason
stdout { codec => rubydebug }
}
.\logstash.bat -e 'output stdout with codec output {=> rubydebug'
Log files
[2018-03-18T21:27:01,147][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"***************/fb_apache/configuration"}
[2018-03-18T21:27:01,151][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>*******************/netflow/configuration"}
[2018-03-18T21:27:01,181][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"****************jruby/1.9/gems/x-pack-5.6.1-java/modules/arcsight/configuration"}
[2018-03-18T21:27:01,787][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, { at line 1, column 8 (byte 8) after output "}
[2018-03-18T21:27:02,134][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@localhost:9200/]}}
[2018-03-18T21:27:02,138][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-03-18T21:27:02,359][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@localhost:9200/"}
[2018-03-18T21:27:02,361][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2018-03-18T21:27:02,363][INFO ][logstash.pipeline ] Starting pipeline {"id"=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2}
[2018-03-18T21:27:02,367][INFO ][logstash.pipeline ] Pipeline .monitoring-logstash started
[2018-03-18T21:27:02,497][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-18T21:27:05,382][WARN ][logstash.agent ] stopping pipeline {:id=>".monitoring-logstash"}
On the otherhand.... I tried to comment out the if and else part and leave the stdout { codec => rubydebug } uncommented, I get a long list of input/output coming in and not fit to put in here......
output {
#if "%beat%" in [type]
#if [type] =~ /^.*beat.*$/ # not working currently need to check...
#if [tags] =~ /beat/
# commenting......remaining...
stdout { codec => rubydebug }
}
Let me know if I need to send some other short information?
Okay, so the type field contains e.g. "icmp", not anything related to beats. If you want to route all events from Packetbeat (or whatever is producing the data) you can use this:
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.