Hi Guys,
LS 6.3.x
ES 6.3.x
I have two logstashs servers across 2 datacentres connected over the internet using the lumberjack protocol. I used https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html as a referernce and confirmed using their generator input that events where recieved on the downstream server.
I then configured everything to actually send my real events over the logstash servers and eventually into elasticsearch.
According to Logstash in debug mode, the output acknowledged the event:
[2018-08-13T13:22:29,569][DEBUG][logstash.pipeline ] output received {"event"=>{"message"=>"2018-08-13T09:03:13.540Z {name=APP-02-PB} 2018-08-13 08:41:02,742 [B008] [Com.**.Web.Services.**.**] [(null)] [321] [INFO] [Trace] - Request Xml: <pkt><methodcall name=\"end\" timestamp=\"2018/08/13 08:41:02.489\" system=\"test1\"><auth login=\"loginuser\" password=\"test1\" /><call seq=\"ddfffdd-962c-4d96-8d88-ffffffdddd\" token=\"B008-SADFASDFasdfsadfsadfdsfa333242314dsafasdfdsafsdfsdafSDAFASDF+opX6wy0K8QoHB3spjhu7qYtsA+xiw==\" reference=\"Ref11\" did=\"73356\" offline=\"false\"><extinfo /></call></methodcall></pkt>", "@version"=>"1", "@timestamp"=>2018-08-13T13:22:28.673Z, "tags"=>["beats_input_codec_plain_applied"]}}
If I'm reading that right? I'm getting this for everything, including metricbeat, filebeat etc - anything that goes over the 2 logstash servers
Upstream server:
input {
beats {
client_inactivity_timeout => 1200
id => "LOG-01-PB"
port => 5001
host => "172.25.53.15"
}
}
output {
lumberjack {
flush_size => 300
idle_flush_time => 3
hosts => ["ip.addr"]
port => 5002
ssl_certificate => "/etc/logstash/ssl/lumberjack.cert"
}
lumberjack {
flush_size => 300
idle_flush_time => 3
hosts => ["ip.addr"]
port => 5003
ssl_certificate => "/etc/logstash/ssl/lumberjack.cert"
}
}
downstream server:
input {
beats {
client_inactivity_timeout => 1200
id => "LOG-001-LON"
port => 5001
}
}
input {
beats {
client_inactivity_timeout => 1200
port => 5002
ssl => true
ssl_certificate => "/etc/logstash/ssl/lumberjack.cert"
ssl_key => "/etc/logstash/ssl/lumberjack.key"
}
}
filter {
#nginx logs
if "nginx-logs" in [tags] {
}
#platform logs .net
else if "platform-logs" in [tags] {
mutate {
copy => { "@timestamp" => "beat_timestamp" }
}
date {
match => [ "time", "yyyy-MM-dd HH:mm:ss,SSS" ]
remove_field => [ "time" ]
}
}
#iis logs
else if "iis-logs" in [tags] {
}
# nodejs logs
else if "nodejs-logs" in [tags] {
}
output {
if "nodejs-logs" in [tags] {
elasticsearch {
hosts => ["ipadd.1", "ipadd.2", "ipadd.2"]
index => "nodejs-%{+YYYY.MM.dd}"
manage_template => false
user => logstash_writer
password => ''
}
}
else if "iis-logs" in [tags] {
elasticsearch {
hosts => ["ipadd.1", "ipadd.2", "ipadd.2"]
index => "filebeat-%{+YYYY.MM.dd}"
manage_template => false
user => logstash_writer
password => ''
}
}
else if "platform-logs" in [tags] {
elasticsearch {
hosts => ["ipadd.1", "ipadd.2", "ipadd.2"]
index => "platform-%{+YYYY.MM.dd}"
manage_template => false
user => logstash_writer
password => ''
}
}
I've removed some unneeded bits to save room!
Everything looks good, this above config works (without lumberjack) when the upstream server was doing all the processing and sending events direct to elasticsearch but the logstash server couldn't handle it when it was busy so i was trying to just send the events to the local logostash server then onto the more powerful (downstream) LS server and then on to elasticsearch from there. The logstash servers in DC2 are being used by beats locally to that DC with no problems whatsoever. The servers are all pretty much the same config wise across both DC's
Can anyone help me figure out where I can go next? Which logging bits do i need to turn to debug on ES? Neither LS or ES are showing any errors with logging set to INFO.