LS -> LS (Lumberjack) - documents not being indexed?

Hi Guys,

LS 6.3.x
ES 6.3.x

I have two logstashs servers across 2 datacentres connected over the internet using the lumberjack protocol. I used https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html as a referernce and confirmed using their generator input that events where recieved on the downstream server.

I then configured everything to actually send my real events over the logstash servers and eventually into elasticsearch.

According to Logstash in debug mode, the output acknowledged the event:

[2018-08-13T13:22:29,569][DEBUG][logstash.pipeline        ] output received {"event"=>{"message"=>"2018-08-13T09:03:13.540Z {name=APP-02-PB} 2018-08-13 08:41:02,742 [B008] [Com.**.Web.Services.**.**] [(null)] [321] [INFO] [Trace] - Request Xml: <pkt><methodcall name=\"end\" timestamp=\"2018/08/13 08:41:02.489\" system=\"test1\"><auth login=\"loginuser\" password=\"test1\" /><call seq=\"ddfffdd-962c-4d96-8d88-ffffffdddd\" token=\"B008-SADFASDFasdfsadfsadfdsfa333242314dsafasdfdsafsdfsdafSDAFASDF+opX6wy0K8QoHB3spjhu7qYtsA+xiw==\" reference=\"Ref11\" did=\"73356\" offline=\"false\"><extinfo /></call></methodcall></pkt>", "@version"=>"1", "@timestamp"=>2018-08-13T13:22:28.673Z, "tags"=>["beats_input_codec_plain_applied"]}}

If I'm reading that right? I'm getting this for everything, including metricbeat, filebeat etc - anything that goes over the 2 logstash servers

Upstream server:

input {
  beats {
client_inactivity_timeout => 1200
id => "LOG-01-PB"
port => 5001
host => "172.25.53.15"
  }
}


output {

   lumberjack {
 flush_size => 300
 idle_flush_time => 3
 hosts => ["ip.addr"]
 port => 5002
 ssl_certificate => "/etc/logstash/ssl/lumberjack.cert"
   }

   lumberjack {
 flush_size => 300
 idle_flush_time => 3
 hosts => ["ip.addr"]
 port => 5003
 ssl_certificate => "/etc/logstash/ssl/lumberjack.cert"
   }

}

downstream server:

input {
  beats {
client_inactivity_timeout => 1200
id => "LOG-001-LON"
port => 5001
  }
}

input {
  beats {
   client_inactivity_timeout => 1200
   port => 5002
   ssl => true
   ssl_certificate => "/etc/logstash/ssl/lumberjack.cert"
   ssl_key => "/etc/logstash/ssl/lumberjack.key"
 }
}


filter {

#nginx logs
if "nginx-logs" in [tags] {
  
 }
 #platform logs .net
 else if "platform-logs" in [tags] {

  mutate {
     copy => { "@timestamp" => "beat_timestamp" }
  }

  

  date {
    match => [ "time", "yyyy-MM-dd HH:mm:ss,SSS" ]
    remove_field => [ "time" ]
  }

 }
 #iis logs
 else if "iis-logs" in [tags] {

   
 }
 # nodejs logs 
 else if "nodejs-logs" in [tags] {
  
 }

output {

if "nodejs-logs" in [tags] {

  elasticsearch {
hosts => ["ipadd.1", "ipadd.2", "ipadd.2"]
index => "nodejs-%{+YYYY.MM.dd}"
manage_template => false
user => logstash_writer
password => ''
  }
 }
else if "iis-logs" in [tags] {

   elasticsearch {
hosts => ["ipadd.1", "ipadd.2", "ipadd.2"]
 index => "filebeat-%{+YYYY.MM.dd}"
 manage_template => false
 user => logstash_writer
 password => ''
   }
  }
else if "platform-logs" in [tags] {

   elasticsearch {
hosts => ["ipadd.1", "ipadd.2", "ipadd.2"]
 index => "platform-%{+YYYY.MM.dd}"
 manage_template => false
 user => logstash_writer
 password => ''
   }

}

I've removed some unneeded bits to save room!

Everything looks good, this above config works (without lumberjack) when the upstream server was doing all the processing and sending events direct to elasticsearch but the logstash server couldn't handle it when it was busy so i was trying to just send the events to the local logostash server then onto the more powerful (downstream) LS server and then on to elasticsearch from there. The logstash servers in DC2 are being used by beats locally to that DC with no problems whatsoever. The servers are all pretty much the same config wise across both DC's

Can anyone help me figure out where I can go next? Which logging bits do i need to turn to debug on ES? Neither LS or ES are showing any errors with logging set to INFO.

Your lumberjack output plugins are sending to ports 5002 and 5003, which only partially matches what the downstream server is listening to (5001 instead of 5003). As one of the outputs can not work, this will end up blocking the entire pipeline.Preformatted text

Yeah, sorry - there is nat going on here, the ports are correct. As stated in my post I can see events in logstash on the downstream server but they're not going into ES.

Can anyone offer any advice?

The ports do not look correct to me. Can you show the current configuration if that is changed from what you posted before?

Hi -

As I said previously, there are firewalls between the two DC's and we have port address translation going on - I can see data getting to the downstream servers so i know it's not a port issue. I also posted an example debug log entry showing that the output recieved the data but it's not in the index when i look in kibana..

Do you have monitoring installed for Logstash? If so, can you check that all filters are processing data?

Did you check the elasticsearch logs? Sometimes there are errors there and nothing in the logstash logs.

Yes, I'm not seeing any errors in the logstash logs, even at debug level..

I don't know what logger to enable just for indexing errors, can you point me in the right direction?

Thanks,

Michael

You don't need to enable it. Look whereever you pointed path.logs to in elasticsearch.yml.

I've checked the standard logs (/var/log/elasticsearch/node1/ELS-001-LON-node1/es-cluster-lon.log) and it isn't showing any errors.. I guess I may need to up the log level but unsure how, as I only want to see the indexing operations logs, right?

If it is not logging an error I would take another approach first. Run

curl -X GET "http://localhost:9600/_node/stats/pipelines"

That should return a passel of JSON, which, once pretty printed, includes a pipelines object, which for each of your pipelines contains plugins, inputs, and outputs arrays. The outputs will look like this:

            "outputs" => [
                [0] {
                               "id" => "7109a9790f7c0454d7d922f9d6e3e33b9c930fce216662fb5c72be5a20ba2218",
                           "events" => {
                                       "out" => 71289841,
                        "duration_in_millis" => 39559775,
                                        "in" => 71289841
                    },
                    "bulk_requests" => {
                        "successes" => 2061854,
                        "responses" => {
                            "200" => 2061854
                        }
                    },
                        "documents" => {
                        "successes" => 71289841
                    },
                             "name" => "elasticsearch"
                }
            ],

That's a happy plugin that has indexed 70 MM documents using 2 MM bulk requests. Note the ids are random but every input, output, and filter can be named using the id option. So you could set something like

id => "es-out-iis"

Then see which output is doing what.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.