# LS -\> LS (Lumberjack) - documents not being indexed? **URL:** \ **Category:** Logstash\ **Created:** [August 13, 2018, 6:57pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224 "2018-08-13T18:57:30Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![if-meaton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/if-meaton/32/34035_2.png) [@if-meaton](https://discuss.elastic.co/u/if-meaton)\ **Post date:** [August 13, 2018, 6:57pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/1 "2018-08-13T18:57:31Z")
Hi Guys, LS 6.3.x ES 6.3.x I have two logstashs servers across 2 datacentres connected over the internet using the lumberjack protocol. I used [https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html](https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html) as a referernce and confirmed using their generator input that events where recieved on the downstream server. I then configured everything to actually send my real events over the logstash servers and eventually into elasticsearch. According to Logstash in debug mode, the output acknowledged the event: ``` [2018-08-13T13:22:29,569][DEBUG][logstash.pipeline] output received {"event"=>{"message"=>"2018-08-13T09:03:13.540Z {name=APP-02-PB} 2018-08-13 08:41:02,742 [B008] [Com. **.Web.Services.**.**] [(null)] [321] [INFO] [Trace] - Request Xml: ", "@version"=>"1", "@timestamp"=>2018-08-13T13:22:28.673Z, "tags"=>["beats_input_codec_plain_applied"]}} ``` If I'm reading that right? I'm getting this for everything, including metricbeat, filebeat etc - anything that goes over the 2 logstash servers Upstream server: ``` input { beats { client_inactivity_timeout => 1200 id => "LOG-01-PB" port => 5001 host => "172.25.53.15" } } output { lumberjack { flush_size => 300 idle_flush_time => 3 hosts => ["ip.addr"] port => 5002 ssl_certificate => "/etc/logstash/ssl/lumberjack.cert" } lumberjack { flush_size => 300 idle_flush_time => 3 hosts => ["ip.addr"] port => 5003 ssl_certificate => "/etc/logstash/ssl/lumberjack.cert" } } ``` downstream server: ``` input { beats { client_inactivity_timeout => 1200 id => "LOG-001-LON" port => 5001 } } input { beats { client_inactivity_timeout => 1200 port => 5002 ssl => true ssl_certificate => "/etc/logstash/ssl/lumberjack.cert" ssl_key => "/etc/logstash/ssl/lumberjack.key" } } filter { #nginx logs if "nginx-logs" in [tags] { } #platform logs .net else if "platform-logs" in [tags] { mutate { copy => { "@timestamp" => "beat_timestamp" } } date { match => ["time", "yyyy-MM-dd HH:mm:ss,SSS"] remove_field => ["time"] } } #iis logs else if "iis-logs" in [tags] { } # nodejs logs else if "nodejs-logs" in [tags] { } output { if "nodejs-logs" in [tags] { elasticsearch { hosts => ["ipadd.1", "ipadd.2", "ipadd.2"] index => "nodejs-%{+YYYY.MM.dd}" manage_template => false user => logstash_writer password => '' } } else if "iis-logs" in [tags] { elasticsearch { hosts => ["ipadd.1", "ipadd.2", "ipadd.2"] index => "filebeat-%{+YYYY.MM.dd}" manage_template => false user => logstash_writer password => '' } } else if "platform-logs" in [tags] { elasticsearch { hosts => ["ipadd.1", "ipadd.2", "ipadd.2"] index => "platform-%{+YYYY.MM.dd}" manage_template => false user => logstash_writer password => '' } } ``` I've removed some unneeded bits to save room! Everything looks good, this above config works (without lumberjack) when the upstream server was doing all the processing and sending events direct to elasticsearch but the logstash server couldn't handle it when it was busy so i was trying to just send the events to the local logostash server then onto the more powerful (downstream) LS server and then on to elasticsearch from there. The logstash servers in DC2 are being used by beats locally to that DC with no problems whatsoever. The servers are all pretty much the same config wise across both DC's Can anyone help me figure out where I can go next? Which logging bits do i need to turn to debug on ES? Neither LS or ES are showing any errors with logging set to INFO. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 13, 2018, 7:39pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/2 "2018-08-13T19:39:55Z")
Your lumberjack output plugins are sending to ports `5002` and `5003`, which only partially matches what the downstream server is listening to (`5001` instead of `5003`). As one of the outputs can not work, this will end up blocking the entire pipeline.`Preformatted text` ---
**Author:** ![if-meaton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/if-meaton/32/34035_2.png) [@if-meaton](https://discuss.elastic.co/u/if-meaton)\ **Post date:** [August 13, 2018, 7:41pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/3 "2018-08-13T19:41:56Z")
Yeah, sorry - there is nat going on here, the ports are correct. As stated in my post I can see events in logstash on the downstream server but they're not going into ES. ---
**Author:** ![if-meaton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/if-meaton/32/34035_2.png) [@if-meaton](https://discuss.elastic.co/u/if-meaton)\ **Post date:** [August 14, 2018, 2:46pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/4 "2018-08-14T14:46:28Z")
Can anyone offer any advice? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 14, 2018, 2:58pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/5 "2018-08-14T14:58:13Z")
The ports do not look correct to me. Can you show the current configuration if that is changed from what you posted before? ---
**Author:** ![if-meaton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/if-meaton/32/34035_2.png) [@if-meaton](https://discuss.elastic.co/u/if-meaton)\ **Post date:** [August 14, 2018, 3:12pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/6 "2018-08-14T15:12:27Z")
Hi - As I said previously, there are firewalls between the two DC's and we have port address translation going on - I can see data getting to the downstream servers so i know it's not a port issue. I also posted an example debug log entry showing that the output recieved the data but it's not in the index when i look in kibana.. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 14, 2018, 3:44pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/7 "2018-08-14T15:44:48Z")
Do you have monitoring installed for Logstash? If so, can you check that all filters are processing data? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 14, 2018, 5:03pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/8 "2018-08-14T17:03:08Z")
Did you check the elasticsearch logs? Sometimes there are errors there and nothing in the logstash logs. ---
**Author:** ![if-meaton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/if-meaton/32/34035_2.png) [@if-meaton](https://discuss.elastic.co/u/if-meaton)\ **Post date:** [August 14, 2018, 7:02pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/9 "2018-08-14T19:02:51Z")
Yes, I'm not seeing any errors in the logstash logs, even at debug level.. ---
**Author:** ![if-meaton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/if-meaton/32/34035_2.png) [@if-meaton](https://discuss.elastic.co/u/if-meaton)\ **Post date:** [August 14, 2018, 7:03pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/10 "2018-08-14T19:03:19Z")
I don't know what logger to enable just for indexing errors, can you point me in the right direction? Thanks, Michael ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 14, 2018, 7:57pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/11 "2018-08-14T19:57:53Z")
> [@if-meaton](#): > > I don't know what logger to enable just for indexing errors, can you point me in the right direction? You don't need to enable it. Look whereever you pointed path.logs to in elasticsearch.yml. ---
**Author:** ![if-meaton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/if-meaton/32/34035_2.png) [@if-meaton](https://discuss.elastic.co/u/if-meaton)\ **Post date:** [August 14, 2018, 8:02pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/12 "2018-08-14T20:02:49Z")
I've checked the standard logs (/var/log/elasticsearch/node1/ELS-001-LON-node1/es-cluster-lon.log) and it isn't showing any errors.. I guess I may need to up the log level but unsure how, as I only want to see the indexing operations logs, right? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 14, 2018, 8:31pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/13 "2018-08-14T20:31:49Z")
If it is not logging an error I would take another approach first. Run curl -X GET "[http://localhost:9600/\_node/stats/pipelines](http://localhost:9600/_node/stats/pipelines)" That should return a passel of JSON, which, once pretty printed, includes a pipelines object, which for each of your pipelines contains plugins, inputs, and outputs arrays. The outputs will look like this: ``` "outputs" => [ [0] { "id" => "7109a9790f7c0454d7d922f9d6e3e33b9c930fce216662fb5c72be5a20ba2218", "events" => { "out" => 71289841, "duration_in_millis" => 39559775, "in" => 71289841 }, "bulk_requests" => { "successes" => 2061854, "responses" => { "200" => 2061854 } }, "documents" => { "successes" => 71289841 }, "name" => "elasticsearch" } ], ``` That's a happy plugin that has indexed 70 MM documents using 2 MM bulk requests. Note the ids are random but every input, output, and filter can be named using the id option. So you could set something like ``` id => "es-out-iis" ``` Then see which output is doing what. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 11, 2018, 8:31pm UTC](https://discuss.elastic.co/t/ls-ls-lumberjack-documents-not-being-indexed/144224/14 "2018-09-11T20:31:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.