Indices Creation Failed in ES via Logstash from Filebeat

Greetings, and Happy New Year!

We have 2 filbeats (from different servers, IP_A and IP_B) are connecting to Logstash in IP_C. Then we have the ES and Kibana set up in IP_C too. The problem I'm facing is that I'm only able to query the logs from filebeat in IP_A and nothing from the filebeat in IP_B.

I've ran on debug mode on the filebeat in IP_B and logstash, I could see the message is pushed across to logstash. However, when I query in ES, there's nothing.

Both filebeats have identical configuration. I'm running on filebeat (version 5.6.3), logstash (version 5.6.2) and ES (version 5.6.2).

I've been scratching my head for a few days now and couldn't figure out the root cause for this. Any help/advice is much appreciated.
Thank you.

May be logstash can not parse the logs from the other server?

Hi Dave,

The Logstash did received the events from filebeat in IP_B. The communication there is confirmed successful as I could see the messages in Logstash when running it on debug mode.

I just wonder if there's any known issue behind this kind of design, as I've seen some people raising the exact same issue (unfortunately with no reply).

I don't know. I moved your question to #logstash in case someone else has an idea.

I think it could help to share some of your logstash debug logs.

What does you Logstash config look like? Do the data from both Filebeat instances go into the same pipeline?

Thanks Dave.

Hi Christian,

The Logstash config is as below. Yes, both Filebeat instances are going into the same pipeline.

input {
    beats {
        port => 5400
    }
}

filter {
 grok {
   match => { 'message' => '%{IPORHOST:clientip}  - - \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{URIPATHPARAM:request}(?: HTTP/%{NUMBER:httpversion})?|)\" %{NUMBER:answer} (?:%{NUMBER:byte}|-) (?:\"(?:%{URI:referrer}|-))\" %{QS:agent} "%{IPORHOST:proxyip}" "%{IPORHOST:hostname}" "%{GREEDYDATA:ident}"'}
 }
 mutate {
   convert => ["bytes", "integer"]
 }
 geoip {
   source => "clientip"
   target => "geoip"
   add_tag => [ "nginx-geoip" ]
 }
 date {
   match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
   remove_field => [ "timestamp" ]
 }
 useragent {
   source => "agent"
 }
}

output {
 elasticsearch {
   hosts => ["localhost:9200"]
   index => "weblog-%{+YYYY.MM.dd}"
   document_type => "nginx_logs"
 }
 stdout { codec => rubydebug }
}

Below are snippets of the DEBUG logs in the Logstash via zgrep:

/var/log/logstash.own/logstash-plain-2017-12-28.log.gz:[2017-12-28T06:31:58,258][DEBUG][logstash.pipeline ] output received {"event"=>{"request"=>"/app/profile/edit.do?task=doShareChartData", "agent"=>""Mozilla/5.0 (Linux; Android 6.0.1; OPPO R9s Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/46.0.2490.76 Mobile Safari/537.36"", "proxyip"=>"108.xxx.xxx.227", "minor"=>"0", "ident"=>"xxxxx777", "os_minor"=>"0", "os_major"=>"6", "source"=>"/home/nginxlogs/nginx-access.log", "type"=>"log", "patch"=>"2490", "hostname"=>"www.xxxx.com", "major"=>"46", "clientip"=>"116.xxx.xxx.86", "@version"=>"1", "beat"=>{"name"=>"fb2.xxx.xxx.com", "hostname"=>"fb2.xxx.xxx.com", "version"=>"5.6.3"}, "host"=>"fb2.xxx.xxx.com", "geoip"=>{"city_name"=>"xxx", "timezone"=>"xxx", "ip"=>"xxx", "latitude"=>xxx, "country_name"=>"xxx", "country_code2"=>"xx", "continent_code"=>"xx", "country_code3"=>"xx", "region_name"=>"xxxx", "location"=>{"lon"=>xxx, "lat"=>xxx}, ......}}

(Apologies, i had to "mask" certain values).
Thanks for helping out. However, I can't see how the issue is on the Logstash, as I'm able to see the logs which are coming in from Filebeat in IP_B. I have different name and hostname on the beat.name and beat.hostname elements.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.