Indices Creation Failed in ES via Logstash from Filebeat

Elastic Stack Logstash
1

Greetings, and Happy New Year!

We have 2 filbeats (from different servers, IP_A and IP_B) are connecting to Logstash in IP_C. Then we have the ES and Kibana set up in IP_C too. The problem I'm facing is that I'm only able to query the logs from filebeat in IP_A and nothing from the filebeat in IP_B.

I've ran on debug mode on the filebeat in IP_B and logstash, I could see the message is pushed across to logstash. However, when I query in ES, there's nothing.

Both filebeats have identical configuration. I'm running on filebeat (version 5.6.3), logstash (version 5.6.2) and ES (version 5.6.2).

I've been scratching my head for a few days now and couldn't figure out the root cause for this. Any help/advice is much appreciated.
Thank you.

2

May be logstash can not parse the logs from the other server?

3

Hi Dave,

The Logstash did received the events from filebeat in IP_B. The communication there is confirmed successful as I could see the messages in Logstash when running it on debug mode.

I just wonder if there's any known issue behind this kind of design, as I've seen some people raising the exact same issue (unfortunately with no reply).

4

I don't know. I moved your question to #logstash in case someone else has an idea.

I think it could help to share some of your logstash debug logs.

5

What does you Logstash config look like? Do the data from both Filebeat instances go into the same pipeline?

6

Thanks Dave.

Hi Christian,

The Logstash config is as below. Yes, both Filebeat instances are going into the same pipeline.

input {
    beats {
        port => 5400
    }
}

filter {
 grok {
   match => { 'message' => '%{IPORHOST:clientip}  - - \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{URIPATHPARAM:request}(?: HTTP/%{NUMBER:httpversion})?|)\" %{NUMBER:answer} (?:%{NUMBER:byte}|-) (?:\"(?:%{URI:referrer}|-))\" %{QS:agent} "%{IPORHOST:proxyip}" "%{IPORHOST:hostname}" "%{GREEDYDATA:ident}"'}
 }
 mutate {
   convert => ["bytes", "integer"]
 }
 geoip {
   source => "clientip"
   target => "geoip"
   add_tag => [ "nginx-geoip" ]
 }
 date {
   match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
   remove_field => [ "timestamp" ]
 }
 useragent {
   source => "agent"
 }
}

output {
 elasticsearch {
   hosts => ["localhost:9200"]
   index => "weblog-%{+YYYY.MM.dd}"
   document_type => "nginx_logs"
 }
 stdout { codec => rubydebug }
}

Below are snippets of the DEBUG logs in the Logstash via zgrep:

/var/log/logstash.own/logstash-plain-2017-12-28.log.gz:[2017-12-28T06:31:58,258][DEBUG][logstash.pipeline ] output received {"event"=>{"request"=>"/app/profile/edit.do?task=doShareChartData", "agent"=>""Mozilla/5.0 (Linux; Android 6.0.1; OPPO R9s Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/46.0.2490.76 Mobile Safari/537.36"", "proxyip"=>"108.xxx.xxx.227", "minor"=>"0", "ident"=>"xxxxx777", "os_minor"=>"0", "os_major"=>"6", "source"=>"/home/nginxlogs/nginx-access.log", "type"=>"log", "patch"=>"2490", "hostname"=>"www.xxxx.com", "major"=>"46", "clientip"=>"116.xxx.xxx.86", "@version"=>"1", "beat"=>{"name"=>"fb2.xxx.xxx.com", "hostname"=>"fb2.xxx.xxx.com", "version"=>"5.6.3"}, "host"=>"fb2.xxx.xxx.com", "geoip"=>{"city_name"=>"xxx", "timezone"=>"xxx", "ip"=>"xxx", "latitude"=>xxx, "country_name"=>"xxx", "country_code2"=>"xx", "continent_code"=>"xx", "country_code3"=>"xx", "region_name"=>"xxxx", "location"=>{"lon"=>xxx, "lat"=>xxx}, ......}}

(Apologies, i had to "mask" certain values).
Thanks for helping out. However, I can't see how the issue is on the Logstash, as I'm able to see the logs which are coming in from Filebeat in IP_B. I have different name and hostname on the beat.name and beat.hostname elements.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.