I'm using ELK 6.5.4
Filebeat = 10 nodes of Nginx
Logstash Server = 1 nodes(10.5.202.201) = (8 core, 128GB Ram) || 3 nodes = 1 node of (8 core, 128GB Ram) & 2 nodes of (2 core, 4GB RAM)
I'm getting logs from 2 nodes (nginx001 & nginx002) of nginx. While checking logstash logs in debug mode I'm getting logs from another servers too but those logs are not shown on Kibana.
Sample logstash log where I'm getting logs from node nginx008 too. (Likewise I'm getting logs from nginx001 to nginx010). But in Kibana only 2 nodes logs (nginx001 & nginx002) are showing.
[2019-01-31T13:43:57,555][DEBUG][logstash.pipeline ] filter received {"event"=>{"input"=>{"type"=>"log"}, "message"=>"10.5.10.10 - - [31/Jan/2019:10:30:36 +0530] \"GET /my/v2/home/ham2?density=XHDPI HTTP/1.1\" 200 6689 \"-\" \"Dalvik/2.1.0 (Linux; U; Android 6.0.1; HTC Desire 10 lifestyle Build/MMB29M)\"ResponseTime=0.006 BackendConnectTime=\"0.000\" BackendResponseTime=\"0.006\" backend_host=\"10.5.2.2:8080\"", "source"=>"/data/nginx/access.log", "tags"=>["beats_input_codec_plain_applied"], "@version"=>"1", "offset"=>889329952, "host"=>{"containerized"=>true, "name"=>" **nginx008** ", "os"=>{"platform"=>"rhel", "version"=>"7.5 (Maipo)", "codename"=>"Maipo", "family"=>""}, "architecture"=>"x86_64", "id"=>"7ea08555621c4ac68fa586f12e9a9a46"}, "log_type"=>"maa-nginx-log", "beat"=>{"name"=>" **nginx008** ", "version"=>"6.5.4", "hostname"=>" **nginx008** "}, "@timestamp"=>2019-01-31T08:13:53.802Z, "prospector"=>{"type"=>"log"}}}
If I stop all filebeat & test one filebeat at a time then there is no issue all servers logs will appear. But when I start all filebeats It shows logs from 2 nodes only or sometime 4 nodes but never all nodes logs at a time.
Filebeat Config -
> filebeat.inputs:
> - type: log
> enabled: true
> paths:
> - /data/nginx/access.log
> fields:
> log_type: maa-nginx-log
> fields_under_root: true
>
> ignore_older: 2m
>
> filebeat.config.modules:
> # Glob pattern for configuration loading
> path: ${path.config}/modules.d/*.yml
>
> # Set to true to enable config reloading
> reload.enabled: false
> setup.template.settings:
> index.number_of_shards: 3
>
> output.logstash:
> hosts: ["10.5.202.201:5044"]
> # hosts: ["10.5.202.201:5044", "10.5.202.202:5044", "10.5.202.203:5044"]
> loadbalance: true
> worker: 4
Please suggest what am I missing & can do to fix this weird problem.