Two load balanced servers send same logs to logstash. But data not update real time in kibana dashboard

Elastic Stack Beats
1

Hi all,

DESCRIPTION

I have two logs 2 send to logstash. But these two logs available in two servers due load balancing. So I send them all to centralized logstash server as follows

image

following is my filebeat config of two servers

filebeat.prospectors:

  • input_type: log

    paths:

    • /app/TYPE_1-SERVICE/logs/log_type1*.log
    • /app/TYPE_2-SERVICE/logs/log_type2*.log

    ignore_older: 10h

output.logstash:

hosts: ["172.26.29.126:5044"]

I start filebeat in both servers as follows

sudo /etc/init.d/filebeat start

Out put is

2017/11/03 03:49:20.024553 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.3
2017/11/03 03:49:20.024660 metrics.go:23: INFO Metrics logging every 30s
2017/11/03 03:49:20.024730 logstash.go:90: INFO Max Retries set to: 3
2017/11/03 03:49:20.024908 outputs.go:108: INFO Activated logstash as output plugin.
2017/11/03 03:49:20.025080 publish.go:300: INFO Publisher name: #######
2017/11/03 03:49:20.025387 async.go:63: INFO Flush Interval set to: 1s
2017/11/03 03:49:20.025413 async.go:64: INFO Max Bulk Size set to: 2048

ISSUE

But my kibana dashboard doesn't update real time. It updates with a lag for both log types. Specially one log get delayed significantly (log_type1*.log)

following are real observations

for log_type1*.log

current time: 9.40 A.M.
Updated time in dash board: 8.03 A.M.

image
image.png1085×323 15.5 KB

for log_type2*.log

current time: 9.40 A.M.
Updated time in dash board: 9.35 A.M.

image
image.png1070×345 16.9 KB

Can some one please help me!!!!!

Thanks a lot!

2

Please help me. This is bit urgent. :slightly_frowning_face:

3

What does it look like if you change the time-picker to show the last 4 hours with a 30 second refresh interval instead of today? What is the load on the host where Elasticsearch, Kibana and Elasticsearch is installed (CPU and disk I/O)? What does your Logstash config look like?

4

Actually i thought issue sorted! But still one specific log is getting lag! Can any one please tell me a possible reason for this? Please refer the above attached diagram and other information to get an idea,