Sorry about my English. I will try to describe what's going on. And also still quite newbie on the ELK process..
Here is the exception i found in logstash's log:
[2018-01-23T09:07:20,699][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"process_cluster_event_timeout_exception", "reason"=>"failed to process cluster event (create-index [logstash-2018.01.23], cause [auto(bulk api)]) within 1m"})
Here is my Elasticsearch health check:
{"cluster_name":"elasticsearch","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":26,"active_shards":26,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":26,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":50.0}
After i logged in the Elasticsearch server, within the console which i started logstash, and another console i started the elasticsearch.. when i was checking the error msg in logstash's console. somehow it starts working plus.. it starts processing log files which it has processed already. Somehow filebeat is resending the old logs.. Our log has rotation. when it reaches 10M, it will bubble up to log.1.
Everything is still in testing stage. I started the elasticsearch logstash filebeat on Last Fri. It works fine till today's 12:00:00am...
THank you for your help!!