LogStash throwing error 503

Elastic Stack Logstash
1

The Logastash loggers are throwing below errors continuously.

"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2018-07-30T16:58:20.262000+0530", :message=>"too many attempts at sending event. dropping:
The Elastic-search loggers doesn't have the any errors.

Logstash - 1.7.0 version
Elastic search 1.6.0 version

The health is RED status
Kindly guide to resolve

2

Why is the cluster in RED state? What does the cluster health API?

3

Cluster API health is as below

status "red"
timed_out false
number_of_nodes 5
number_of_data_nodes 1
active_primary_shards 704
active_shards 704
relocating_shards 0
initializing_shards 0
unassigned_shards 955
number_of_pending_tasks 0
number_of_in_flight_fetch 0

As per my knowledge few indices are in RED state and internally those indices few shards are in UNASSIGNED state.Further i have not much knowledge to debug.

4

How much heap do you have on your data node? It would seem you have far too many shards for a cluster that size. Please read this blog post for guidance on shards and sharding.

5

HW:16 GB RAM , 8 core windows 2008 R2 OS
Elastic search : 6GB allocated
LogStash: 2GB allocated
Harddisk:1TB total, 550 GB free

6

You have far too many shards for that size of heap and need to reduce that significantly or scale the cluster up/out in order to get a stable cluster. As you are on such an old version this will require you to delete and/or reindex data if you can not increase resources.

7

Thanks . Can you suggest on how much resource need to meet the 120% of current data.I will do it..

8

Even if you add resources you probably need to change your sharding policy and reduce the shard count substantially. having so many small shards is very inefficient. I do not have any recommendation about how much resources to add as I have not used this version in years, But adding another data node of the same size while setting the number of replicas to 0 might give you enough headroom to restructure your shards.

9

Thank you sir,
These actions are going to perform and let you know.
Reduce the shards/index from 3 to 1
Delete the the index of 3 months older
Replica size to zero

Still Instead of adding one more node, I hope some increase resource might help.Suggest on this irrespective of version.Its not a problem and a kind of learning for us.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.