We are shipping all of our F5 logs to Redis and then into Elasticsearch but they seem to be getting backed up. We are seeing a delay of sometimes 4 hours or more. They continue to process and it seems to catchup over night when traffic is slow. The issues seems to be with the Logstash shipper as Redis is not showing records waiting to be processed. Any suggestions on how to speed up the processing of the F5 logs to keep them near real time in Kibana??
Does either your logstash or Elasticsearch seem to be consuming CPU?
Also, when looking at the log file lines doe the @timestamp match the Log file timedtamp? You could just be seeing a GMT rather then current timezone.
What is your F5 Logging rate?
The @timestamp does match the log file. I am not seeing an increase in CPU for either of them.
When I look at Kibana I am not seeing any F5 logs for the last hour. We will slowly trickle in and as the day progresses we will see that hour gap continue to grow.
Kibana is showing around 90,000 for a five minute period. Around 250 per second.
Ok, that does not seem like a lot
Can you post your logstash config? Is logstash, redis and Elastic search all on the same system? If so what do the system performance look like?
Also, how many nodes are you running
Oh and you said that Redis is not showing any queued messages?
The shipper is using nearly 100% of the CPU and the iowait is at 13. We are working on adding more RAM and CPU's but would running multiple shippers also help? I am processing F5 logs from 2 locations.
100% is not a lot unless your currently only using 1 core (unless you only have 1 or 2). The IOWait should not be on the shipper unless this is Memory swapping in which case that is real bad and probably your bottle neck. How big is your JVM on the shipper.
How many Nodes on ES do you have?
Have you check your elasticsearch states to see the indexing/merge rates. You can do this with Paramedic/Marvel/Kofpr
There are only 2 cpu's on the vm. The jvm is only 500m.
We have four ES nodes.
I haven't checked the the elasticsearch states. Is Paramedic/Marvel/Kofpr part of Shield?
Is Paramedic/Marvel/Kofpr part of Shield?
Paramedic and Kopf are freely available plugins while Marvel is included in all Elastic subscriptions (some of which also include Shield).