Hi all,
We have been trying since a month to stabilize the integration of vpc flow logs using elastic agent.
There is always a lag of 2-3 days in the ingestion, the messages in the sqs queue never seems to do down because till the time a log file is processes by elastic agent another 2 files are received.
My cluster is made up of 3 hot nodes 2 of them having 10 CPU cores, 64 GB memory and high io ELB disks and the 3rd hot node having a 32 cores graviton CPU, 64 GB memory and same disk as above. We also have 2 dedicated ingest only nodes, and 2 warm nodes. Any tips on how to improve the ingestion performance and delay the log delay will be very helpful.
