Hi Team,
During performance testing, we receive many log files at a moment. We are facing logs lagging in our environment. So, that it takes too much time to reach the normal flow. So, our concern is, we have to control the log flows from filebeat or logstash in any kind of way, while the application team is doing the performance testing !
Suppose, if the application team pushes the 1k of logs while doing the performance testing. In the meantime, kibana needs to be trying to receive the whole 1k of logs. But it doesn't receive the whole logs due to not being able to handle that much load at a time.
So, In that case we need to get the logs like 10mins some count of logs and another 10 mins some count of logs and this is the way we would like to control the logs while doing performance testing.
The above given log document sizes are just for example only.
This is the log flow - Filebeat/Metricbeat - AWS Kafka - Logstash - ES - Kibana to visualise the logs.
We would like to know whether we could control the log flow. Could you please tell us.
thanks, balaji