I read log into elasticsearch using the _bulk endpoint and manage around 10Klines/sec. I have very modest regexp filtering just @timestamp, operationid and log, if I add more elaborative filtering the insert rate goes down. (the server can deliver 80K lines/sec)
Is there a way to refilter this index later on. I only have a count as index. creating uuids or a unique of log takes too long.
A perfect 1 stage parsing leaves me with 400 lines/sec