There are several problems in audit pipeline that made this module unusable for our filebeat instance.
-
Our audit log line looks like this (we're using ubuntu 16.04 with it's reposiory auditd)
node=test-01 type=EOE msg=audit(1556015556.878:10185941):
But as seen in auditd ingest pipeline
There is no support for "node=*" prefix, which resulted in grok parsing errors. -
Ingest pipeline is using painless script, which seems to be recompiled with every log line. That's why when we manually fixed previous problem, we encountered another one, for all our pipelines (this one is for iis access logs, same for audit):
elasticsearch | [2019-04-18T13:41:54,180][DEBUG][o.e.a.b.TransportBulkAction] [10dc8199d6dd] failed to execute pipeline [iis-7.5-access-pipeline] for document [iis_access_log-2019.04.18/doc/null] elasticsearch | org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalStateException: pipeline with id [iis-7.5-access-pipeline] could not be loaded, caused by [ElasticsearchParseException[Error updating pipeline with id [iis-7.5-access-pipeline]]; nested: GeneralScriptException[Failed to compile inline script [iis.access.time] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; GeneralScriptException[Failed to compile inline script [iis.access.time] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; org.elasticsearch.common.breaker.CircuitBreakingException: [script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting] elasticsearch | at org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:188) ~[elasticsearch-6.5.4.jar:6.5.4] elasticsearch | at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:134) ~[elasticsearch-6.5.4.jar:6.5.4] elasticsearch | at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:97) ~[elasticsearch-6.5.4.jar:6.5.4] elasticsearch | at org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:479) ~[elasticsearch-6.5.4.jar:6.5.4] elasticsearch | at org.elasticsearch.ingest.IngestService.access$100(IngestService.java:68) ~[elasticsearch-6.5.4.jar:6.5.4] elasticsearch | at org.elasticsearch.ingest.IngestService$4.doRun(IngestService.java:408) [elasticsearch-6.5.4.jar:6.5.4] elasticsearch | at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.4.jar:6.5.4] elasticsearch | at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4] elasticsearch | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] elasticsearch | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] elasticsearch | at java.lang.Thread.run(Thread.java:834) [?:?] elasticsearch | Caused by: java.lang.IllegalArgumentException: java.lang.IllegalStateException: pipeline with id [iis-7.5-access-pipeline] could not be loaded, caused by [ElasticsearchParseException[Error updating pipeline with id [iis-7.5-access-pipeline]]; nested: GeneralScriptException[Failed to compile inline script [iis.access.time] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; GeneralScriptException[Failed to compile inline script [iis.access.time] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; org.elasticsearch.common.breaker.CircuitBreakingException: [script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting]