So at some point it would appear my pipeline names got changed and so i'm recieving this error a lot with debugging on:
[2017-12-12T01:21:44,884][DEBUG][o.e.a.b.TransportBulkAction] [i-0b38fb9c609ca9464] failed to execute pipeline [go-log-pipeline] for document [go-2017.12.12/doc/null]
java.lang.IllegalArgumentException: pipeline with id [go-log-pipeline] does not exist
at org.elasticsearch.ingest.PipelineExecutionService.getPipeline(PipelineExecutionService.java:189) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.ingest.PipelineExecutionService.access$100(PipelineExecutionService.java:41) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:88) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0.jar:6.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Trying to track this down I moved the logging up to TRACE but that yielded little results. Is there any way to get this log to include a src address? It would be very helpful for tracking down that run-away filebeat sender in a cluster of 1k nodes.