Sorry should have explained. I am using the built in ASA module for elasticsearch. I am not using logstash currently. Originally this was the case with my own custom filter, however, this prevented me from using the ILM feature which is desperately needed due to the volume of documents that are being processed.
It is issues like this where open source really falls short. I am trying to present this as a viable alternative to SPLUNK but so far it is failing.