I've seen an issue in recent days with sending data to elasticsearch 2.1 where inbound messages contained an "_uid" field which clashed with the type of the in-built meta-field "_uid". This led to indexing failing, shards becoming unassigned and the cluster going into a red state.
Given elasticsearch's apparent heightened sensitivity around type handling I feel I need to strip any leading underscores from inbound field names as a safety measure. However, as our log formats vary widely we do not know in andvance what the field names will be. Also, we do virtually no filtering in logstash as we pre-format our log lines as json. So our exposure to filtering is very limited...
So, can someone show me how I can parse inbound messages to strip leading underscores from field names? Also, am I taking the right approach here? Could I mitigate this issue by modifying the logstash template?