I'm trying to use the bulk API to push log files from AWS CloudWatch to Elasticsearch, and I'm running into issues because the Lambda function we're using (which is the one AWS generates for pushing logs from CW to ES) is erroring out when sending documents that have the same fields with different types.
Since these are log entries from many sources, the mapping is dynamic, and I can't guarantee the fields are all unique per index. I would like to be able to use the setting for "index.mapping.ignore_malformed" to solve this problem and at least drop the offending field instead of rejecting the whole document. The problem is I can't figure out where I'm supposed to put that in the call to the bulk API. There doesn't seem to be any documentation to what metadata exactly the "index" action takes.
Is anyone familiar with this problem, or could anyone point me in the direction of a solution?