Mapper_parsing_exception on data from AWS Lambda

Elastic Stack Elasticsearch
1

I'm trying to send http logs from AWS Cloudwatch to ES (Kibana), and a large chunk of logs are not being sent with the following error message:

mapper_parsing_exception, "reason":"object mapping for [responseData.client] tried to parse field [client] as object, but found a concrete value

I've tried to force my logs to stringify responseData.client so that it's always a string, however the error is persisting.

The index that AWS is sending to is incremented every day (it's called cwl-<today's date>) so my understanding was that it would create a new index after the change I implemented and keep the responseData.client field as a string, but it appears not to be working.

Is there a way to force ES to keep this field as a string?

Thank you!

2

Yes, create an index template with the desired mapping.

3

Thanks, there are two things I am concerned about before attempting this change:

  • there are 2000+ different fields in that index: do I need to include them all in the index template? Can I send a mapping for the problematic field only?
  • a new index is being created every day - my understanding is that its mappings are a function of whatever data it is sent from AWS (apologies for my lack of knowledge but I inherited this process and am not sure how it was initially set up). Could my mapping that I submit for, for example “cwl-01-10-2019”, get overwritten the next day when the index “cwl-02-10-2019” is initialised?

Thanks again.