# Elasticsearch mapping for adding custom metadata payload to log entries in ELK stack **URL:** \ **Category:** Elasticsearch\ **Created:** [February 8, 2015, 1:52am UTC](https://discuss.elastic.co/t/elasticsearch-mapping-for-adding-custom-metadata-payload-to-log-entries-in-elk-stack/22031 "2015-02-08T01:52:09Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Tomi\_SF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomi_sf/32/928_2.png) [@Tomi\_SF](https://discuss.elastic.co/u/Tomi_SF)\ **Post date:** [February 8, 2015, 1:52am UTC](https://discuss.elastic.co/t/elasticsearch-mapping-for-adding-custom-metadata-payload-to-log-entries-in-elk-stack/22031/1 "2015-02-08T01:52:09Z")
We're using ELK stack for log management of a distributed application. In our log entries we have a field called "data" that can be of arbitrary type. We use this field to allow the developer to log any custom metadata releated to the incident that occurred. For example the following three entries could be samples of items we'd like to be able to log and store to Elasticsearch. {"message" : "Fire Alarm", "level" : "emergency", "data" : { "Location" : 1, "Temperature" : 76.3 } } {"message" : "Remote query exceeded warning timeout", "level" : "warning", "data" : 500} {"message" : "UPS not available", "level" : "warning", "data" : { "Location" : "San Francisco", "Rack" : 1 } } All other fields map nicely to Elasticsearch built-in field types but we don't know how to treat the "data" field as the value can be of any Elasticsearch type and different for each entry. When using the built-in Elasticsearch mapping shipping with Logstash, only the mapping of the first entry will be used. In the above example only the first message would be stored to the Elasticsearch index. It's okay if the data field is not indexed. But we'd still like to be able to store and retrieve it from elasticsearch. What kind of mapping document would work? I was trying different options but didn't seem to find one that would do what we needed. The logstash 1.5 default Elasticsearch mapping is below. { "template" : "logstash-_", "settings" : { "index.refresh\_interval" : "5s" }, "mappings" : { "default" : { "\_all" : {"enabled" : true}, "dynamic\_templates" : [ { "message\_field" : { "match" : "message", "match\_mapping\_type" : "string", "mapping" : { "type" : "string", "index" : "analyzed", "omit\_norms" : true } } }, { "string\_fields" : { "match" : "_", "match\_mapping\_type" : "string", "mapping" : { "type" : "string", "index" : "analyzed", "omit\_norms" : true, "fields" : { "raw" : {"type": "string", "index" : "not\_analyzed", "ignore\_above" : 256} } } } } ], "properties" : { "@version": { "type": "string", "index": "not\_analyzed" }, "geoip" : { "type" : "object", "dynamic": true, "path": "full", "properties" : { "location" : { "type" : "geo\_point" } } } } } } } Thanks, Tomi -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 9, 2015, 10:56am UTC](https://discuss.elastic.co/t/elasticsearch-mapping-for-adding-custom-metadata-payload-to-log-entries-in-elk-stack/22031/2 "2015-02-09T10:56:34Z")
Hi Tomi, You could do it like this: DELETE test PUT test { "mappings": { "doc": { "properties": { "data": { "type": "object", "enabled": false }, "level": { "type": "string" }, "message": { "type": "string" } } } } } PUT test/doc/1 { "message" : "Fire Alarm", "level" : "emergency", "data" : { "Location" : 1, "Temperature" : 76.3 } } PUT test/doc/2 { "message" : "Remote query exceeded warning timeout", "level" : "warning", "data" : 500 } PUT test/doc/3 { "message" : "UPS not available", "level" : "warning", "data" : { "Location" : "San Francisco", "Rack" : 1 } } HTH -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > Le 8 févr. 2015 à 02:52, Tomi SF [tomi.maila@gmail.com](mailto:tomi.maila@gmail.com) a écrit : > > We're using ELK stack for log management of a distributed application. In our log entries we have a field called "data" that can be of arbitrary type. We use this field to allow the developer to log any custom metadata releated to the incident that occurred. For example the following three entries could be samples of items we'd like to be able to log and store to Elasticsearch. > > {"message" : "Fire Alarm", "level" : "emergency", "data" : { "Location" : 1, "Temperature" : 76.3 } } > {"message" : "Remote query exceeded warning timeout", "level" : "warning", "data" : 500} > {"message" : "UPS not available", "level" : "warning", "data" : { "Location" : "San Francisco", "Rack" : 1 } } > > All other fields map nicely to Elasticsearch built-in field types but we don't know how to treat the "data" field as the value can be of any Elasticsearch type and different for each entry. When using the built-in Elasticsearch mapping shipping with Logstash, only the mapping of the first entry will be used. In the above example only the first message would be stored to the Elasticsearch index. > > It's okay if the data field is not indexed. But we'd still like to be able to store and retrieve it from elasticsearch. What kind of mapping document would work? I was trying different options but didn't seem to find one that would do what we needed. The logstash 1.5 default Elasticsearch mapping is below. > > { > "template" : "logstash-_", > "settings" : { > "index.refresh\_interval" : "5s" > }, > "mappings" : { > "default" : { > "\_all" : {"enabled" : true}, > "dynamic\_templates" : [ { > "message\_field" : { > "match" : "message", > "match\_mapping\_type" : "string", > "mapping" : { > "type" : "string", "index" : "analyzed", "omit\_norms" : true > } > } > }, { > "string\_fields" : { > "match" : "_", > "match\_mapping\_type" : "string", > "mapping" : { > "type" : "string", "index" : "analyzed", "omit\_norms" : true, > "fields" : { > "raw" : {"type": "string", "index" : "not\_analyzed", "ignore\_above" : 256} > } > } > } > } ], > "properties" : { > "@version": { "type": "string", "index": "not\_analyzed" }, > "geoip" : { > "type" : "object", > "dynamic": true, > "path": "full", > "properties" : { > "location" : { "type" : "geo\_point" } > } > } > } > } > } > } > > Thanks, > > Tomi > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/057E8068-E1A5-47CE-83C0-9B91B61AFA98%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/057E8068-E1A5-47CE-83C0-9B91B61AFA98%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:34am UTC](https://discuss.elastic.co/t/elasticsearch-mapping-for-adding-custom-metadata-payload-to-log-entries-in-elk-stack/22031/3 "2017-07-06T00:34:11Z")