Elasticsearch mapping for adding custom metadata payload to log entries in ELK stack

Elastic Stack Elasticsearch
1

We're using ELK stack for log management of a distributed application. In
our log entries we have a field called "data" that can be of arbitrary
type. We use this field to allow the developer to log any custom metadata
releated to the incident that occurred. For example the following three
entries could be samples of items we'd like to be able to log and store to
Elasticsearch.

{"message" : "Fire Alarm", "level" : "emergency", "data" : { "Location" : 1,
"Temperature" : 76.3 } }
{"message" : "Remote query exceeded warning timeout", "level" : "warning",
"data" : 500}
{"message" : "UPS not available", "level" : "warning", "data" : { "Location"
: "San Francisco", "Rack" : 1 } }

All other fields map nicely to Elasticsearch built-in field types but we
don't know how to treat the "data" field as the value can be of any
Elasticsearch type and different for each entry. When using the built-in
Elasticsearch mapping shipping with Logstash, only the mapping of the first
entry will be used. In the above example only the first message would be
stored to the Elasticsearch index.

It's okay if the data field is not indexed. But we'd still like to be able
to store and retrieve it from elasticsearch. What kind of mapping document
would work? I was trying different options but didn't seem to find one that
would do what we needed. The logstash 1.5 default Elasticsearch mapping is
below.

{
"template" : "logstash-",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true
}
}
}, {
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed",
"ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}

Thanks,

Tomi

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hi Tomi,

You could do it like this:

DELETE test
PUT test
{
"mappings": {
"doc": {
"properties": {
"data": {
"type": "object",
"enabled": false
},
"level": {
"type": "string"
},
"message": {
"type": "string"
}
}
}
}
}
PUT test/doc/1
{
"message" : "Fire Alarm",
"level" : "emergency",
"data" : {
"Location" : 1,
"Temperature" : 76.3
}
}
PUT test/doc/2
{
"message" : "Remote query exceeded warning timeout",
"level" : "warning",
"data" : 500
}
PUT test/doc/3
{
"message" : "UPS not available",
"level" : "warning",
"data" : {
"Location" : "San Francisco",
"Rack" : 1
}
}

HTH

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 8 févr. 2015 à 02:52, Tomi SF tomi.maila@gmail.com a écrit :

We're using ELK stack for log management of a distributed application. In our log entries we have a field called "data" that can be of arbitrary type. We use this field to allow the developer to log any custom metadata releated to the incident that occurred. For example the following three entries could be samples of items we'd like to be able to log and store to Elasticsearch.

{"message" : "Fire Alarm", "level" : "emergency", "data" : { "Location" : 1, "Temperature" : 76.3 } }
{"message" : "Remote query exceeded warning timeout", "level" : "warning", "data" : 500}
{"message" : "UPS not available", "level" : "warning", "data" : { "Location" : "San Francisco", "Rack" : 1 } }

All other fields map nicely to Elasticsearch built-in field types but we don't know how to treat the "data" field as the value can be of any Elasticsearch type and different for each entry. When using the built-in Elasticsearch mapping shipping with Logstash, only the mapping of the first entry will be used. In the above example only the first message would be stored to the Elasticsearch index.

It's okay if the data field is not indexed. But we'd still like to be able to store and retrieve it from elasticsearch. What kind of mapping document would work? I was trying different options but didn't seem to find one that would do what we needed. The logstash 1.5 default Elasticsearch mapping is below.

{
"template" : "logstash-",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true
}
}
}, {
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}

Thanks,

Tomi

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/057E8068-E1A5-47CE-83C0-9B91B61AFA98%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.