Using Doc Values

msimos · September 10, 2015, 11:51pm

Hi,

In your logstash install there is a file called:

vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.5-java/lib/logstash/outputs/elasticsearch/elasticsearch-template.json

This is the template which logstash applies if it doesn't find it in Elasticsearch. For any field which is "type": "string" and "index": "not_analyzed" then you can add "doc_values": true. You can also apply doc values to any other field type like double, long, etc. so long as the field is not a string and analyzed. For example:

               "@timestamp": {
                  "type": "date",
                  "doc_values": true,
                  "format": "dateOptionalTime"
               }

Its best to define your fields in your mapping and set doc values for each field. If you're only using dynamic mapping to create your fields then you need to add this into the template. I created one which will automatically apply doc values to any dynamically created field:

gist.github.com

https://gist.github.com/msimos/1e080f47870a9b309444

gistfile1.txt

{
    "template" : "logstash-*",
    "settings" : {
      "index.refresh_interval" : "5s",
      "index.query.default_field": "message"
    },
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [ {
          "string_fields" : {

This file has been truncated. show original

Its not something I have really tested so you should not put it into production without testing this. Use this as a way to autogenerate your fields when importing some logs so you can build your own template. To add it to Elasticsearch do:

curl -XDELETE http://localhost:9200/_template/logstash
curl -XPUT http://localhost:9200/_template/logstash -d @template.json

If you encounter any problems you can revert back to the original template by using the above commands and using the template file that comes with logstash.