Why I am getting a double mapping for my keyword fields?

Hello,

I'm using http_poller to bring some info from an API to ES. My output config looks like this.

output {
    if [type] == 'ServerPool'{
        elasticsearch {
            hosts => "localhost:9200"
            index => "ovm-%{+YYYY.MM.dd}"
            document_type => "%{[type]}" #server,vm,etc
            template => "/etc/logstash/templates/ovm-template.json"
            template_name => "ovm"
            template_overwrite => true
        }
    }
}

And my template is somewhat like this:

{
  "template" : "ovm-*",
  "mappings" : {
    "_default" : {
      "_all" : {
        "enabled" : false
      },
      "properties" : {
        "id" : {
          "properties" : {
            "type" : {"type" : "keyword"},
            "value" : {"type" : "keyword"},
            "uri" : {"type" : "keyword"},
            "name" : {"type" : "keyword"}
          },
        "name" :  {"type" : "keyword"},
        "description" : {"type" : "text"},
        "generation" : {"type" : "long"}
        }
      }
    },

    // A Server Pool is a grouping of associated Servers and Vms that may run on those Servers. It also acts as a container for other associated objects such as Affinity Groups.
    "ServerPool" : {
      "properties" : {
        "@timestamp" : {"type" : "date"},
        "project" : {"type" : "keyword"},
        "environment" : {"type" : "keyword"},
         .
         .
         .

For some reason, after I index some documents, if I query the mappings for this index, all my "keyword" types are also being stored as a text mapping too.

curl -XGET 'localhost:9200/ovm-*/_mapping/ServerPool?pretty'
{
  "ovm-2017.03.28" : {
    "mappings" : {
      "ServerPool" : {
        "properties" : {
          "id" : {
            "properties" : {
              "name" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "type" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              .
              .
              .

I can see the duplicated fields in kibana too, for example, 'ïd.name" and "id.name.keyword". How can I just store the fields as "keywords", what else do I have to do in my template for this to work?

Thank you,

N

Your index got the default mappings, which indicates that the template has not been applied. So either the index has been created after the template has been put, or the template has not been successfully added to Elasticsearch.

Can you issue a GET call to check that your template actually exists on the Elasticsearch side, and then create a dummy index that matches the template (eg. ovm-foo) to check that the template gets applied correctly?

1 Like

Hello jpountz!

You are right the template was not being saved in ES. The logstash logs showed something funny, I had 2 instances of logstash running due to the fact that it was configured for sysV and upstart at the same time (I'm running in rhel 6). After fixing that the log showed the template was being rejected because I added a comment in it, the documentation says:

Index templates provide C-style /* */ block comments. Comments are allowed
everywhere in the JSON document except before the initial opening curly bracket.

But I'm getting this error when logstash tries to create the template

Failed to install template. {:message=>"Unexpected character ('/' (
code 47)): maybe a (non-standard) comment? (not recognized as one since Feature 'ALLOW_COMMENTS' not enabled for parser)\n at [Sourc
e: [B@3f7668d7; line: 35, column: 6]", :class=>"LogStash::Json::ParserError", :backtrace=>["/usr/share/logstash/logstash-core/lib/lo
gstash/json.rb:41:in `jruby_load'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/l
ogstash/outputs/elasticsearch/template_manager.rb:41:in `read_template_file'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/log
stash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/template_manager.rb:25:in `get_template'", "/usr/share/logs
tash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:
in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outp
uts/elasticsearch/common.rb:54:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsear
ch-6.2.6-java/lib/logstash/outputs/elasticsearch/common.rb:21:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output
_delegator_strategies/shared.rb:8:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:37:in `registe
r'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:282:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/l
ogstash/pipeline.rb:293:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/lo
gstash/pipeline.rb:293:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:302:in `start_workers'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:232:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:
387:in `start_pipeline'"]}

I looked around but found no specific option to turn the comments on, I wonder if I should create an issue with this.

Thank you!

Edit 1: Btw I'm aware that in my OP I showed the comment starting with // I also tried with the C-style block comments notation /**/ with no success, same error appeared.

N

Yes, please open an issue on Logstash. Thanks!

Moved to #logstash

Just to be sure, should I open the issue in the main github logstash repository or in the elasticsearch output plugin repository?

Thank you,

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.