Logstash throwing error while index data into elastic search

Elastic Stack Logstash
1

Hello ,
I got following exception while index data into Elasticsearch using logstash

Could not index event to elasticsearch
error"=>{"type"=>"strict_dynamic_mapping_exception", "reason"=>"mapping set to strict, dynamic introduction of [caseid] within [globalsearch_comments] is not allowed"}}}}

i think logstash trying to match the response json with index properties, but logstash response is always lowercase, Is there any way to configure logstash which returns json keys same as index properties ??

it is working fine if i use lowercase json keys for properties ,while creating the index.

Following is my json structure to create the index.

        {
       "mappings":{
      "globalsearch_comments":{
         "dynamic":"strict",
         "properties":{
            "CommentID":{
               "type":"integer"
            } ,
            "CaseID":{
               "type":"integer"
            }         }
      }
   }
}

Logstash conf file :

        input {
  jdbc { 
    jdbc_connection_string => "jdbc:sqlserver://XXXXXXXX;databaseName=XXXXXX"
    jdbc_user => "XXX"
    jdbc_password => "XXXXX$"
    jdbc_driver_library => "D:\sqljdbc4-4.0.jar"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    statement => "SELECT *  FROM  dbo.Comments"
    }
  }
filter {
  mutate {
    remove_field => ["@version", "@timestamp"]
  }
}
output {
  stdout { codec => rubydebug}
  elasticsearch {
  "hosts" => "localhost:9200"
  "index" => "globalsearch_comments"
  "document_type" => "globalsearch_comments"
  "document_id" => "%{commentid}"
  }
}
2

It seems like you have set the mappings to be strict, which prevents new fields to be added. I suspect a new column has been added to the comments table and as this is picked up due to the SELECT * FROM comments statement, it is causing mapping failures.

3

thanks for your response, i don't want to add the dynamic properties so i set it strict, but issue was while index data into elastic, logstash returning json with lower case keys always. but while iam creating index i specified uppercase json keys so it failed.

no new columns are added to comments table.

4

Hello Christian_Dahlqvist, i found solution , i just need to mutate each Json key. like
rename => { "commentid" => "CommentID" } by default it will return json keys in lowercase only

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.