Could not index event to Elasticsearch - object mapping for customersegment_age_classe tried to parse field customersegment_age_classe as object, but found a concrete value

Elastic Stack Logstash
1

Hello everyone and thanks for passing by !

I'm actually trying to import data into Elasticsearch (with ELK stack) from a view in a dockerized mysql with Logstash but i'm failing miserably for some weeks now as i'm a complete beginner in all these ELK things. I can't find where's my mistake and how to solve this error.
(For the sake of readability, i cut down some fields in the request)

Could not index event to Elasticsearch.
{:status=>400, :action=>["index", {:_id=>nil, :_index=>"announces_brands_profils_exclusions", :routing=>nil}, {"customersegment_age_classe"=>"[{\"ageclasse_id\": 1, \"ageclasse_label\": \”abcde\”, \"ageclasse_category\": \"brand_customer_segment.age_classe\"}, {\"ageclasse_id\": 2, \"ageclasse_label\": \”fghij\”, \"ageclasse_category\": \"brand_customer_segment.age_classe\"}
]", "announce_mechanism"=>”this is mecanism”, "profil_updated_at"=>"2020-11-21”, "announce_title"=>"will to trade",  "@timestamp"=>2021-11-17T09:21:02.187Z, "customersegment_csp"=>"[{\"csp_id\": 56, \"csp_label\": \"Non pertinent\", \"social_category\": \"brand_customer_segment.csp\”}]”,” is_premium"=>0}], :response=>{"index"=>{"_index"=>"announces_brands_profils_exclusions", "_type"=>"_doc", "_id"=>"j44zLX0BqQzOPA_sFbx-", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [customersegment_age_classe] tried to parse field [customersegment_age_classe] as object, but found a concrete value"}}}}

Here's my logstash conf

input {
  jdbc {
    jdbc_connection_string => "jdbc:mysql://mysql:3306/CONNECTION_STRING"
    jdbc_user => "USER"
    jdbc_password => "PASSWORD"
    jdbc_driver_library => "/usr/share/logstash/jarfile/mysql-connector-java-8.0.26.jar"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    schedule => "* * * * *"
    statement => "SELECT * FROM VIEW_IN_QUESTION"
    }
  }

output {
	elasticsearch {
		hosts => "elasticsearch:9200"
    index => "announces_brands_profils_exclusions"
		user => "USER"
		password => "PASSWORD"
		ecs_compatibility => disabled
    template => "path_to_template.json"
	}
}

And finally, here's the template.json

{
  "settings": {},
  "mappings": {
    "properties": {
    "announce_mechanism": {
      "type": "keyword"
    },
    "announce_title": {
      "type": "text"
    },
    "customersegment_age_classe": {
      "type": "nested",
      "dynamic": "true"
    },
    "profil_updated_at": {
      "type": "date"
    },
    "customersegment_csp": {
      "type": "nested",
      "dynamic": "true"
    }
    }
  },
  "aliases": {}
}

I'd appreciate any help and stay available if you need any more informations !
Thanks and have a great day !

2

Hi Maksonsse!

The error you're running into is because of this part of your document.

Here customersegment_age_classe is a string. The string happens to contain JSON, but elasticsearch won't automatically parse JSON into an object to go into a nested field like you have in your mapping.

You have some options. Here are some I can see, in rough order of my own preference:

  • update the mysql query to return results as a JSON array (not sure if mysql can do that natively, but it might)
  • use an elasticsearch json processor in an ingest pipeline and switch the logstash output to use that pipeline
  • use a logstash json filter to parse the json within logstash before sending to elasticsearch

Which one you should use will depend on the details of your environment. There may even be other possibilities available for you.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.