# Could not index event to Elasticsearch

**URL:** https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/124672
**Category:** Logstash
**Created:** [March 20, 2018, 3:08am UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/124672 "2018-03-20T03:08:00Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)
#### Post date: [March 20, 2018, 3:08am UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/124672/1 "2018-03-20T03:08:01Z")

</div>

This is my json item,

 ![47](https://us1.discourse-cdn.com/elastic/original/3X/1/3/1315202eac1c9d3684ecad9234f9100bd815ff0f.jpg)

And this is my config

```
input {
	file {
		path => "/Users/apple/Desktop/RiskComb/RiskComb.json"
		start_position => "beginning"
		sincedb_path => "/Users/apple/Desktop/RiskComb/read_pos.txt"
		#sincedb_path => "/dev/null"
		codec => json
		

	}
}

filter {

	mutate{
		rename => ["_id" , "ID"]
		rename => ["_nestedCalls" , "nestedCalls"]
	}
	date{
		match => ["ts", "yyyy-MM-dd HH:mm:ss.SSS ZZ"]
	}

}

output {
	elasticsearch {
		hosts => "localhost:9200"
		index => "riskcombbdata"
		document_type => "eventdata"
	}
	stdout { codec => rubydebug } 
	
}

```

The error info is  
`[2018-03-19T23:03:15,552][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"riskcombbdata", :_type=>"eventdata", :_routing=>nil}, #<LogStash::Event:0x4df9660b>], :response=>{"index"=>{"_index"=>"riskcombbdata", "_type"=>"eventdata", "_id"=>"a2pcQWIBTNBnrTnHXqWR", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [nestedCalls.result.internalKeyValuePairs.be.score] of different type, current_type [text], merged_type [ObjectMapper]"}}}}`

I am not sure how to deal with this problem.

---

<div class="post-metadata">

### Author: ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [March 20, 2018, 6:03am UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/124672/2 "2018-03-20T06:03:54Z")

</div>

In Elasticsearch, a given field must be consistent in type across an entire index; Elasticsearch is rejecting your request because the field `nestedCalls.result.internalKeyValuePairs.be.score` is different than the current mapping, which was likely auto-detected from a previous document.

---

<div class="post-metadata">

### Author: ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)
#### Post date: [March 20, 2018, 6:39am UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/124672/3 "2018-03-20T06:39:12Z")

</div>

This is my current structure.

 ![15](https://us1.discourse-cdn.com/elastic/original/3X/d/f/df0a9b842adba4e034ccf3056dc49b3cad23cacd.jpg)

And I removed the field("\_nestedCalls") in the filter of logstash.

So it seems like the problem now is the name of other fields.

For instance, "result.internalKeyValuePairs.be.score" and "result.internalKeyValuePairs.be.score.reason".

I did do the auto mappings in this scenarios.

But it threw the error like this,

`[2018-03-20T02:31:13,219][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"riskcombdata", :_type=>"eventdata", :_routing=>nil}, #<LogStash::Event:0x229c1f41>], :response=>{"index"=>{"_index"=>"riskcombdata", "_type"=>"eventdata", "_id"=>"g2oaQmIBTNBnrTnHxKUi", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Can't merge a non object mapping [result.internalKeyValuePairs.be.score] with an object mapping [result.internalKeyValuePairs.be.score]"}}}}`

I tried to rename "result.internalKeyValuePairs.be.score" to "result\_internalKeyValuePairs\_be\_score" to avoid ES recognizing it as the object. What I want is just the name which is plain text, no other meaning. But it also threw me the error above.

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 20, 2018, 7:56am UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/124672/4 "2018-03-20T07:56:16Z")

</div>

When you have dots in field names these are expanded as in [the example under the section 'Dots in Field Names'](https://www.elastic.co/blog/elasticsearch-5-0-0-alpha3-released). This means that `result.internalKeyValuePairs.be.score.reason` will require `result.internalKeyValuePairs.be.score` to be mapped as an object. As every field name within an index, e.g. `score`, must have a single mapping, this clashes with `result.internalKeyValuePairs.be.score`, which higher up appears to be mapped to a float.

Replacing the dots in the field names with something else like you have tried should resolve the issue. can you show us the changes you made that did not work?

---

<div class="post-metadata">

### Author: ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)
#### Post date: [March 20, 2018, 5:20pm UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/124672/5 "2018-03-20T17:20:27Z")

</div>

Sure.

This is my updated config file.

```
	file {
		path => "/Users/apple/Desktop/RiskComb/RiskComb.json"
		start_position => "beginning"
		sincedb_path => "/Users/apple/Desktop/RiskComb/read_pos.txt"
		#sincedb_path => "/dev/null"
		codec => json
		

	}
}
filter {
	mutate{
		rename => ["_id" , "ID"]
		remove_field => "_nestedCalls"
		rename => { "result.internalKeyValuePairs.be.anomaly" => "result_internalKeyValuePairs_be_anomaly" }
		rename => { "result.internalKeyValuePairs.re.initialize.currlocation.info" => "result_internalKeyValuePairs_re_initialize_currlocation_info" }
		rename => { "result.internalKeyValuePairs.score.threshold" => "result_internalKeyValuePairs_score_threshold" }
		rename => { "result.internalKeyValuePairs.input.vipaccount" => "result_internalKeyValuePairs_input_vipaccount" }
		rename => { "result.internalKeyValuePairs.be.score" => "result_internalKeyValuePairs_be_score" }
		rename => { "result.riskThreshold" => "result_riskThreshold" }
		rename => { "result.statusMessage" => "result_statusMessage" }
		rename => { "result.policyVersion" => "result_policyVersion" }
		rename => { "result.riskScore" => "result_riskScore" }
		rename => { "result.internalKeyValuePairs.score.actual" => "result_internalKeyValuePairs_score_actual" }
		rename => { "result.internalKeyValuePairs.score.mapping" => "result_internalKeyValuePairs_score_mapping" }
		rename => { "result.eventId" => "result_eventId" }
		rename => { "result.internalKeyValuePairs.input.userid" => "result_internalKeyValuePairs_input_userid" }
		rename => { "result.internalKeyValuePairs.be.score.reason" => "result_internalKeyValuePairs_be_score_reason" }
	}
	date{
		match => ["ts", "yyyy-MM-dd HH:mm:ss.SSS ZZ"]
	}
}
output {
	elasticsearch {
		hosts => "localhost:9200"
		index => "riskcombdata"
		document_type => "eventdata"
	}
	stdout { codec => rubydebug } 
	
}

```

I renamed all those fields from dot to underline.

But still,

`[2018-03-20T02:31:13,219][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"riskcombdata", :_type=>"eventdata", :_routing=>nil}, #<LogStash::Event:0x229c1f41>], :response=>{"index"=>{"_index"=>"riskcombdata", "_type"=>"eventdata", "_id"=>"g2oaQmIBTNBnrTnHxKUi", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Can't merge a non object mapping [result.internalKeyValuePairs.be.score] with an object mapping [result.internalKeyValuePairs.be.score]"}}}}`

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 20, 2018, 5:22pm UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/124672/6 "2018-03-20T17:22:22Z")

</div>

Can you show the output from `rubydebug` so we can see what an event looks like after this transformation?

---

<div class="post-metadata">

### Author: ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)
#### Post date: [March 20, 2018, 5:23pm UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/124672/7 "2018-03-20T17:23:41Z")

</div>

Okay. Here is the output of rubydebug.

```
{
                                                         "jurHash" => "112256955",
          "result.internalKeyValuePairs.re.initialize.preveventid" => "8289018662",
                                                            "host" => "appletekiMacBook-Pro.local",
                                                "result.riskScore" => 0,
                                                       "operation" => "evaluateRisk",
                                                        "@version" => "1",
                      "result.internalKeyValuePairs.score.mapping" => "[0.0, 50.0]->[0.0, 4.0],50.0=4.0,[50.0, 100.0]->[4.0, 12.0]",
                       "result.internalKeyValuePairs.score.actual" => "0.00",
                                                              "ts" => "2018-03-15 20:41:31.909 +0000",
                   "result.internalKeyValuePairs.input.vipaccount" => "112256955",
                                                       "requestId" => "1521146491",
                                            "result.riskThreshold" => 50,
                                                  "result.eventId" => "8301496062",
                                                            "path" => "/Users/apple/Desktop/RiskComb/RiskComb.json",
                         "result.internalKeyValuePairs.be.anomaly" => "false",
                                            "result.policyVersion" => "1.0",
                                                     "wsdlVersion" => "1_8",
                                                  "durationMillis" => 47,
                                                           "txnId" => "vipus8B566BD2EBE87BD8",
                           "result.internalKeyValuePairs.be.score" => "0.09999999999999998",
                                                   "result.status" => "0000",
                       "result.internalKeyValuePairs.input.userid" => "1023145742",
                    "result.internalKeyValuePairs.score.threshold" => "4.00",
                                            "result.statusMessage" => "Success",
                                                      "@timestamp" => 2018-03-15T20:41:31.909Z,
                                                              "ID" => "train1be-d1-tc.1521146491909.2887345",
                                                       "extUserId" => "73M9A9",
    "result.internalKeyValuePairs.re.initialize.currlocation.info" => "united states/michigan/holt:[42.638,-84.522]",
                    "result.internalKeyValuePairs.be.score.reason" => "Transaction exactly matched known good cluster"

```

It seems it did not rename the field

---

<div class="post-metadata">

### Author: ![Pororo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pororo/32/27711_2.png) [@Pororo](https://discuss.elastic.co/u/Pororo)
#### Post date: [March 22, 2018, 1:42am UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/124672/8 "2018-03-22T01:42:51Z")

</div>

I guess if it is because of the sequence of those operations. Like the ingesting happens first and then renaming. What I thought before is the logtash did all mapping settings first and then ingest the data into elasticsearch. But yah, I have been stuck here for two days and have no ideas now:frowning:

Updated: I just deleted some conflict fields and keep the renaming command. It successfully ingested the data but it shows the renaming function did not work at all.

Updated: Problem solved!! It is the syntax of rename function... I think I followed the reference documents of the old version.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 19, 2018, 1:55am UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/124672/9 "2018-04-19T01:55:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
