Could not index event to Elasticsearch

Elastic Stack Logstash
1

This is my json item,

47
屏幕快照 2018-03-19 下午6.40.47.jpg1668×1938 701 KB

And this is my config

input {
	file {
		path => "/Users/apple/Desktop/RiskComb/RiskComb.json"
		start_position => "beginning"
		sincedb_path => "/Users/apple/Desktop/RiskComb/read_pos.txt"
		#sincedb_path => "/dev/null"
		codec => json
		

	}
}

filter {

	mutate{
		rename => ["_id" , "ID"]
		rename => ["_nestedCalls" , "nestedCalls"]
	}
	date{
		match => ["ts", "yyyy-MM-dd HH:mm:ss.SSS ZZ"]
	}

}

output {
	elasticsearch {
		hosts => "localhost:9200"
		index => "riskcombbdata"
		document_type => "eventdata"
	}
	stdout { codec => rubydebug } 
	
}

The error info is
[2018-03-19T23:03:15,552][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"riskcombbdata", :_type=>"eventdata", :_routing=>nil}, #<LogStash::Event:0x4df9660b>], :response=>{"index"=>{"_index"=>"riskcombbdata", "_type"=>"eventdata", "_id"=>"a2pcQWIBTNBnrTnHXqWR", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [nestedCalls.result.internalKeyValuePairs.be.score] of different type, current_type [text], merged_type [ObjectMapper]"}}}}

I am not sure how to deal with this problem.

1 Like
2

In Elasticsearch, a given field must be consistent in type across an entire index; Elasticsearch is rejecting your request because the field nestedCalls.result.internalKeyValuePairs.be.score is different than the current mapping, which was likely auto-detected from a previous document.

3

This is my current structure.

15
屏幕快照 2018-03-20 上午2.34.15.jpg1718×1456 543 KB

And I removed the field("_nestedCalls") in the filter of logstash.

So it seems like the problem now is the name of other fields.

For instance, "result.internalKeyValuePairs.be.score" and "result.internalKeyValuePairs.be.score.reason".

I did do the auto mappings in this scenarios.

But it threw the error like this,

[2018-03-20T02:31:13,219][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"riskcombdata", :_type=>"eventdata", :_routing=>nil}, #<LogStash::Event:0x229c1f41>], :response=>{"index"=>{"_index"=>"riskcombdata", "_type"=>"eventdata", "_id"=>"g2oaQmIBTNBnrTnHxKUi", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Can't merge a non object mapping [result.internalKeyValuePairs.be.score] with an object mapping [result.internalKeyValuePairs.be.score]"}}}}

I tried to rename "result.internalKeyValuePairs.be.score" to "result_internalKeyValuePairs_be_score" to avoid ES recognizing it as the object. What I want is just the name which is plain text, no other meaning. But it also threw me the error above.

4

When you have dots in field names these are expanded as in the example under the section 'Dots in Field Names'. This means that result.internalKeyValuePairs.be.score.reason will require result.internalKeyValuePairs.be.score to be mapped as an object. As every field name within an index, e.g. score, must have a single mapping, this clashes with result.internalKeyValuePairs.be.score, which higher up appears to be mapped to a float.

Replacing the dots in the field names with something else like you have tried should resolve the issue. can you show us the changes you made that did not work?

5

Sure.

This is my updated config file.

	file {
		path => "/Users/apple/Desktop/RiskComb/RiskComb.json"
		start_position => "beginning"
		sincedb_path => "/Users/apple/Desktop/RiskComb/read_pos.txt"
		#sincedb_path => "/dev/null"
		codec => json
		

	}
}
filter {
	mutate{
		rename => ["_id" , "ID"]
		remove_field => "_nestedCalls"
		rename => { "result.internalKeyValuePairs.be.anomaly" => "result_internalKeyValuePairs_be_anomaly" }
		rename => { "result.internalKeyValuePairs.re.initialize.currlocation.info" => "result_internalKeyValuePairs_re_initialize_currlocation_info" }
		rename => { "result.internalKeyValuePairs.score.threshold" => "result_internalKeyValuePairs_score_threshold" }
		rename => { "result.internalKeyValuePairs.input.vipaccount" => "result_internalKeyValuePairs_input_vipaccount" }
		rename => { "result.internalKeyValuePairs.be.score" => "result_internalKeyValuePairs_be_score" }
		rename => { "result.riskThreshold" => "result_riskThreshold" }
		rename => { "result.statusMessage" => "result_statusMessage" }
		rename => { "result.policyVersion" => "result_policyVersion" }
		rename => { "result.riskScore" => "result_riskScore" }
		rename => { "result.internalKeyValuePairs.score.actual" => "result_internalKeyValuePairs_score_actual" }
		rename => { "result.internalKeyValuePairs.score.mapping" => "result_internalKeyValuePairs_score_mapping" }
		rename => { "result.eventId" => "result_eventId" }
		rename => { "result.internalKeyValuePairs.input.userid" => "result_internalKeyValuePairs_input_userid" }
		rename => { "result.internalKeyValuePairs.be.score.reason" => "result_internalKeyValuePairs_be_score_reason" }
	}
	date{
		match => ["ts", "yyyy-MM-dd HH:mm:ss.SSS ZZ"]
	}
}
output {
	elasticsearch {
		hosts => "localhost:9200"
		index => "riskcombdata"
		document_type => "eventdata"
	}
	stdout { codec => rubydebug } 
	
}

I renamed all those fields from dot to underline.

But still,

[2018-03-20T02:31:13,219][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"riskcombdata", :_type=>"eventdata", :_routing=>nil}, #<LogStash::Event:0x229c1f41>], :response=>{"index"=>{"_index"=>"riskcombdata", "_type"=>"eventdata", "_id"=>"g2oaQmIBTNBnrTnHxKUi", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Can't merge a non object mapping [result.internalKeyValuePairs.be.score] with an object mapping [result.internalKeyValuePairs.be.score]"}}}}

6

Can you show the output from rubydebug so we can see what an event looks like after this transformation?

7

Okay. Here is the output of rubydebug.

{
                                                         "jurHash" => "112256955",
          "result.internalKeyValuePairs.re.initialize.preveventid" => "8289018662",
                                                            "host" => "appletekiMacBook-Pro.local",
                                                "result.riskScore" => 0,
                                                       "operation" => "evaluateRisk",
                                                        "@version" => "1",
                      "result.internalKeyValuePairs.score.mapping" => "[0.0, 50.0]->[0.0, 4.0],50.0=4.0,[50.0, 100.0]->[4.0, 12.0]",
                       "result.internalKeyValuePairs.score.actual" => "0.00",
                                                              "ts" => "2018-03-15 20:41:31.909 +0000",
                   "result.internalKeyValuePairs.input.vipaccount" => "112256955",
                                                       "requestId" => "1521146491",
                                            "result.riskThreshold" => 50,
                                                  "result.eventId" => "8301496062",
                                                            "path" => "/Users/apple/Desktop/RiskComb/RiskComb.json",
                         "result.internalKeyValuePairs.be.anomaly" => "false",
                                            "result.policyVersion" => "1.0",
                                                     "wsdlVersion" => "1_8",
                                                  "durationMillis" => 47,
                                                           "txnId" => "vipus8B566BD2EBE87BD8",
                           "result.internalKeyValuePairs.be.score" => "0.09999999999999998",
                                                   "result.status" => "0000",
                       "result.internalKeyValuePairs.input.userid" => "1023145742",
                    "result.internalKeyValuePairs.score.threshold" => "4.00",
                                            "result.statusMessage" => "Success",
                                                      "@timestamp" => 2018-03-15T20:41:31.909Z,
                                                              "ID" => "train1be-d1-tc.1521146491909.2887345",
                                                       "extUserId" => "73M9A9",
    "result.internalKeyValuePairs.re.initialize.currlocation.info" => "united states/michigan/holt:[42.638,-84.522]",
                    "result.internalKeyValuePairs.be.score.reason" => "Transaction exactly matched known good cluster"

It seems it did not rename the field

8

I guess if it is because of the sequence of those operations. Like the ingesting happens first and then renaming. What I thought before is the logtash did all mapping settings first and then ingest the data into elasticsearch. But yah, I have been stuck here for two days and have no ideas now:frowning:

Updated: I just deleted some conflict fields and keep the renaming command. It successfully ingested the data but it shows the renaming function did not work at all.

Updated: Problem solved!! It is the syntax of rename function... I think I followed the reference documents of the old version.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.