Could not index event to Elasticsearch. "reason"=>"if _id is specified it must not be empty"}

Elastic Stack Logstash
1

Hello,

I have setup a logstash pipeline. It works fine bringing the updates and refreshing the index. However, it throws an error while bringing updates. I am trying to understand the reasons behind.

output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "contactsx"
document_type => "contactx"
document_id => "%{symbol}"
}

[2020-05-25T11:59:13,962][WARN ][logstash.outputs.elasticsearch][company_listing] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"", :_index=>"contactsx", :routing=>nil, :_type=>"contactx"}, #LogStash::Event:0x56578469], :response=>{"index"=>{"_index"=>"contactsx", "_type"=>"contactx", "_id"=>"", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"if _id is specified it must not be empty"}}}}

by changing it to this "with a constant "_Z", document_id => "%{symbol}_z", there are no errors.

output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "contactsx"
document_type => "contactx"
document_id => "%{symbol}_z"
}

Can you please explain what's going on here?
The column symbol is primary key in the db.
Why does logstash say id is NULL in the first setting?

2

Apparently you have an event where the symbol field exists but is empty.

3

Symbol is one of the columns in the SQL statement. And it is primary key. So it's not empty or null, also it's unique

4

When you use

 document_id => "%{symbol}_z"

it seems likely you have a document with id equal to "_z". What does that document look like if check the JSON tab in an expanded event in the Discover interface of kibana?

5

The pipeline config , and the sql for document_id => "%{symbol}_z"
and document_id => "%{symbol}".

The sql has a few columns from a table, with symbol being the primary key, so it's unique and not null.

However, when I set document_id => "%{symbol}" it gives the aforementioned error.

Since it said NULL for id, I thought let me add a constant like "_Z" and with this added, there are no errors.

The end result in both cases are the same. However, one gives a WARNING in the logs, while the other is clear.

6

Right. What does the resulting document look like?

7

With _Z appended to the pipeline directive

symbol:
HCHC
@version:
1
@timestamp:
2020-05-27
companyname:
HC2 Holdings, Inc.
_id:
HCHC_z
_type:
contactx
_index:
contactsx
_score:
0
################################################################

Without _Z appended
#####################

symbol:
HCHC
companyname:
HC2 Holdings, Inc.
@timestamp:
2020-05-27
@version:
1
_id:
HCHC
_type:
contactx
_index:
contactsx
_score:
0

8

You are saying that the "Without _Z appended" event results in an illegal_argument_exception?

9

yes. exactly.

10

I cannot imagine why elasticsearch would do that. I suggest you ask a question over in the elasticsearch forum.

11

Ok thanks

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.