Using external document_id

Elastic Stack Logstash
1

Hi,

Whenever i try to assign the document_id with some attribute from the documents, it creates just one new document on ElasticSearch with the document_id as the name of the attribute i'm trying to use. What i want is the actual amount of documents retrieved, with the document_id set as the primary key from the source database.

output {
elasticsearch {
hosts => ["placeholder"]
index => "item"
document_id => "%{i.RecordId}"
doc_as_upsert => true
user => "placeholder"
password => "placeholder"
}
}

specifically, creates a new document that looks like this;

"_index" : "item",
"_type" : "_doc",
"_id" : "%{i.RecordId}",
"_score" : 1.0,
"_source" : {

I've tried a few different approaches, like copying the recordid into a metadata field, but nothing has worked so far.

Thanks for the help in advance

2

If you add

output { stdout { codec => rubydebug } }

then what does an event look like?

3

"recordid" => 5259,
"itemnumber2" => nil,
"netprice" => 66.48,
"isstandard" => false,
"ean" => "4008321515285",
"@version" => "1",
"fk_unit" => 6,
"isactive" => true,
"itemonstock" => 0.0,
"priority" => 1,
"@timestamp" => 2020-03-03T08:11:00.437Z,
"itemalias" => nil,
"wholesalername" => "placeholder",
"discountamount" => 16.62,
"unitname" => "Styk",
"dateupdated" => nil,
"fk_catalog" => 2,
"name" => "placeholder",
"itemnumbernumerical" => 5651001372,
"pricematrixitemid" => 2899,
"datecreated" => 2018-03-21T16:43:17.550Z,
"fk_catalogitemgroup" => 145,
"sellingprice" => 132.96,
"wholesalerid" => 6,
"itemnumber" => "5651001372",
"issellingpricecalculated" => true,
"markedfordelete" => nil,
"allowancecharge" => nil,
"altdescription" => nil,
"discountpercent" => 20.0,
"description" => "placeholder",
"grossprice" => 83.1
}

4

It looks like you have a field called [recordid]. You do not have a field called [i.RecordId].

5

everytime you connect to sqlserver/oracle/mysql and combine multiple table. record gets presented to elk from tablename.fieldname to fieldname (only) and all in lowercase

6

The problem is, however, that no matter how i format it, it ends up using the name of the attribute as the document_id, all the ways that have been suggested to me have not worked so far. Creating a new metadata field ends up adding "recordid" as the document id as well, for an example.

I can't help but wonder if i simply just write it wrong - i'm fairly new to Logstash, so this is completely possible though

filter
{
mutate
{
add_field => {"[@metadata][id]" => "%[recdordid]"
}
}
}

output {
elasticsearch {
hosts => ["placeholder"]
index => "item"
doc_as_upsert => true
document_id => "%{[@metadata][id]}"
user => "placeholder"
password => "placeholder"
}
}

as an example

7

The field appears to be called recordid, not recdordid. There is no need to use [@metadata], you can reference the field directly.

8

"_index" : "item",
"_type" : "_doc",
"_id" : "%[recordid]",
"_score" : 1.0,
"_source" : {

Is the output i get when just trying to reference the field. Again, i'm new to logstash and i don't know if this fits the syntax, but it's what i have been suggested to write, and it doesn't seem to work the way i want it to

9

You need braces in a sprintf reference, so it should be %{[recordid]}. (In this case the brackets are optional.)

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.