Insert in to existing index using some condition

Elastic Stack Logstash
1

Here is what I am trying to do
I have a existing index with

{ 
    index= my_index
    id=1
    fname= xyz
    lname = abc
    emp_num=123
    loction=usa
}

now i am reading new data from new source A
which have fname, lname, address

I would like to insert that address in to elk record if fname and lname match

{
   index=my_index
   id=1
   fname= xyz
    lname = abc
    emp_num=123
    loction=usa
    address=new_address_here
}

how do I do this? would like to use logstash. and source is sqlserver. connection string is already establish.

2

any idea?

3

after your jdbc , you can use elasticsearch filter to query the existing document in ES, add the new field and use update mode on elasticsearch output.

4

can I simple use translate filter?
where my dictionary file is something like this.

fname:lnama:address

5

is this syntax looks ok?
I am getting

[0] "_elasticsearch_lookup_failure"

 elasticsearch {
      hosts => ["elkdev01:9200"]
      index => "fw_lifespan"
      user => "${elastic_user}"
      password => "${elastic_password}"

      #here I want to also compare fw_vendor.keyword
      query => 'fw_version.keyword:"%{[firmware]}"' 

     # if match would like to take this two field from fw_lifespan index
      fields => [ "fw_recommended","fw_expiration_date"] }
   }
6

the syntax seems ok to me. do you have user and password stored as local variables ? does the log showed failure reason ?

7

yes that username/password works. I can connect to it

8

no errors in the log? maybe increase log level to debug

9

no error. it is printing this
[0] "_elasticsearch_lookup_failure"

this shows up only on few document not on all documents

10

are you able to search the failed docs in elasticsearch?
if yes, I would increase the log level in loglevel to debug or trace to see why it fails

11

After lot of testing, it is working now. here is final configuration

filter{

   elasticsearch {
      hosts => ["hostname:9200"]
      index => "fw_lifespan"
      user => "${elastic_user}"
      password => "${elastic_password}"
      query => "fw_version:%{firmware} AND fw_vendor:%{vendor}"
      fields => { "fw_recommended" => "fw_recommended"
                  "fw_expiration_date" => "fw_expiration_date" }
   }
}

two new field will come from fw_lifespan index to new index if it match firmware and vendor.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.