Extract document id into another variable in logstash

Elastic Stack Logstash
1

I have a need to get document id for a particular condition. Get the document_id from the document matching the query in elasticsearch input and assign it to a variable and later use that document id to delete a document through elasticsearch output plugin. How can i do this ?

2

With the elasticsearch input's docinfo option you can get the document id and other metadata extracted to fields in the events it creates.

3

My input is an xml document which i am going to send over tcp. I usually parse the xml and store it as event in elasticearch. Now for this condition also i have to parse the input through tcp with xml filter and then use one particular field to get the previous event. I used the elasticsearch filter plugin where i can use fields option to copy data from previous event to new event. Problem is when i use this condition the tcp is not working.

 tcp {
            port => 5043
            type => "tcp"
    }

xml {
                    store_xml => true
                    source => message
                    target => parsed_message
                    force_array => false
                    remove_field => "message"
            }
if [parsed_message][NotificationType] == "CLEAR" {
                    elasticsearch {
                            hosts => ["x.x.x.x:9200","x.x.x.x:9200"]
                            user => someuser
                            password => xxxxxxx
                            index => "previousevent"
                            query => "parsed_message.Name: %{[parsed_message][Name]} AND last_notified_at: %{last_notified_at}"
                            fields => {"document_id" => ref_id }
                    }
            }
4

Problem is when i use this condition the tcp is not working.

What do you mean?

5

I got it working. The error was because the filter elasticsearch was not installed.

6

Now the problem is the filter is querying elasticsearch but the "ref_id" is always coming as nil. Could someone help ?

7

My issue is the same as in Logstash elasticsearch filter error TypeError: can't convert nil into String

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.