Logstash Lookup Fields

Elastic Stack Logstash
21

While running /usr/share/logstash/bin/logstash -f .conf file i can see the tag in command line

22

The common add_tag certainly does a sprintf of the new value, so if you end up with "%{[data][tag]}" as a tag that is telling you that the [data] object does not contains a [tag] field].

23

But it contains the tag field in it.

my_file.json

{
        "123456": {
                                "tag": "my_tag"
                        },

my filter looks like

translate {
                    source => "[output_fields][acc_id]"
                    target => "[data]"
                    fallback => '{"acc_id":"not_found"}'
                    dictionary_path => "/home/lookup.json"
                }
if [data] == '{"acc_id":"not_found"}' {
                                                        drop {}
        }
else {
                mutate { add_tag => ["%{[data][tag]}"] }
                }
24

That does not match your fallback option, so it will never drop anything. Any event with the fallback string will also have a "%{[data][tag]}" tag.

25

How i can add tag then?

i wanted to tag then on basis of tag i wanted to send the logs to pipeline

output {
        if "my_tag" in [tags] {
                                pipeline { send_to => ["pipe1"] }
                        }
      if "my_tag2" in [tags] {
                                pipeline { send_to => ["pipe2"] }
                        }
        else {
                pipeline { send_to => ["pipen"] }
                }
}
26

That's how i'm seeing in command line

{
             "data" => {
        "tag" => "my_tag"
    },
        "lookup_id" => "1234",
       "@timestamp" => 2021-11-18T01:32:15.677Z,
         "sequence" => 0,
             "tags" => [
        [0] "my_tag"
    ],
    "output_fields" => {
        "tenant_id" => "1234567890"
    },
         "@version" => "1"
}
27

OK, so that's working. Did you read my previous response explaining why some events will have a "%{[data][tag]}" tag?

28

Yup and i made changes accordingly.

30

I do not understand what needs fixing in that. It is, as far as I can tell, exactly what you wanted.

31

sorry for misunderstandings.
i was asking how i can fix that?

"tags":["%{[data][tag]}"]
32

If you are still getting that value in [tags], what does the rest of the event look like in the rubydebug output?

33

Interesting..
When i use input generator then i can see the tags correctly. But when i use any other input like file then i cannot see the tag in this way "tags":["%{[data][tag]}"]

34

This is the output

{
    "@timestamp" => 2021-11-18T05:21:34.609Z,
       "message" => "{\"name\": \"test\", \"output_fields\": {\"acc_id\": \"123456\"}, \"lookup_id\": \"hj\"}",
          "path" => "/home/test.log"
}
35

Well now I am completely confused because that has no [data] field and no [tags] at all.

36

I was able to fix that. the issue was stringf was not working in translate filter.

37

Another issue i'm getting i'm seeing duplicate data after applying Translate filter.