Logstash Lookup Fields

Elastic Stack Logstash
21

While running /usr/share/logstash/bin/logstash -f .conf file i can see the tag in command line

22

The common add_tag certainly does a sprintf of the new value, so if you end up with "%{[data][tag]}" as a tag that is telling you that the [data] object does not contains a [tag] field].

23

But it contains the tag field in it.

my_file.json

{
        "123456": {
                                "tag": "my_tag"
                        },

my filter looks like

translate {
                    source => "[output_fields][acc_id]"
                    target => "[data]"
                    fallback => '{"acc_id":"not_found"}'
                    dictionary_path => "/home/lookup.json"
                }
if [data] == '{"acc_id":"not_found"}' {
                                                        drop {}
        }
else {
                mutate { add_tag => ["%{[data][tag]}"] }
                }
24

That does not match your fallback option, so it will never drop anything. Any event with the fallback string will also have a "%{[data][tag]}" tag.

25

How i can add tag then?

i wanted to tag then on basis of tag i wanted to send the logs to pipeline

output {
        if "my_tag" in [tags] {
                                pipeline { send_to => ["pipe1"] }
                        }
      if "my_tag2" in [tags] {
                                pipeline { send_to => ["pipe2"] }
                        }
        else {
                pipeline { send_to => ["pipen"] }
                }
}
26

That's how i'm seeing in command line

{
             "data" => {
        "tag" => "my_tag"
    },
        "lookup_id" => "1234",
       "@timestamp" => 2021-11-18T01:32:15.677Z,
         "sequence" => 0,
             "tags" => [
        [0] "my_tag"
    ],
    "output_fields" => {
        "tenant_id" => "1234567890"
    },
         "@version" => "1"
}
27

OK, so that's working. Did you read my previous response explaining why some events will have a "%{[data][tag]}" tag?

28

Yup and i made changes accordingly.

30

I do not understand what needs fixing in that. It is, as far as I can tell, exactly what you wanted.

31

sorry for misunderstandings.
i was asking how i can fix that?

"tags":["%{[data][tag]}"]
32

If you are still getting that value in [tags], what does the rest of the event look like in the rubydebug output?

33

Interesting..
When i use input generator then i can see the tags correctly. But when i use any other input like file then i cannot see the tag in this way "tags":["%{[data][tag]}"]

34

This is the output

{
    "@timestamp" => 2021-11-18T05:21:34.609Z,
       "message" => "{\"name\": \"test\", \"output_fields\": {\"acc_id\": \"123456\"}, \"lookup_id\": \"hj\"}",
          "path" => "/home/test.log"
}
35

Well now I am completely confused because that has no [data] field and no [tags] at all.

36

I was able to fix that. the issue was stringf was not working in translate filter.

37

Another issue i'm getting i'm seeing duplicate data after applying Translate filter.

39

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.