# Logstash Lookup Fields

**URL:** <https://discuss.elastic.co/t/logstash-lookup-fields/289357>\
**Category:** Logstash\
**Created:** [November 16, 2021, 6:02pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357 "2021-11-16T18:02:26Z")\
**Posts on this page:** 17\
**Page:** 2

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 18, 2021, 12:43am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/21 "2021-11-18T00:43:18Z")

</div>

While running `/usr/share/logstash/bin/logstash -f .conf file` i can see the tag in command line

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 18, 2021, 12:46am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/22 "2021-11-18T00:46:36Z")

</div>

The common add\_tag certainly [does a sprintf](https://github.com/elastic/logstash/blob/c3e498a24becced85ed274fbe25e357a7dd087db/logstash-core/lib/logstash/util/decorators.rb#L57) of the new value, so if you end up with "%{[data][tag]}" as a tag that is telling you that the [data] object does not contains a [tag] field].

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 18, 2021, 12:49am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/23 "2021-11-18T00:49:40Z")

</div>

But it contains the tag field in it.

my\_file.json

```auto
{
        "123456": {
                                "tag": "my_tag"
                        },

```

my filter looks like

```auto
translate {
                    source => "[output_fields][acc_id]"
                    target => "[data]"
                    fallback => '{"acc_id":"not_found"}'
                    dictionary_path => "/home/lookup.json"
                }
if [data] == '{"acc_id":"not_found"}' {
                                                        drop {}
        }
else {
                mutate { add_tag => ["%{[data][tag]}"] }
                }

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 18, 2021, 1:03am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/24 "2021-11-18T01:03:02Z")

</div>

> [@John\_snow](#):
>
> `if [data] == '{"acc_id":"not_found"}' {`

That does not match your fallback option, so it will never drop anything. Any event with the fallback string will also have a "%{[data][tag]}" tag.

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 18, 2021, 1:09am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/25 "2021-11-18T01:09:44Z")

</div>

How i can add tag then?

i wanted to tag then on basis of tag i wanted to send the logs to pipeline

```auto
output {
        if "my_tag" in [tags] {
                                pipeline { send_to => ["pipe1"] }
                        }
      if "my_tag2" in [tags] {
                                pipeline { send_to => ["pipe2"] }
                        }
        else {
                pipeline { send_to => ["pipen"] }
                }
}

```

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 18, 2021, 1:33am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/26 "2021-11-18T01:33:16Z")

</div>

That's how i'm seeing in command line

```auto
{
             "data" => {
        "tag" => "my_tag"
    },
        "lookup_id" => "1234",
       "@timestamp" => 2021-11-18T01:32:15.677Z,
         "sequence" => 0,
             "tags" => [
        [0] "my_tag"
    ],
    "output_fields" => {
        "tenant_id" => "1234567890"
    },
         "@version" => "1"
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 18, 2021, 1:43am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/27 "2021-11-18T01:43:36Z")

</div>

> [@John\_snow](#):
>
> ```auto
> "tags" => [
> [0] "my_tag"
> ],
> 
> ```

OK, so that's working. Did you read my previous response explaining why some events will have a "%{[data][tag]}" tag?

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 18, 2021, 1:47am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/28 "2021-11-18T01:47:25Z")

</div>

Yup and i made changes accordingly.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 18, 2021, 1:49am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/30 "2021-11-18T01:49:37Z")

</div>

I do not understand what needs fixing in that. It is, as far as I can tell, exactly what you wanted.

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 18, 2021, 1:56am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/31 "2021-11-18T01:56:14Z")

</div>

> [@John\_snow](#):
>
> `"tags":["%{[data][tag]}"]`

sorry for misunderstandings.  
i was asking how i can fix that?

```auto
"tags":["%{[data][tag]}"]

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 18, 2021, 2:53am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/32 "2021-11-18T02:53:42Z")

</div>

> [@John\_snow](#):
>
> `"tags":["%{[data][tag]}"]`

If you are still getting that value in [tags], what does the rest of the event look like in the rubydebug output?

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 18, 2021, 3:34am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/33 "2021-11-18T03:34:47Z")

</div>

Interesting..  
When i use input `generator` then i can see the tags correctly. But when i use any other input like `file` then i cannot see the tag in this way `"tags":["%{[data][tag]}"]`

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 18, 2021, 1:42pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/34 "2021-11-18T13:42:01Z")

</div>

This is the output

```auto
{
    "@timestamp" => 2021-11-18T05:21:34.609Z,
       "message" => "{\"name\": \"test\", \"output_fields\": {\"acc_id\": \"123456\"}, \"lookup_id\": \"hj\"}",
          "path" => "/home/test.log"
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 18, 2021, 3:21pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/35 "2021-11-18T15:21:18Z")

</div>

Well now I am completely confused because that has no [data] field and no [tags] at all.

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 18, 2021, 5:15pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/36 "2021-11-18T17:15:16Z")

</div>

I was able to fix that. the issue was stringf was not working in translate filter.

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 18, 2021, 6:30pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/37 "2021-11-18T18:30:57Z")

</div>

Another issue i'm getting i'm seeing duplicate data after applying Translate filter.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 16, 2021, 6:35pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/39 "2021-12-16T18:35:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/logstash-lookup-fields/289357.md?page=1)