# Logstash Lookup Fields

**URL:** <https://discuss.elastic.co/t/logstash-lookup-fields/289357>\
**Category:** Logstash\
**Created:** [November 16, 2021, 6:02pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357 "2021-11-16T18:02:26Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 16, 2021, 6:02pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/1 "2021-11-16T18:02:26Z")

</div>

i wanted to use a lookup table in Logstash to check if account id exisits in lookub table then it will grab the output location from the lookup file. e--g

```auto
Lookup File
account_id, output_location, secrets,
123, s3, abcdef

```

i want to apply that in output to see if account exists in lookup table/file then only it will send the message to output

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 16, 2021, 6:12pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/2 "2021-11-16T18:12:10Z")

</div>

You can use a translate filter to do the lookup, then `drop {}` if the event contains the fallback value.

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 16, 2021, 6:43pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/3 "2021-11-16T18:43:30Z")

</div>

Can we use dynamo db or any other aws service for lookup?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 16, 2021, 7:18pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/4 "2021-11-16T19:18:45Z")

</div>

You might be able to call the Dynamo API using an http filter, and you can do pretty much anything in a ruby filter. So it is very likely possible, but I have no experience and cannot offer any advice.

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 16, 2021, 7:20pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/5 "2021-11-16T19:20:12Z")

</div>

Can i defined my lookup file like this?

```auto
{
        "123456" : {
                                "bucket" : "my_bucket",
                                "iam_role": "iam_role",
                                "region": "us-east-1",
                                },
        789012" : {
                                "bucket" : "my_bucket2",
                                "iam_role": "iam_role2",
                                "region": "us-east-2",
                                }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 16, 2021, 7:29pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/6 "2021-11-16T19:29:03Z")

</div>

I believe so. [This](https://alexmarquardt.com/enriching-data-with-the-logstash-translate-filter/) blog has an example of using a JSON dictionary.

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 16, 2021, 9:22pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/7 "2021-11-16T21:22:57Z")

</div>

Thanks for the sharing. That was very helpful.  
i stored all values in data as

```auto
source => "[acc_id]"
target => "[data]"

```

How i can get those values in output and assign to variables?

```auto
         "data" => {
        "secret_access_key" => "abc",
            "access_key_id" => "def",
                   "bucket" => "my_bucket",
                   "region" => "us-east-1",
                   "prefix" => "nnew"
    },
    "acc_id" => "1234567890",
    "lookup_id" => "1234",
     "sequence" => 0
}

```

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 16, 2021, 9:24pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/8 "2021-11-16T21:24:16Z")

</div>

I'm trying to do like this and getting error

```auto
access_key_id => "%[data][access_key_id]"

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 16, 2021, 11:06pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/9 "2021-11-16T23:06:05Z")

</div>

> [@John\_snow](#):
>
> `"%[data][access_key_id]"`

Try `"%{[data][access_key_id]}"` . You need the {} for a sprintf reference.

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 16, 2021, 11:15pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/10 "2021-11-16T23:15:20Z")

</div>

Thanks for the response again.  
translate filter enriched the message which i don't want. I want to look into lookup file on basis of account id and if account id exists, then want to grab all the secrets defined against that account id and then use those secrets to send the actual event to output.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 16, 2021, 11:57pm UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/11 "2021-11-16T23:57:28Z")

</div>

> [@John\_snow](#):
>
> translate filter enriched the message which i don't want.

If you use `target => "[@metadata][secrets]"` you can use sprintf references to it but it will not get sent by the output with the rest of the fields on the event.

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 17, 2021, 12:40am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/12 "2021-11-17T00:40:23Z")

</div>

I tried it but still getting the error.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 17, 2021, 1:15am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/13 "2021-11-17T01:15:00Z")

</div>

What error message are you getting?

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 17, 2021, 1:19am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/14 "2021-11-17T01:19:22Z")

</div>

s3 - Uploading failed, retrying (#5 of Infinity) {:exception=\>Aws::S3::Errors::InvalidAccessKeyId, message=\>"The AWS Access Key Id you provided does not exist in our records."

after running ` /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/lookup_test.conf` my message looks like

```auto
{
             "data" => {
        "secret_access_key" => "def",
                   "bucket" => "my_bucket",
                   "region" => "us-east-1",
            "access_key_id" => "abc
    },
        "lookup_id" => "1234",
    "output_fields" => {
        "account_id" => "123456789012"
    },
         "sequence" => 0
}

```

and my output plugin looks like

```auto
output {
        stdout { codec => "rubydebug" }
        s3 {
                access_key_id => "%{[data][access_key_id]}"
                other info ....
       }
}

```

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 17, 2021, 1:26am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/15 "2021-11-17T01:26:51Z")

</div>

Here is actual log,

```auto
{"output_fields": {"account_id": "123456789012"}, "lookup_id": "1234"}

```

adding translate filter

```auto
translate {
                    source => "[output_fields][account_id]"
                    target => "[data]"
                    fallback => '{"tenant_id":"not_found"}'
                    dictionary_path => "/home/lookup.json"
                }

```

log after translate

```auto
{
             "data" => {
        "secret_access_key" => "def",
                   "bucket" => "my_bucket",
                   "region" => "us-east-1",
            "access_key_id" => "abc
    },
        "lookup_id" => "1234",
    "output_fields" => {
        "account_id" => "123456789012"
    },
         "sequence" => 0
}

```

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 17, 2021, 1:31am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/16 "2021-11-17T01:31:12Z")

</div>

/home/lookup.json file looks like

```auto
{
        "123456789012": {
                                "access_key_id": "abc",
                                "secret_access_key": "def",
                                "bucket" : "my_bucket"
                                }

}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 17, 2021, 1:32am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/17 "2021-11-17T01:32:03Z")

</div>

> [@John\_snow](#):
>
> `access_key_id => "%{[data][access_key_id]}"`

😮 The output [does not](https://github.com/logstash-plugins/logstash-mixin-aws/blob/8606fb445df5698ee214781dacbefa213ef9e782/lib/logstash/plugin_mixins/aws_config/v2.rb#L41) sprintf the access\_key\_id! It has to be a constant.

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 17, 2021, 1:36am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/18 "2021-11-17T01:36:07Z")

</div>

Oh,  
my use case is to send logs to different aws account on basis of account id. I need to change the bucket name and secrets

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 17, 2021, 2:20am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/19 "2021-11-17T02:20:21Z")

</div>

You could modify the plugin to make the sprintf call and build it yourself.

If you have a small number of different buckets you could use an if else in the output section to choose which s3 output to route to.

---

<div class="post-metadata">

**Author:** ![John\_snow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_snow/32/77761_2.png) [@John\_snow](https://discuss.elastic.co/u/John_snow)\
**Post date:** [November 18, 2021, 12:41am UTC](https://discuss.elastic.co/t/logstash-lookup-fields/289357/20 "2021-11-18T00:41:39Z")

</div>

Hey,  
I'm adding tags to my log using sprintf but when i try to pass logs but it seems it is not adding the tag.  
that's how 'm passing the tag

```auto
if [data] == '{"acc_id":"not_found"}' {
                                                        drop {}
        }
        else {
                mutate { add_tag => ["%{[data][tag]}"] }
                }

```

But getting tags like this in my output file

```auto
"tags":["%{[data][tag]}"]

```

[Next page](https://discuss.elastic.co/t/logstash-lookup-fields/289357.md?page=2)