Logstash cache lookup data

Elastic Stack Logstash
1

Hi,

I am moving a legacy logging system to Elastic stack, and there are couple fields in the existing system that are using foreign keys for a lookup table (around 1000 rows, but it can grow or shrink since users can create new items). The lookup tables are simple, having just the ID and the description.

I am collecting all the new logs coming to the system, but some heavy users already expressed some concerns, because when they search data in Kibana, they only get the ID, and not the description, making it hard to name the source of the logs.

I imported the lookup table to elasticsearch, and using the elasticsearch filter, I am getting the description for each log, adding it to the output. Now I have the ID and the description, however I am overloading the system with search requests, for each log coming to the system.

I was wondering if there is a way to cache some data in logstash, so I don't have to go every single time to elasticsearch. That would be way more efficient.

I was thinking adding that lookup table in a local Redis instance, but not sure if that's the best practice.

Thanks,
Rob

2

The jdbc_static filter plugin might be a good option if the data set is reasonably in size and does not change very frequently. The jdbc_streaming filter plugin might also be an option. These both cache data, so should take some load off your cluster.

This webinar contains a discussion about the various enrichment plugins available.

2 Likes
3

Hi Christian,

Thanks for the quick response really appreciate. I looked at the plugin, and it seems that it is indeed a good fit, since according to the documentation, it will keep one instance for the JVM.

It would be nice having the same functionality from elasticsearch, instead of a database, because I'll have to keep a table somewhere just for that (I already have that information in elasticsearch).

I will also check if there is a way to get the database/password from a config file, instead of hardcoded in my logstash configuration, since it will be used in multiple processes.

Thanks a lot!
Rob

4

Christian,

The webinar that you added to your post is great. Now it's very clear. Thanks a lot!

Cheers,
Rob
P.S.: Still think that a similar plugin to load the lookup table from elasticsearch would be better for me :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.