Enriching log data with database or other sources

Elastic Stack Logstash
1

Hey,

I could use some help please. I have two questions

I have log data coming in (Log4J, Log4Net etc etc) and this contains a user ID, for example “12345”

I want to enrich each log line with more information about the user, such as email, work location, job title etc. These details are present in another database, SQL or potentially in future other sources.

But on the simple database use case.

Question 1: What would I do pipeline wise in Logstash so as the logs come in the log data is enriched / log lines extended by connecting to the DB, matching the user ID from the log line, querying that users details and adding certain fields of the user details from the DB to the log line

Question 2: Certain information, let’s say a combination of 3 pieces of data from the log line will determine a certain field to fetch from a DB to enrich a log line. So it’s only in certain combinations of data in the log line that id want to grab certain data from the database and enhance / enrich the log line. What’s best way to do that?

Thanks all, big love

2

Hello and welcome,

You could use the jdbc_streaming filter plugin to query a database while indexing.

You can use conditionals in logstash and check if you have the information you need and only then run the jdbc_streaming filter plugin.

What you want to do is possible with logstash, but how you will do that depend on many factors like how you are receiving your data, how you are parsing it, what you want to enrich, how the source data for the enrich looks like.

You would need to provide a lot more of context and examples to have a better answer.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.