Best practice for adding user profile information to event stream

[danielmcquillen]

Is there a best practice in logstash for adding extra user information to incoming realtime log events?

I'm wondering what the best way is to lookup, for example, MySQL user profile data by the incoming user_id and add profile information to incoming event.

Or is that kind of operation better done in some kind of periodic reindexing operation? (The analytics don't have to be real-time.)

Either way, the event is stored in ES with extra user profile info so we can, say, group by certain user characteristics when analyzing data, etc.

[warkolm (Mark Walkom)]

You can use a SQL store, you could also store that user info in Elasticsearch and then use the Elasticsearch filter to add it in your Logstash pipeline.

That'd be cheaper than reindexing to add it.