Combine and enrich data before ingestion

Elastic Stack Elasticsearch
1

Hello,

I have a web application that generate log files which are then ingested to via Beat-Logstash to ElasticSearch. Now I need to combine information from different log lines or even different log files before insert the log documents to elasticsearch.

E.g. the username below appears in line 50 but I need to also include it in the document of line 100, which refers to a user action. The glue between the two lines in the session-id.

50: [2018-12-13 13:45:32] [INFO] [Yud52I234hGVgdj] User with username=test loggedin with success
51: [2018-12-13 13:45:33] [INFO] [Yud52I234hGVgdj] Initializing user assets
...
100: [2018-12-13 13:48:15] [INFO] [Yud52I234hGVgdj] Page Catalogue entered

I want username from line 50 to populate to the document of line 100.

Are there approaches to enrich the data during ingest, other than add the username in the log of line 100?

Thank you!

2

If you are looking to analyse behaviour of sessions taking an entity-centric indexing approach may be of interest.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.