Update Dataset with Information comming delayed

Elastic Stack Elasticsearch
1

Hello dear elastic users,

im currently having an issue implementing a logic which updates fields after they have been stored in the elastic.

Let me try to explain this to the best of my ability:

Theres a log entrance comming in to elastic which contains fields like

  • Theres a CONNECT Logline comming in. The system provides a connectionID
  • A couple of seconds or minutes later a BIND Logline comes in with the same connectionID in addition now we have filled user field.
    My goal is to add this user field to the connect log line...
    .... this is the basic task...

Now heres where things really get complicated.... it is possible that multiple same connection ids occour. The data enhanchement must only work for the last operation...

any ideas on how to do this?
best regards michael

2

How are you sending logs to elasticsearch ? with Logstash ?

3

They are comming from a TD agent logshipper

4

No idea about it, but anyway, elasticsearch provide an upsert option to update existing document

hat's something that logstash support OOB

5

Would the enrich processor be a viable option?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.