Conditional updating of existing record

Elastic Stack Logstash
1

Hi,
I am very new to elastic and would like to get an advice.
I have messages looking something like that:

THREAD_ID:101 MSG_TYPE:1 FINAL_STATE:EXPIRED
THREAD_ID:101 MSG_TYPE:2 FINAL_STATE:DELIVERED

They arrive in an uncontrolled manner:
-sometimes MSG_TYPE:1 comes first,
-sometimes MSG_TYPE:2 comes first,
-sometimes only one of them arrive,
-they never arrive from the same file.

I need to write such logstash config that only MSG_TYPE:1 info would be updated and I need to add additional field indicating which came first. For example if MSG_TYPE:2 came second it should look like that:
THREAD_ID:101 MSG_TYPE:2 FINAL_STATE:DELIVERED RECEIVED:1,2
And if it arrived first:
THREAD_ID:101 MSG_TYPE:2 FINAL_STATE:DELIVERED RECEIVED:2,1

The solution I'm looking for should work in real time. Could you please advise me on what filters or methods to use in this case?

2

I think you could do this with the "update" action on the Elasticsearch output. You would need to set the DOC_ID. I assume your thread_id field is unique?

3

Yes it is unique and I do know how to set DOC_ID. But with the update action message is updated by whichever comes last, isn't it? So in this case it works only if messages comes:
THREAD_ID:101 MSG_TYPE:1 FINAL_STATE:EXPIRED
THREAD_ID:101 MSG_TYPE:2 FINAL_STATE:DELIVERED
Then I do get what I initially want:
THREAD_ID:101 MSG_TYPE:2 FINAL_STATE:DELIVERED

But if MSG_TYPE:2 would come first and MSG_TYPE:1 later, my message would look like:
THREAD_ID:101 MSG_TYPE:1 FINAL_STATE:EXPIRED

And I always need to keep the MSG_TYPE:2 information.

4

Could you do a conditional output based on the message type field?

5

Sorry I am not sure what you mean. Could you write a rough example of how conditional output should look in this case?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.