Handle multiple loglines as batch with condition

[zorkin82]

Hello together,
I'm looking for a solution for the following problem:

• I have a logfile tat looks something like this:

2017-03-24T13:18:07,147 INFO [topology-scan_Worker-1]: Executing [PersistTopologyToZipCommand]
2017-03-24T13:18:07,152 ERROR [topology-scan_Worker-1]: Error during command execution: Error during topology scan execution! (BasicCommandExecutorImpl)
java.lang.NullPointerException
at java.util.Hashtable.put(Unknown Source) ~[?:1.8.0_121]
2017-03-24T13:18:07,159 INFO [topology-scan_Worker-1]: Executing [2] fallback commands...
2017-03-24T13:18:07,159 INFO [topology-scan_Worker-1]: Executing [MovePersistedXmlTopologyFileCommand]
2017-03-24T13:18:07,159 INFO [topology-scan_Worker-1]: Moving topology file [topology-data-170324131807.tmp]...
2017-03-24T13:18:07,160 INFO [topology-scan_Worker-1]: Execution of command
2017-03-24T13:18:07,160 INFO [topology-scan_Worker-1]: Executing
2017-03-24T13:18:07,165 INFO [topology-scan_Worker-1]: Execution of command
2017-03-24T13:18:07,165 INFO [topology-scan_Worker-1]: Destroying SSHCommandExecutionContext...
2017-03-24T13:18:32,751 INFO [topology-scan_Worker-2]: Closing all SSH related artifacts if required...
2017-03-24T13:18:32,751 INFO [topology-scan_Worker-2]: Closing session output reader...
2017-03-24T13:18:32,751 INFO [topology-scan_Worker-2]: Closing ssh session...
2017-03-24T13:18:32,760 INFO [topology-scan_Worker-2]: Closing ssh client...
2017-03-24T13:18:32,761 INFO [topology-scan_Worker-2]: Finished topology scan execution
2017-03-24T13:18:57,210 INFO [topology-scan_Worker-3]: Starting topology scan execution...
2017-03-24T13:18:57,210 INFO [topology-scan_Worker-3]: Setting up SSHCommandExecutionContext...
2017-03-24T13:18:57,211 INFO [topology-scan_Worker-3]: Identifying object to scan...
2017-03-24T13:18:57,211 INFO [topology-scan_Worker-3]: Closing all SSH related artifacts if required...
2017-03-24T13:18:57,573 INFO [topology-scan_Worker-3]: Trying SSH connect: (
2017-03-24T13:18:58,454 INFO [topology-scan_Worker-3]: Authentication result: COMPLETE
2017-03-24T13:19:01,709 INFO [topology-scan_Worker-3]: Destroying SSHCommandExecutionContext...
2017-03-24T13:19:01,710 INFO [topology-scan_Worker-3]: Closing all SSH related artifacts if required...
2017-03-24T13:19:01,710 INFO [topology-scan_Worker-3]: Closing session output reader...
2017-03-24T13:19:01,710 INFO [topology-scan_Worker-3]: Closing ssh session...
2017-03-24T13:19:01,712 INFO [topology-scan_Worker-3]: Closing ssh client...
2017-03-24T13:19:01,712 INFO [topology-scan_Worker-3]: Finished topology scan execution

• the log ist grouped by "batches" that are identified by a worker ( [topology-scan_Worker-3] )
• if an error occurs in one worker process ("batch") I need to handle the complete log batch in a separate way
• the logs are stored in elasticsearch

So based on the given log example I want to do:

1. All Log lines with the identifier [topology-scan_Worker-3] should be send to a error index in elasticsearch because there was a ERROR inside the batch
2. All other lines should be send to another index in elasticsearch

Are there any suggestions how to solve this with logstash or if it is unpossible.

Best regards
Dirk

[guyboertje]

Logstash as it is now is not able to do this. We treat every event as if we have never seen any other events before - it is stateless.

In terms of your data - is the above section a complete log or part of a stream of events?
I can't see any relationship between topology-scan_Worker-1, topology-scan_Worker-2 and topology-scan_Worker-3 in the data. As a human I can but as a machine I could not.