DSL Query with related log messages

[max_knopp (maximilien)]

I al quite new in ELK and i need some help to create a quite complexe (at least it is my point of view) Query DSL filter in ELK . I imported with logstash the following log.

22-01-2019 12:14:05:493 | p5u05KTuVn | Client1 | requête depuis: 192.168.10.23 | URL demandée: http://api.myserver.local/api/user/get/pattern/naming/Poids_Berlingot/10cm
22-01-2019 12:14:05:500 | t680tLvgpx | Client2 | requête depuis: 192.168.10.23 | URL demandée: http://api.myserver.local/api/user/get/pattern/naming/Margo/38
22-01-2019 12:14:05:493 | p5u05KTuVn | Client1 | token authentifié | user_id: 29
22-01-2019 12:14:05:519 | p5u05KTuVn | Client1 | naming.json du modèle acheté envoyé | user_id : 29
22-01-2019 12:14:05:530 | t680tLvgpx | Client2 | token authentifié | user_id: 10
22-01-2019 12:14:05:559 | t680tLvgpx | Client2 | naming.json du modèle acheté envoyé | user_id : 10
...
22-01-2019 12:17:30:029 | FTuxiUB469 | Client1 | requête depuis: 80.12.63.20 | URL demandée: http://api.coutureconnectee.com/m2m/set/pacoevent/demo
22-01-2019 12:17:30:029 | FTuxiUB469 | Client1 | Réception d'un évènement PACO | Evènement: passage en mode démo
22-01-2019 12:17:33:018 | pd68v14hPm | Client2 | requête depuis: 80.12.63.20 | URL demandée: http://api.coutureconnectee.com/m2m/set/pacoevent/demo
22-01-2019 12:17:33:018 | pd68v14hPm | Client2 | Réception d'un évènement PACO | Evènement: passage en mode démo
22-01-2019 12:18:09:423 | tpc69IUT1n | Client1 | requête depuis: 192.168.10.23 | URL demandée: http://api.myserver.local/api/user/get/pattern/naming/Margo/38
22-01-2019 12:18:09:423 | tpc69IUT1n | Client1 | impossible de décoder le token | erreur: jwt expired

All fields are separated by the "|" char, all field are extracted to individual tag with the logstash filter (field name are the column number (col02, col03...) The timestamp at the begining of each line is also correctly parsed. When a client (col03) do a request to the server, a transaction ID (col02) is created to identify all messages generated when the server is processing the request.

First I need to identify and count a sequence of messages:

1. a message containing "get/pattern/naming"
2. getting the transaction ID (col02) of that message and the Client id (col03)
3. for the transaction ID extracted checking if there is a message containing "token authentifié" and a message containing "naming.json du modèle acheté envoyé"
4. if the point 3) is verified with the client Id extracted searching the first next message containing "set/pacoevent/demo"

in a second time (if possible) I need to get the time difference between the first message of the sequence and the last one.

if someone can help me to create that filter it would be great

[Marius_Dragomir]

You can create a filter from the filter builder in Kibana Discover for each of the first 3 requirements and then you can see them in Discover. But that's about it. The rest isn't really possible in Kibana and would require a custom app to do.