Exceptions chronology over the linked log records


(Dmitry Reshetnik) #1

Hi,
There is a system with a lot of components. Each component has 1+ instances. And each instances writes each log file.

Service1 (service1.[PID1].log)
Service2 (service2.[PID2].log)
Service3 (service3.[PID3].log)
Service4 (service4.[PID4].log)

There is a "main" service (for example Service1), which knows about all related instances. All log formats are the same.
In "main" service logs there are records which related services PID.
So I a set of files with PID in the name and I have some records with related PID in fields:

{
"pid": "1",
"name": "Service1",
"timestamp": "....",
"referenced_pid": "2"
}

{
"pid": "1",
"name": "Service1",
"timestamp": "....",
"referenced_pid": "3"
}

{
"pid": "1",
"name": "Service1",
"timestamp": "....",
"referenced_pid": "4"
}

{
"pid": "2",
"name": "Service2",
"exception": ": ",
"timestamp": "...."
}

{
"pid": "3",
"name": "Service3",
"exception": ": ",
"timestamp": "...."
}

{
"pid": "4",
"name": "Service4",
"timestamp": "...."
}

What I want to do:
I'd like to select all records from "Service1" and JOIN with all log records on "referenced_pid" = "pid".
And query some information from these logs.
For example I'd like to build chronology of exceptions from logs:

  1. take all logs from the system (it's 1 join described above)
  2. select all exception and timestamp pairs
  3. create some kind of linear diagram to be able to watch chronology and explore messages

Is that possible to do that?


(Mark Walkom) #2

You cannot join with KB because ES does not natively support it.


(system) #3