In Stackdriver Log Viewer, all log-lines within a give HTTP request (in our Google AppEngine app) appear together in a single section. This is very useful in tracking behavior inside a single HTTP request.
How do we do this in Kibana?
Do these log messages have anything in common? e.g., is there a request identifier included as part of the message?
At a previous employer that heavily leveraged micro-services, we included an HTTP Middleware in each of our applications that would find an "activity id" header (or generate a UUID if not present), apply it as metadata to the current request, and ensure any downstream http requests included the header. Our logging layer would also ensure that the activity id for each request was a part of any structured logging messages, which were then consumed by Logstash using a JSON Filter Plugin. This worked really well, since in Kibana, all we had to do was filter down on that single field to get the entire chain of requests.
is there a request identifier
Yes
At a previous employer...
Looks good. How did you visually bundle a set of lines?
In other words, it's essential that lines from a single request/activity follow each other sequentially rather than being mixed into other requests. And this sequence of lines should be visually distinguished, perhaps as a foldable section.
The visual representation is a bit out of scope for Logstash, which is a tool for data pipelines. I don't personally have an answer in this regard (we used Kibana tooling, but I do not currently know what the state-of-the-art is there).
Once the data is in Elasticsearch, a terms aggregation on the activity id (ordered by minimum timestamp, descending), and within that a top-hits aggregation (ordered by timestamp), could get you the results in the order you mentioned.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.