elasticsearch + kibana + logstash + filebeat latest 6.4.1 to collect and analyze web logs. The columns of my log are like:
timestamp, http_method, request_uri, http_status, host, user_agent, client_ip, client_port
I have configured ELK to show my logs in Kibana. But now I want to see my logs in sessions. I hope the log lines can be grouped by session and shown in Kibana's
Discover page. In my scenario, the log lines with the same
(host, client_ip) belong to the same session.
I hope to have this:
- Session table
name, client_ip, host session1, www.google.com, 126.96.36.199 session2, www.bing.com, 188.8.131.52 session3, www.google.com, 184.108.40.206
When I click one of the above session (e.g., session1), I can see all the records of that session in the following 2nd table:
- Log table
timestamp, http_method, request_uri, http_status, host, user_agent, client_ip, client_port November 5th 2018, 21:33:17.773, POST, /index.html, 200, www.google.com, chrome 59, 220.127.116.11, 1234 November 5th 2018, 21:33:18.773, POST, /abc.html, 200, www.google.com, chrome 59, 18.104.22.168, 1234 November 5th 2018, 21:33:19.773, POST, /index.html, 404, www.google.com, chrome 59, 22.214.171.124, 5678
I know Elasticsearch does flat indexing, it's not easy to have hierarchy between documents. I'm OK to create separated indices for the above two tables. I know Dashboard can show two Discover tables at the same time. But my question is:
How to link these two tables? When I click one item in the Session table, the Log table will show corresponding contents?
Or is there any other way to fulfill my requirement (view session-based logs easily in Kibana)? Thanks.