i have a log file as below. Fields are indexed as [thread_id] [API] [method] [timestamp] [GREEDYDATA] respectively.
describing more on first two lines ,
1st line --> 1111 is the thread_id for api=/API1 with init time (method=GET) as Feb 11 22:27:13 PST 2020 and
2nd line ---> Response to first line (method=RESP) with response timestamp as Wed Feb 11 22:27:15 PST 2020
my requirement is to calculate time diff (GET timestamp - RESP timestamp) for an API with unique thread id. Can someone please guide me on this.
1111 /API1 GET Wed Feb 11 22:27:13 PST 2020 XYZ
1111 /API1 RESP Wed Feb 11 22:27:15 PST 2020 status=200
1112 /API2 GET Wed Feb 11 22:28:13 PST 2020 XYZ
1112 /API2 RESP Wed Feb 11 22:28:15 PST 2020 status=200
1113 /API3 GET Wed Feb 11 22:29:13 PST 2020 XYZ
1113 /API3 RESP Wed Feb 11 22:30:15 PST 2020 status=200
in your case(and the original problem as well), start and end time was filtered and ingested in one go(and in one line). But in my case, these logs will be ingested live (app server logs), so at a given time, logstash will have access to only start time (API init call) and may be after 2-3 seconds it will have end time in another line. this make it very complex to calculate difference before ingesting to ES.
please correct me if i got it wrong. and if you have any solution which can work on already ingested data.