Hi,
I am new to the Logstash.
I've below log that I want to read in from a file, it is basically from another logstash server. Below is just one row from the file, there are millions of such rows.
{"@timestamp":"2022-12-01T13:30:00.004Z","message":"<190>Dec 1 14:29:59 10.62.161.199 AA-AMG3U: 0950198238 NN [MDA 8/4]:
LN44 SA 2022 Dec 1 14:29:59:87 CET 17 4001 10.XX.133.XX 56560 401 91.235.10.25 15179 2400160261XXXXX_467000XXXX1_35292011220XXXXX_string1 | LN44 SD 2022 Dec 1 14:29:59:87 CET 17 4001 10.XX.133.XX 56560 401 91.235.10.25 15179 2400160261XXXXX_467000XXXX1_35292011220XXXXX_string2 | LN44 SA 2022 Dec 1 14:29:59:87 CET 17 4001 10.XX.133.XX 56560 401 91.235.10.25 15179 2400160261XXXXX_467679XXXX2_35292011220XXXXX_string1 \n","@version":"1","host":"100.62.161.XXX"}
- Read only the message part.
- Message has the header part "<190>Dec 1 14:29:59 10.62.161.199 AA-AMG3U: 0950198238 NN [MDA 8/4]: . I only need AA-AMG3U:
- Records inside the message are pipe separated '|'
- Merge two events if there is a match between the fields, the first and 2nd record are a match except the "SD" and "Timestamp". SD record has timestamp greater than SA.
- If there is no match like the 3rd record, should send it as it is.
Output should be as below with 2 records,
AA-AMG3U:, LN44, SA 2022 Dec 1 14:29:59:87 CET 17 4001 10.XX.133.XX 56560 401 91.235.10.25 15179 2400160261XXXXX_467000XXXX1_35292011220XXXXX_string1, Dec 1 14:30:59:87 CET 17
AA-AMG3U:, LN44 SA 2022 Dec 1 14:29:59:87 CET 17 4001 10.XX.133.XX 56560 401 91.235.10.25 15179 2400160261XXXXX_467679XXXX2_35292011220XXXXX_string1
Output record in first record has the timestamp appended from the second record from input.
Can something be done using logstash, at the moment we are using traditional database and that's taking longer to query and store.