Filebeats--Multiline pattern not working for Java GC output log, Need help

Elastic Stack Beats
1

Hello,

I have the below logfile which outputs from jvm GC verbose. Need to get individual GC's into different json messages. Like all GC(0) into 1 message dn all GC(1) into other message etc and send to logstash.

Trying below multiline, not working, pls let me know

> - type: log
>   enabled: true
>   paths:
>     - "/opt/hc/*.log"
>   multiline.pattern: 'gc-start(.)*?'
>   multiline.negate: true
>   multiline.match: after

Below is the output log file:

[2024-02-14T03:28:36.882+0000][118][safepoint   ] Safepoint "Cleanup", Time since last: 1000937099 ns, Reaching safepoint: 7926 ns, Cleanup: 16591 ns, At safepoint: 5842 ns, Total: 30359 ns
[2024-02-14T03:28:37.883+0000][118][safepoint   ] Safepoint "Cleanup", Time since last: 1000983154 ns, Reaching safepoint: 7901 ns, Cleanup: 15673 ns, At safepoint: 5759 ns, Total: 29333 ns
[2024-02-14T03:28:37.979+0000][118][gc,start    ] GC(0) Pause Young (Normal) (G1 Evacuation Pause)
[2024-02-14T03:28:37.980+0000][118][gc,task     ] GC(0) Using 4 workers of 4 for evacuation
[2024-02-14T03:28:37.980+0000][118][gc,age      ] GC(0) Desired survivor size 4194304 bytes, new threshold 15 (max threshold 15)
[2024-02-14T03:28:37.989+0000][118][gc,age      ] GC(0) Age table with threshold 15 (max threshold 15)
[2024-02-14T03:28:37.989+0000][118][gc,age      ] GC(0) - age   1:    6334648 bytes,    6334648 total
[2024-02-14T03:28:37.989+0000][118][gc,phases   ] GC(0)   Pre Evacuate Collection Set: 0.3ms
[2024-02-14T03:28:37.989+0000][118][gc,phases   ] GC(0)   Merge Heap Roots: 0.2ms
[2024-02-14T03:28:37.989+0000][118][gc,phases   ] GC(0)   Evacuate Collection Set: 7.6ms
[2024-02-14T03:28:37.989+0000][118][gc,phases   ] GC(0)   Post Evacuate Collection Set: 0.8ms
[2024-02-14T03:28:37.989+0000][118][gc,phases   ] GC(0)   Other: 0.9ms
[2024-02-14T03:28:37.989+0000][118][gc,heap     ] GC(0) Eden regions: 25->0(21)
[2024-02-14T03:28:37.989+0000][118][gc,heap     ] GC(0) Survivor regions: 0->4(4)
[2024-02-14T03:28:37.989+0000][118][gc,heap     ] GC(0) Old regions: 0->0
[2024-02-14T03:28:37.989+0000][118][gc,heap     ] GC(0) Archive regions: 2->2
[2024-02-14T03:28:37.989+0000][118][gc,heap     ] GC(0) Humongous regions: 0->0
[2024-02-14T03:28:37.989+0000][118][gc,metaspace] GC(0) Metaspace: 12164K(12480K)->12164K(12480K) NonClass: 10907K(11072K)->10907K(11072K) Class: 1257K(1408K)->1257K(1408K)
[2024-02-14T03:28:37.989+0000][118][gc          ] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 51M->8M(1028M) 9.877ms
[2024-02-14T03:28:37.989+0000][118][gc,cpu      ] GC(0) User=0.03s Sys=0.01s Real=0.01s
[2024-02-14T03:28:37.989+0000][118][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 95968199 ns, Reaching safepoint: 6448 ns, Cleanup: 5160 ns, At safepoint: 10047516 ns, Total: 10059124 ns
[2024-02-14T03:28:38.466+0000][118][gc,start    ] GC(1) Pause Young (Normal) (G1 Evacuation Pause)
[2024-02-14T03:28:38.466+0000][118][gc,task     ] GC(1) Using 4 workers of 4 for evacuation
[2024-02-14T03:28:38.466+0000][118][gc,age      ] GC(1) Desired survivor size 4194304 bytes, new threshold 1 (max threshold 15)
[2024-02-14T03:28:38.480+0000][118][gc,age      ] GC(1) Age table with threshold 1 (max threshold 15)
[2024-02-14T03:28:38.480+0000][118][gc,age      ] GC(1) - age   1:    7549776 bytes,    7549776 total
[2024-02-14T03:28:38.480+0000][118][gc,phases   ] GC(1)   Pre Evacuate Collection Set: 0.3ms
[2024-02-14T03:28:38.480+0000][118][gc,phases   ] GC(1)   Merge Heap Roots: 0.1ms
[2024-02-14T03:28:38.480+0000][118][gc,phases   ] GC(1)   Evacuate Collection Set: 12.7ms
[2024-02-14T03:28:38.480+0000][118][gc,phases   ] GC(1)   Post Evacuate Collection Set: 0.9ms
[2024-02-14T03:28:38.480+0000][118][gc,phases   ] GC(1)   Other: 0.3ms
[2024-02-14T03:28:38.480+0000][118][gc,heap     ] GC(1) Eden regions: 21->0(21)
[2024-02-14T03:28:38.480+0000][118][gc,heap     ] GC(1) Survivor regions: 4->4(4)
[2024-02-14T03:28:38.480+0000][118][gc,heap     ] GC(1) Old regions: 0->4
[2024-02-14T03:28:38.480+0000][118][gc,heap     ] GC(1) Archive regions: 2->2
[2024-02-14T03:28:38.480+0000][118][gc,heap     ] GC(1) Humongous regions: 2->0
[2024-02-14T03:28:38.480+0000][118][gc,metaspace] GC(1) Metaspace: 12554K(12800K)->12554K(12800K) NonClass: 11270K(11392K)->11270K(11392K) Class: 1283K(1408K)->1283K(1408K)
[2024-02-14T03:28:38.480+0000][118][gc          ] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 54M->16M(1028M) 14.460ms
[2024-02-14T03:28:38.480+0000][118][gc,cpu      ] GC(1) User=0.04s Sys=0.01s Real=0.01s
[2024-02-14T03:28:38.480+0000][118][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 476473693 ns, Reaching safepoint: 6561 ns, Cleanup: 35412 ns, At safepoint: 14587635 ns, Total: 14629608 ns

Thanks

2

This doesn't seem to match anything, in your log you have gc,start not gc-start, have you tried to change it?

3

Yep, thanks it worked now. Its silly mis..