Hello, for these set of input lines:
SAMPLE INPUT
INFO | jvm 1 | 2017/08/28 17:54:14 | 25306.601: Total time for which application threads were stopped: 0.0008632 seconds, Stopping threads took: 0.0000905 seconds
INFO | jvm 1 | 2017/08/28 17:54:17 | Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.141-b15 mixed mode):
INFO | jvm 1 | 2017/08/28 17:54:17 |
INFO | jvm 1 | 2017/08/28 17:54:17 | "something-1" something
INFO | jvm 1 | 2017/08/28 17:54:17 | java.lang.Thread.State: TIMED_WAITING (on object monitor)
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | - locked something
INFO | jvm 1 | 2017/08/28 17:54:17 |
INFO | jvm 1 | 2017/08/28 17:54:17 | Exception in thread "something-2" something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | Caused by: something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | ... 12 more
INFO | jvm 1 | 2017/08/28 17:54:17 | Caused by: something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | at something
INFO | jvm 1 | 2017/08/28 17:54:17 | ... 15 more
INFO | jvm 1 | 2017/08/28 17:54:17 |
INFO | jvm 1 | 2017/06/20 19:58:39 | Heap
INFO | jvm 1 | 2017/06/20 19:58:39 | PSYoungGen total 2540032K, used 935016K [0x0000000759080000, 0x00000007fff80000, 0x0000000800000000)
INFO | jvm 1 | 2017/06/20 19:58:39 | eden space 2342400K, 36% used [0x0000000759080000,0x000000078d9adef8,0x00000007e8000000)
INFO | jvm 1 | 2017/06/20 19:58:39 | from space 197632K, 37% used [0x00000007f3e80000,0x00000007f866c470,0x00000007fff80000)
INFO | jvm 1 | 2017/06/20 19:58:39 | to space 195072K, 0% used [0x00000007e8000000,0x00000007e8000000,0x00000007f3e80000)
INFO | jvm 1 | 2017/06/20 19:58:39 | ParOldGen total 5470208K, used 1553638K [0x000000060b200000, 0x0000000759000000, 0x0000000759080000)
INFO | jvm 1 | 2017/06/20 19:58:39 | object space 5470208K, 28% used [0x000000060b200000,0x0000000669f399e8,0x0000000759000000)
INFO | jvm 1 | 2017/06/20 19:58:39 | PSPermGen total 230400K, used 230349K [0x00000005eb200000, 0x00000005f9300000, 0x000000060b200000)
INFO | jvm 1 | 2017/06/20 19:58:39 | object space 230400K, 99% used [0x00000005eb200000,0x00000005f92f3650,0x00000005f9300000)
INFO | jvm 1 | 2017/06/20 19:58:39 |
INFO | jvm 1 | 2017/06/20 19:58:47 | 135695.454: Total time for which application threads were stopped: 0.0026100 seconds, Stopping threads took: 0.0001660 seconds
INFO | jvm 1 | 2017/06/20 19:58:52 | Returning empty Certificate from getAcceptedIssuers
From this, I need following blocks as output:
- 25306.601: Total time .... (and similar single lines starting with a number)
- Full thread dump Java....
- the whole something-1 block
- the whole something-2 block, including 'Caused by' blocks
- Heap block
- Returning empty Certificate..... (and similar single lines)
Currently, I'm using this in my input:
input {
file {
type => "sometype"
path => "somepath"
start_position => "beginning"
#ignore_older => 0
sincedb_path => "NUL"
#Forces to re-parse
codec => multiline {
pattern => "(?m)(%{DATA}|%{DATA}|%{DATA}| (( )|(\t))+)|(%{DATA}|%{DATA}|%{DATA}| Caused)"
what => "previous"
auto_flush_interval => 2
}
}
}
The output is then fed to elasticsearch.
My issue is the processing rate is very slow. It is like 4-5 docs as output, per minute.
I usually get 2000-2500 docs/s as output, for some other type of log files.
I thought that maybe multiline codec is the one that is causing such drastic performance drop.
Please help.
Thank you.