Hi , what im trying to do is pretty simple and straight forward but cant figure out why its not working. When I was using logstash with file input it parsed my logs correctly meaning that it separated my log file by \n into messages which would then be parsed by my pattern. However when I actually tried to implement it into my project ( using http input , text/plain ) logstash couldnt parse this as multiple lines and simply bundled it into one message with each line ending in \n.
Example bellow:
2020-12-16 15:43:31.605 INFO 18020 --- [http-nio-8080-exec-3] c.n.w.workflow.service.DataService : Getting groups of task 5fda1d109ceec746643760f8 in case 11.11.2020 13:20 level: 0\n2020-12-16 15:43:31.611 INFO 18020 --- [http-nio-8080-exec-3] c.n.w.workflow.service.DataService : [5fda1d109ceec746643760f5]: Getting data of task 12 [5fda1d109ceec746643760f8]\n2020-12-16 15:43:32.891 DEBUG 18020 --- [http-nio-8080-exec-7] c.n.w.c.l.ControllerRequestLoggingFilter : uri=/api/task/assign/5fda1d109ceec746643760f8;headers=[host:\"localhost:8080\", connection:\"keep-alive\", accept:\"application/json, text/plain, */*\", x-auth-token:\"6a9cb70d-e0e6-4adf-aa7e-e06f41a4ee08\", accept-language:\"sk-SK\", user-agent:\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36\", origin:\"http://localhost:4200\", sec-fetch-site:\"same-site\", sec-fetch-mode:\"cors\", sec-fetch-dest:\"empty\", referer:\"http://localhost:4200/\", accept-encoding:\"gzip, deflate, br\"]\n2020-12-16 15:43:32.908 INFO 18020 --- [http-nio-8080-exec-7] c.n.w.workflow.service.DataService : [5fda1d109ceec746643760f5]: Running actions of transition 12\n2020-12-16 15:43:32.911 INFO 18020 --- [http-nio-8080-exec-7] c.n.w.workflow.service.TaskService : [5fda1d109ceec746643760f5]: Task [GENERATE] in case [11.11.2020 13:20] evaluating rules of event ASSIGN of phase PRE\n2020-12-16 15:43:32.920 INFO 18020 --- [http-nio-8080-exec-7] c.n.w.workflow.service.TaskService : [5fda1d109ceec746643760f5]: Assigning task [GENERATE] to user [super@netgrif.com]\n2020-12-16 15:43:32.920 INFO 18020 --- [http-nio-8080-exec-7] c.n.w.workflow.service.TaskService : [5fda1d109ceec746643760f5]: Start execution of GENERATE in case 11.11.2020 13:20\n2020-12-16 15:43:32.936 INFO 18020 --- [http-nio-8080-exec-7] c.n.w.workflow.service.TaskService : [5fda1d109ceec746643760f5]: Reloading tasks in [11.11.2020 13:20]\n2020-12-16 15:43:33.009 INFO 18020 --- [http-nio-8080-exec-7] c.n.w.workflow.service.DataService : [5fda1d109ceec746643760f5]: Running actions of transition 12\n2020-12-16 15:43:33.012 INFO 18020 --- [http-nio-8080-exec-7] c.n.w.workflow.service.TaskService : [5fda1d109ceec746643760f5]: Task [GENERATE] in case [11.11.2020 13:20] evaluating rules of event ASSIGN of phase POST\n2020-12-16 15:43:33.035 INFO 18020 --- [http-nio-8080-exec-7] c.n.w.workflow.service.TaskService : [5fda1d109ceec746643760f5]: Task [GENERATE] in case [11.11.2020 13:20] assigned to [super@netgrif.com]\n2020-12-16 15:43:33.060 INFO 18020 --- [pool-2-thread-1] c.n.w.e.service.ElasticTaskService : [?]: Task \"5fda1d109ceec746643760fa\" deleted\n2020-12-16 15:43:34.346 INFO 18020 --- [http-nio-8080-exec-1] c.n.w.workflow.service.TaskService : [5fda1d109ceec746643760f5]: Task [GENERATE] in case [11.11.2020 13:20] assigned to [super@netgrif.com] was finished\n
I wondered if I could fix this by adding split function so I did before using the grok filter as follows:
filter {
split{
field => "message"
terminator => "
"
}
grok {
patterns_dir => ["./patterns"]
match => { "message" => "%{LOG_BASE} %{LOG_MESSAGE1_1}" }
}
mutate {
remove_field => ["host","path","@version","@timestamp"]
}
}
I tried putting "\n" into terminator or also as suggested in different topic literal new line "
".
However with no luck . How can I split this message by \n and then parse each line separately ? Thank for the input 