Omair
(Muhammad Omair Siddiqui)
November 18, 2020, 9:31pm
1
Hi.. I'm new to elastic stack and trying to learn using filters to structure my data but unfortunately not getting much help.
Below is the application log
2020-11-19T00:08:34.627+0500 INFO 141799 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605726514602,"formattedTime":"2020-11-19T00:08:34.602+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"975d295ddd3228f50e7f4f2af4211a05","serviceName":"Test Logging","serviceUri":"/testlogging","totalFrontendLatency":23,"totalBackendLatency":23,"isPolicySuccessful":true,"isPolicyViolation":false,"isRoutingFailure":false}
I want to see this log in the below format
"time" : 1518734278251,
"formattedTime" : "2018-02-15T14:37:58.251-08:00",
"nodeId" : "48b5a4bc60fd4db191ddd5258eab0a35",
"nodeName" : "Gateway1",
"nodeIp" : "10.242.45.121",
"serviceId" : "13df5d4767ea633273940598aa2f323c",
"serviceName" : "Test Service 1",
"serviceUri" : "/test1",
"totalFrontendLatency" : 234,
"totalBackendLatency" : 123,
"isPolicySuccessful": true,
"isPolicyViolation" : false,
"isRoutingFailure" : false
I want someone to help me writing the logstash.conf file to achieve this.
Any help would be appreciated.
warkolm
(Mark Walkom)
November 18, 2020, 9:51pm
2
Can you share what you have so far?
Please also format your code/logs/config using the </>
button, or markdown style back ticks. It helps to make things easy to read which helps us help you
Omair
(Muhammad Omair Siddiqui)
November 19, 2020, 11:05am
3
Actual Log Entry
2020-11-19T00:08:34.627+0500 INFO 141799 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605726514602,"formattedTime":"2020-11-19T00:08:34.602+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"975d295ddd3228f50e7f4f2af4211a05","serviceName":"Test Logging","serviceUri":"/testlogging","totalFrontendLatency":23,"totalBackendLatency":23,"isPolicySuccessful":true,"isPolicyViolation":false,"isRoutingFailure":false}
The way i want it
"time" : 1518734278251,
"formattedTime" : "2018-02-15T14:37:58.251-08:00",
"nodeId" : "48b5a4bc60fd4db191ddd5258eab0a35",
"nodeName" : "Gateway1",
"nodeIp" : "10.242.45.121",
"serviceId" : "13df5d4767ea633273940598aa2f323c",
"serviceName" : "Test Service 1",
"serviceUri" : "/test1",
"totalFrontendLatency" : 234,
"totalBackendLatency" : 123,
"isPolicySuccessful": true,
"isPolicyViolation" : false,
"isRoutingFailure" : false
My logstash.conf file
input {
beats {
port => "5044"
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => "http://<elaseticsearch-IP>:9200"
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
stdout{}
}
when im running with this conf file im getting _jsonparsefailure tag in my logs.
ylasri
(Yassine LASRI)
November 19, 2020, 11:17am
4
Your message is not fully jsn, you need to use grok filter first then you access to the json payload of your message
grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{NUMBER:process_id} %{JAVACLASS:logclass}: %{INT:int_value}: metrics: %{GREEDYDATA:json_payload}"}}
json { source => "json_payload" }
I would recommand to use Kibana Grok debugger to quickly test your pattern
Omair
(Muhammad Omair Siddiqui)
November 19, 2020, 1:57pm
5
Thanks for your help Yassine. Much appreciated!
There are some extra spaces in the log between INFO and process_id by default.. When I used your pattern i was unable to get the response in the debugger but then i manually removed the spaces and then the pattern worked. How I can remove these extra spaces using a filter? Below is my updated conf file and the output im getting. still getting _jsonparsefailure tag
> input {
> beats {
> port => "5044"
> }
> }
>
> filter {
> grok {
> match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{NUMBER:process_id} %{JAVACLASS:logclass}: %{INT:int_value}: metrics: %{GREEDYDATA:json_payload}" }
> }
> json {
> source => "json_payload"
> }
> }
>
> output {
> elasticsearch {
> hosts => "http://elasticsearch:9200"
> index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
> }
>
> stdout{}
> }
output im getting
"message" : "2020-11-19T14:37:21.525+0500 INFO 146061 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605778641452,"formattedTime":"2020-11-19T14:37:21.452+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"ce9fee8e5a20855a272555ab9c37a964","serviceName":"assertionMessageCode_7203","serviceUri":"/audit_7203","totalFrontendLatency":70,"totalBackendLatency":69,"isPolicySuccessful":false,"isPolicyViolation":true,"isRoutingFailure":false}",
"tags" : [
"APIGW",
"beats_input_codec_plain_applied",
"_jsonparsefailure"
],
ylasri
(Yassine LASRI)
November 19, 2020, 2:09pm
6
\s* is the pattern to much any number of spaces
%{TIMESTAMP_ISO8601:timestamp}\s*%{LOGLEVEL:log_level}\s*%{NUMBER:process_id}\s*%{JAVACLASS:logclass}:\s*%{INT:int_value}:\s*metrics:\s*%{GREEDYDATA:json_payload}
system
(system)
Closed
December 17, 2020, 2:09pm
7
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.