Hi all, i am trying to parse certain log lines and fetch only required information. I am using if else statements to differentiate according to which my lines should parse. But all log line are parsing using first grok pattern and it is not even entering the statements.
sample Log lines:
INFO 2019-08-28 21:49:57,953 [Operator App Mixins] Request URL: /api/v1/operatorapp/societySystem/manualGateEvent/
INFO 2019-08-28 21:49:57,953 [Operator App Mixins] Request Query Params: <QueryDict: {}>
INFO 2019-08-28 21:49:57,954 [Operator App Mixins] Request Data: {u'gate_number': 2, u'action_type': u'open_gate', u'created': u'28-08-2019 09:49 PM'}
INFO 2019-08-28 21:49:57,958 [Operator App Mixins] Logged In operator- 3374 - M2K Aura Exit, URL: /api/v1/operatorapp/societySystem/manualGateEvent/
INFO 2019-08-28 21:49:57,959 [Operator App Mixins] Event Gate: M2K Aura, Gate-2, Exit
INFO 2019-08-28 21:49:57,967 [Operator App Mixins] URL: /api/v1/operatorapp/societySystem/manualGateEvent/, Operator: 3374 - M2K Aura Exit, Resp Status: 200
INFO 2019-08-28 21:49:57,967 [Operator App Mixins] URL: /api/v1/operatorapp/societySystem/manualGateEvent/, User: 3374 - M2K Aura Exit, Resp Time: 0.024612903595
INFO 2019-08-28 21:49:58,127 [User App Mixins] Request URL: /api/v1/userapp/booking/list/, Method: POST
INFO 2019-08-28 21:49:58,128 [User App Mixins] Request Query Params: <QueryDict: {}>
INFO 2019-08-28 21:49:58,128 [User App Mixins] Request Data: {u'last_booking_id': 0}
INFO 2019-08-28 21:49:58,128 [User App Mixins] Logged In User- 23545 - ANUJ SETHI, Android User, URL: /api/v1/userapp/booking/list/
INFO 2019-08-28 21:49:58,129 [RBS User Views] Fetch Parking Booking List
INFO 2019-08-28 21:49:58,129 [RBS User Views] Request data: {u'last_booking_id': 0}
INFO 2019-08-28 21:49:58,132 [RBS User Views] Sending Bookings List: 0
INFO 2019-08-28 21:49:58,134 [User App Mixins] URL: /api/v1/userapp/booking/list/, User: 23545 - ANUJ SETHI, Resp Status: 200
INFO 2019-08-28 21:49:58,134 [User App Mixins] URL: /api/v1/userapp/booking/list/, User: 23545 - ANUJ SETHI, Resp Time: 0.045
INFO 2019-08-28 21:49:58,617 [Cloud Action Views] Fetch Cloud Action
INFO 2019-08-28 21:49:58,618 [Cloud Action Views] Request Data: {u'device_id': 150}
INFO 2019-08-28 21:49:58,618 [Cloud Action Views] Fetch cloud action V1
INFO 2019-08-28 21:49:58,618 [Cloud Action Views] User: 6382 - Local Server
My filter:
filter {
grok {
match => { "message" => "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:timestamp} \[%{DATA:class}\] %{GREEDYDATA:extra}"
}
}
if [class] =~ "User App Mixins" {
grok {
match => { "message" => "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:timestamp} \[%{DATA:class}\] URL: %{URIPATHPARAM:url},
User: %{NUMBER:user_id} - %{GREEDYDATA:username}, (Resp Status: %{NUMBER:status}|Resp Time: %{NUMBER:duration})"
overwrite => ["message"]
}
}
if "_grokparsefailure" in [tags] {
grok {
match => {"message" => "%{GREEDYDATA:message}"}
}
}
}
elseif [class] =~ "Operator App Mixins"{
grok {
match => { "message" => "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:timestamp} \[%{DATA:class}\] URL: %{URIPATHPARAM:url},
(Operator: %{NUMBER:operator}|User: %{NUMBER:User}) - %{GREEDYDATA:gateName}, (Resp Status: %{NUMBER:status}|Resp Time: %{NUMBER:duration})"
overwrite => ["message"]
}
}
if "_grokparsefailure" in [tags] {
grok {
match => {"message" => "%{GREEDYDATA:message}"}
}
}
}
elseif [class] =~ "RBS Operator Views"{
grok {
match => {
"message" => "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:timestamp} \[%{DATA:class}\] %{GREEDYDATA:junk}"
overwrite => ["message"]
}
}
}
elseif [class] =~ "Society System Views"{
grok {
match => {
"message" => "%{GREEDYDATA:junk}"
remove_field => "message"
overwrite => ["message"]
}
}
}
elseif [class] =~ "Society RFID Logic" {
grok {
match => {
"message" => "%{GREEDYDATA:junk}"
remove_field => "message"
overwrite => "message"
}
}
}
My output:
{
"path" => "/home/rajdeep/Desktop/lg5",
"host" => "rajdeep-ThinkPad-T460s",
"message" => [
[0] "INFO 2019-08-29 00:44:00,206 [Operator App Mixins] URL: /api/v1/operatorapp/societySystem/mqttDisconnected/, Operator: 10290 - Palm Grove Exit, Resp Status: 200",
[1] "INFO 2019-08-29 00:44:00,206 [Operator App Mixins] URL: /api/v1/operatorapp/societySystem/mqttDisconnected/, Operator: 10290 - Palm Grove Exit, Resp Status: 200"
],
"extra" => "URL: /api/v1/operatorapp/societySystem/mqttDisconnected/, Operator: 10290 - Palm Grove Exit, Resp Status: 200",
"tags" => [
[0] "_grokparsefailure"
],
"@version" => "1",
"timestamp" => "2019-08-29 00:44:00,206",
"class" => "Operator App Mixins",
"@timestamp" => 2019-09-09T07:30:52.175Z,
"severity" => "INFO"
}
{
"path" => "/home/rajdeep/Desktop/lg5",
"host" => "rajdeep-ThinkPad-T460s",
"message" => [
[0] "INFO 2019-08-29 00:44:00,206 [Operator App Mixins] URL: /api/v1/operatorapp/societySystem/mqttDisconnected/, User: 10290 - Palm Grove Exit, Resp Time: 0.01766705513",
[1] "INFO 2019-08-29 00:44:00,206 [Operator App Mixins] URL: /api/v1/operatorapp/societySystem/mqttDisconnected/, User: 10290 - Palm Grove Exit, Resp Time: 0.01766705513"
],
"extra" => "URL: /api/v1/operatorapp/societySystem/mqttDisconnected/, User: 10290 - Palm Grove Exit, Resp Time: 0.01766705513",
"tags" => [
[0] "_grokparsefailure"
],
"@version" => "1",
"timestamp" => "2019-08-29 00:44:00,206",
"class" => "Operator App Mixins",
"@timestamp" => 2019-09-09T07:30:52.175Z,
"severity" => "INFO"
}