Not able to parse logs while parsing mix json objects

abhishek1111 · January 19, 2023, 9:57am

Could someone please help me with parsing below mix json logs tried multiple ways of parsing it, but nothing has helped.

Logs

{"log":"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:14:32.361997271Z"}
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.02674995Z"}
{"log":"DEBUG: 2023/01/19 08:15:12 authorise_user_action.go:54: AuthorizationHandler.AuthoriseWrite: TeamID :mule \n","stream":"stdout","time":"2023-01-19T08:15:12.027445654Z"}
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.027456218Z"}
{"log":"INFO: 2023/01/19 08:15:12 authorise_user_action.go:66: AuthorizationHandler.AuthoriseWrite: tokenAuth: \u0026{e21163f1-b2ae-4db7-8bef-0a3967944e3f kambagiriswamy.v} \n","stream":"stdout","time":"2023-01-19T08:15:12.028027946Z"}
{"log":"DEBUG: 2023/01/19 08:15:12 authorise_user_action.go:152: AuthorizationHandler.policyEnforcer: Performing Action From kambagiriswamy.v for Team mule for Data mule_data Action write\n","stream":"stdout","time":"2023-01-19T08:15:12.029332059Z"}
{"log":"DEBUG: 2023/01/19 08:15:12 authorise_user_action.go:160: policyEnforcer: enforcer ok: true \n","stream":"stdout","time":"2023-01-19T08:15:12.029341937Z"}
{"log":"INFO: 2023/01/19 08:15:12 url_controller.go:155: URLController.CreateShortURL: Initiating call to create short URL\n","stream":"stdout","time":"2023-01-19T08:15:12.029344603Z"}
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.029347866Z"}
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.029895955Z"}
{"log":"DEBUG: 2023/01/19 08:15:12 url_controller.go:180: URLController.CreateShortURL: Creating Short URL with Input Data {LongURL:IDFC FIRST Bank Domain: ShortKey: Title: Description: Expiration:0 MetaData: Classifier: ExpirationPolicy: ExpirationType: ClicksAllowed:0 IsExpiring:false}\n","stream":"stdout","time":"2023-01-19T08:15:12.03044091Z"}
{"log":"INFO: 2023/01/19 08:15:12 url_service.go:57: URLService.CreateShortURL: Initiating call to create Short URL\n","stream":"stdout","time":"2023-01-19T08:15:12.03044949Z"}
{"log":"INFO: 2023/01/19 08:15:12 links_repo.go:98: models.CreateLink: Initiating call to create link\n","stream":"stdout","time":"2023-01-19T08:15:12.03103815Z"}
{"log":"INFO: 2023/01/19 08:15:12 url_service.go:117: URLService.CreateShortURL: Created Short URL for l url= IDFC FIRST Bank short key = idfcfrtest.com/h4my1y\n","stream":"stdout","time":"2023-01-19T08:15:12.031605237Z"}
{"log":"INFO: 2023/01/19 08:15:12 analytics_repo.go:64: models.CreateAnalytics: Initiating call to create Analytics\n","stream":"stdout","time":"2023-01-19T08:15:12.031609722Z"}
{"log":"INFO: 2023/01/19 08:15:12 url_service.go:129: URLService.CreateShortURL: Created Initial Analytics short key = idfcfrtest.com/h4my1y\n","stream":"stdout","time":"2023-01-19T08:15:12.032152652Z"}
{"log":"{"level":"info","ts":"2023-01-19T08:15:12.032Z","logging-at":"middlewares/log.go:37","msg":"GIN-REQUEST-LOG","client-ip":"10.164.30.231","duration":5.55,"method":"POST","path":"/api/mule/createShortURL","status":200,"user_id":"","referrer":"","user_agent":"AHC/1.0","body_size":511,"service":"api-microservice","request_id":""}\n","stream":"stdout","time":"2023-01-19T08:15:12.032236067Z"}
{"log":"[GIN] 2023/01/19 - 08:15:12 | 200 | 5.596469ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:15:12.032241259Z"}

abhishek1111 · January 19, 2023, 5:38pm

@here could someone please help me here.

Wolfram_Haussig · January 20, 2023, 7:01am

Hi,

Welcome to the Elastic Forum

Please be patient and wait for an answer, you should not expect response times of less than 24 hours here.

Also, your post does not really contain much useful information:

What ways of parsing did you try already?
What problems occured on parsing?
What do you want to achive?

And please try to format your code with 3 backticks like this:

{"log":"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:14:32.361997271Z"}
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.02674995Z"}

Best regards
Wolfram

Rios · January 20, 2023, 11:50am

It looks as simply but not. I haven't split 3rd type, you can use dissect or csv

filter {

  grok {
    break_on_match => false
    match => {
        "message" => [ "^{\"log\":\"%{DATA:logmsg}\",\"stream\"",
        "\"stream\":\"%{DATA:stream}\",\"time\"",
        "\"time\":\"%{TIMESTAMP_ISO8601:time}\"}"]
        }
  }
    date {
      match => [ "time", "ISO8601"]
      target=> "time" # optionally
    }

   if ( [logmsg] =~ /^(INFO|DEBUG|ERROR|WARNING)/ ) {
    grok {
     match => { "logmsg" => [ "%{LOGLEVEL:loglevel}: %{DATELOG:logtime} %{GREEDYDATA:details}"] }
     pattern_definitions => { "DATELOG" => "%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY} %{TIME}" } 
     remove_field  => [ "logmsg"]
    }

   date {
      match => ["logtime", "YYYY/MM/dd hh:mm:ss"]
      timezone => "Europe/Berlin"
      target=> "logtime"
   }
  }
   else if ( [logmsg] =~ /^{/ ) {
     mutate{ gsub => ["logmsg","[\\n]",""] }
     json { source => "logmsg" }
     
	 date {
      match => [ "ts", "ISO8601"]
      target=> "ts" # optionally
      remove_field  => [ "logmsg"]
    }
   }

    mutate {   remove_field  => [ "log","event" ] } #  "message"

}

Result:

{
    "@timestamp" => 2023-01-20T11:44:10.938671200Z,
        "stream" => "stdout",
      "loglevel" => "DEBUG",
       "details" => "authorise_user_action.go:160: policyEnforcer: enforcer ok: true \\n",
          "time" => 2023-01-19T08:15:12.029Z,
       "message" => "{\"log\":\"DEBUG: 2023/01/19 08:15:12 authorise_user_action.go:160: policyEnforcer: enforcer ok: true \\n\",\"stream\":\"stdout\",\"time\":\"2023-01-19T08:15:12.029341937Z\"}\r",
       "logtime" => 2023-01-19T07:15:12.000Z
}
{
           "msg" => "GIN-REQUEST-LOG",
     "loggig-at" => "middlewares/log.go:37",
      "cliet-ip" => "10.164.30.231",
        "method" => "POST",
         "level" => "ifo",
       "message" => "{\"log\":\"{\"level\":\"info\",\"ts\":\"2023-01-19T08:15:12.032Z\",\"logging-at\":\"middlewares/log.go:37\",\"msg\":\"GIN-REQUEST-LOG\",\"client-ip\":\"10.164.30.231\",\"duration\":5.55,\"method\":\"POST\",\"path\":\"/api/mule/createShortURL\",\"status\":200,\"user_id\":\"\",\"referrer\":\"\",\"user_agent\":\"AHC/1.0\",\"body_size\":511,\"service\":\"api-microservice\",\"request_id\":\"\"}\\n\",\"stream\":\"stdout\",\"time\":\"2023-01-19T08:15:12.032236067Z\"}\r",
       "duratio" => 5.55,
          "path" => "/api/mule/createShortURL",
     "body_size" => 511,
      "referrer" => "",
    "@timestamp" => 2023-01-20T11:44:10.939668100Z,
        "stream" => "stdout",
       "user_id" => "",
       "service" => "api-microservice",
          "time" => 2023-01-19T08:15:12.032Z,
    "request_id" => "",
     "user_aget" => "AHC/1.0",
            "ts" => 2023-01-19T08:15:12.032Z,
        "status" => 200
}
{
    "@timestamp" => 2023-01-20T11:44:10.937675300Z,
        "logmsg" => "[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST \"/api/mule/createShortURL\"\\n",
        "stream" => "stdout",
          "time" => 2023-01-19T08:14:32.361Z,
       "message" => "{\"log\":\"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST \"/api/mule/createShortURL\"\\n\",\"stream\":\"stdout\",\"time\":\"2023-01-19T08:14:32.361997271Z\"}\r"
}

abhishek1111 · January 23, 2023, 9:27am

Hello Wolfram,

Thanks for your reply.

My requirement: as you can see there are 3 different types of log patterns are there in single log file

Pattern 1:
{"log":"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:14:32.361997271Z"}

Pattern 2:
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.02674995Z"}

Pattern 3:
{"log":"{"level":"info","ts":"2023-01-19T14:17:36.672Z","logging-at":"middlewares/log.go:37","msg":"GIN-REQUEST-LOG","client-ip":"10.164.30.231","duration":5.47,"method":"POST","path":"/api/mule/createShortURL","status":200,"user_id":"","referrer":"","user_agent":"AHC/1.0","body_size":516,"service":"api-microservice","request_id":""}\n","stream":"stdout","time":"2023-01-19T14:17:36.672945579Z"}

Problem Statement:

I need to parse all logs and store each log field as separate field.

I tried below parsing logic

Sample Logstash configuration for creating a simple

Beats -> Logstash -> Elasticsearch pipeline.

input {
beats {
port => 5044
}
}

filter {
grok {
match => { "message" => ""log":"%{LOGLEVEL:loglevel}: %{GREEDYDATA:Message}stream":"%{DATA:stream}","time":"%{TIMESTAMP_ISO8601:time}"" }
}
grok {
match => { "message" => ""log":"{\"level\":\"%{LOGLEVEL:loglevel}\",\"ts\":\"%{TIMESTAMP_ISO8601:timestamp}\",\"logging-at\":\"%{DATA:logging_at}\",\"msg\":\"%{DATA:Message}\",\"client-ip\":\"%{IP:client_ip}\",\"duration\":%{NUMBER:duration},\"method\":\"%{DATA:method}\",\"path\":\"%{PATH:path}\",\"status\":%{NUMBER:status},\"user_id\":\"%{DATA:user_id}\",\"referrer\":\"%{DATA:referrer}\",\"user_agent\":\"%{DATA:user_agent}\",\"body_size\":%{NUMBER:body_size},\"service\":\"%{DATA:service}\",\"request_id\":\"%{DATA:request_id}\"}\n","stream":"%{DATA:stream}","time":"%{TIMESTAMP_ISO8601:my_time}"" }
}
mutate {
remove_field => ["message"]
remove_field => ["tags"]
}
}

output {
stdout {codec => rubydebug}
elasticsearch {
hosts => ["http://localhost:9200"]
index => "urlshort777-%{+YYYY.MM.dd}"
}
}

But with above pattern only one pattern i.e pattern 3 gets ingested inside index and other are getting rejected, could you guys please help me with this use case where i have mix json logs.

abhishek1111 · January 23, 2023, 9:35am

Thanks @Rios for the reply but i am still facing issue while parsing all 3 log pattern.

Rios · January 23, 2023, 10:02am

The grok pattern which has been provided supports all three log types.

Wolfram_Haussig · January 23, 2023, 11:26am

As your data is JSON I would use the json Codec first. This way, the third pattern does not need any grok parsing (just renaming fields) and the grok patterns for one and two are simpler to write and understand.

Also, do you have access on the applications writing them or are those three logs read from different Filebeat installations? In this case it would be a great help to add a field or a tag to differentiate between the logs.

abhishek1111 · January 24, 2023, 8:17am

Hello @Wolfram_Haussig i have no control over application logs hence will not be able to add tags to differentiate between logs.

system · February 21, 2023, 8:17am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Parse Json Logs using logstash Logstash	2	275	March 25, 2021
How to parse mix json logs Logstash	28	4459	March 25, 2019
Json array parsing Logstash	1	390	July 6, 2017
Unable to parse nested json using logstash json filter Logstash	6	1187	October 16, 2019
Unable to parse json log to Kibana Logstash	5	772	November 26, 2019

Not able to parse logs while parsing mix json objects

Sample Logstash configuration for creating a simple

Beats -> Logstash -> Elasticsearch pipeline.

Related topics