Not able to parse logs while parsing mix json objects

Elastic Stack Logstash
1

Could someone please help me with parsing below mix json logs tried multiple ways of parsing it, but nothing has helped.

Logs

{"log":"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:14:32.361997271Z"}
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.02674995Z"}
{"log":"DEBUG: 2023/01/19 08:15:12 authorise_user_action.go:54: AuthorizationHandler.AuthoriseWrite: TeamID :mule \n","stream":"stdout","time":"2023-01-19T08:15:12.027445654Z"}
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.027456218Z"}
{"log":"INFO: 2023/01/19 08:15:12 authorise_user_action.go:66: AuthorizationHandler.AuthoriseWrite: tokenAuth: \u0026{e21163f1-b2ae-4db7-8bef-0a3967944e3f kambagiriswamy.v} \n","stream":"stdout","time":"2023-01-19T08:15:12.028027946Z"}
{"log":"DEBUG: 2023/01/19 08:15:12 authorise_user_action.go:152: AuthorizationHandler.policyEnforcer: Performing Action From kambagiriswamy.v for Team mule for Data mule_data Action write\n","stream":"stdout","time":"2023-01-19T08:15:12.029332059Z"}
{"log":"DEBUG: 2023/01/19 08:15:12 authorise_user_action.go:160: policyEnforcer: enforcer ok: true \n","stream":"stdout","time":"2023-01-19T08:15:12.029341937Z"}
{"log":"INFO: 2023/01/19 08:15:12 url_controller.go:155: URLController.CreateShortURL: Initiating call to create short URL\n","stream":"stdout","time":"2023-01-19T08:15:12.029344603Z"}
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.029347866Z"}
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.029895955Z"}
{"log":"DEBUG: 2023/01/19 08:15:12 url_controller.go:180: URLController.CreateShortURL: Creating Short URL with Input Data {LongURL:IDFC FIRST Bank Domain: ShortKey: Title: Description: Expiration:0 MetaData: Classifier: ExpirationPolicy: ExpirationType: ClicksAllowed:0 IsExpiring:false}\n","stream":"stdout","time":"2023-01-19T08:15:12.03044091Z"}
{"log":"INFO: 2023/01/19 08:15:12 url_service.go:57: URLService.CreateShortURL: Initiating call to create Short URL\n","stream":"stdout","time":"2023-01-19T08:15:12.03044949Z"}
{"log":"INFO: 2023/01/19 08:15:12 links_repo.go:98: models.CreateLink: Initiating call to create link\n","stream":"stdout","time":"2023-01-19T08:15:12.03103815Z"}
{"log":"INFO: 2023/01/19 08:15:12 url_service.go:117: URLService.CreateShortURL: Created Short URL for l url= IDFC FIRST Bank short key = idfcfrtest.com/h4my1y\n","stream":"stdout","time":"2023-01-19T08:15:12.031605237Z"}
{"log":"INFO: 2023/01/19 08:15:12 analytics_repo.go:64: models.CreateAnalytics: Initiating call to create Analytics\n","stream":"stdout","time":"2023-01-19T08:15:12.031609722Z"}
{"log":"INFO: 2023/01/19 08:15:12 url_service.go:129: URLService.CreateShortURL: Created Initial Analytics short key = idfcfrtest.com/h4my1y\n","stream":"stdout","time":"2023-01-19T08:15:12.032152652Z"}
{"log":"{"level":"info","ts":"2023-01-19T08:15:12.032Z","logging-at":"middlewares/log.go:37","msg":"GIN-REQUEST-LOG","client-ip":"10.164.30.231","duration":5.55,"method":"POST","path":"/api/mule/createShortURL","status":200,"user_id":"","referrer":"","user_agent":"AHC/1.0","body_size":511,"service":"api-microservice","request_id":""}\n","stream":"stdout","time":"2023-01-19T08:15:12.032236067Z"}
{"log":"[GIN] 2023/01/19 - 08:15:12 | 200 | 5.596469ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:15:12.032241259Z"}

2

@here could someone please help me here.

3

Hi,

Welcome to the Elastic Forum :tada:

Please be patient and wait for an answer, you should not expect response times of less than 24 hours here.

Also, your post does not really contain much useful information:

  • What ways of parsing did you try already?
  • What problems occured on parsing?
  • What do you want to achive?

And please try to format your code with 3 backticks like this:

{"log":"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:14:32.361997271Z"}
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.02674995Z"}

Best regards
Wolfram

4

It looks as simply but not. I haven't split 3rd type, you can use dissect or csv

filter {

  grok {
    break_on_match => false
    match => {
        "message" => [ "^{\"log\":\"%{DATA:logmsg}\",\"stream\"",
        "\"stream\":\"%{DATA:stream}\",\"time\"",
        "\"time\":\"%{TIMESTAMP_ISO8601:time}\"}"]
        }
  }
    date {
      match => [ "time", "ISO8601"]
      target=> "time" # optionally
    }

   if ( [logmsg] =~ /^(INFO|DEBUG|ERROR|WARNING)/ ) {
    grok {
     match => { "logmsg" => [ "%{LOGLEVEL:loglevel}: %{DATELOG:logtime} %{GREEDYDATA:details}"] }
     pattern_definitions => { "DATELOG" => "%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY} %{TIME}" } 
     remove_field  => [ "logmsg"]
    }

   date {
      match => ["logtime", "YYYY/MM/dd hh:mm:ss"]
      timezone => "Europe/Berlin"
      target=> "logtime"
   }
  }
   else if ( [logmsg] =~ /^{/ ) {
     mutate{ gsub => ["logmsg","[\\n]",""] }
     json { source => "logmsg" }
     
	 date {
      match => [ "ts", "ISO8601"]
      target=> "ts" # optionally
      remove_field  => [ "logmsg"]
    }
   }

    mutate {   remove_field  => [ "log","event" ] } #  "message"

}

Result:

{
    "@timestamp" => 2023-01-20T11:44:10.938671200Z,
        "stream" => "stdout",
      "loglevel" => "DEBUG",
       "details" => "authorise_user_action.go:160: policyEnforcer: enforcer ok: true \\n",
          "time" => 2023-01-19T08:15:12.029Z,
       "message" => "{\"log\":\"DEBUG: 2023/01/19 08:15:12 authorise_user_action.go:160: policyEnforcer: enforcer ok: true \\n\",\"stream\":\"stdout\",\"time\":\"2023-01-19T08:15:12.029341937Z\"}\r",
       "logtime" => 2023-01-19T07:15:12.000Z
}
{
           "msg" => "GIN-REQUEST-LOG",
     "loggig-at" => "middlewares/log.go:37",
      "cliet-ip" => "10.164.30.231",
        "method" => "POST",
         "level" => "ifo",
       "message" => "{\"log\":\"{\"level\":\"info\",\"ts\":\"2023-01-19T08:15:12.032Z\",\"logging-at\":\"middlewares/log.go:37\",\"msg\":\"GIN-REQUEST-LOG\",\"client-ip\":\"10.164.30.231\",\"duration\":5.55,\"method\":\"POST\",\"path\":\"/api/mule/createShortURL\",\"status\":200,\"user_id\":\"\",\"referrer\":\"\",\"user_agent\":\"AHC/1.0\",\"body_size\":511,\"service\":\"api-microservice\",\"request_id\":\"\"}\\n\",\"stream\":\"stdout\",\"time\":\"2023-01-19T08:15:12.032236067Z\"}\r",
       "duratio" => 5.55,
          "path" => "/api/mule/createShortURL",
     "body_size" => 511,
      "referrer" => "",
    "@timestamp" => 2023-01-20T11:44:10.939668100Z,
        "stream" => "stdout",
       "user_id" => "",
       "service" => "api-microservice",
          "time" => 2023-01-19T08:15:12.032Z,
    "request_id" => "",
     "user_aget" => "AHC/1.0",
            "ts" => 2023-01-19T08:15:12.032Z,
        "status" => 200
}
{
    "@timestamp" => 2023-01-20T11:44:10.937675300Z,
        "logmsg" => "[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST \"/api/mule/createShortURL\"\\n",
        "stream" => "stdout",
          "time" => 2023-01-19T08:14:32.361Z,
       "message" => "{\"log\":\"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST \"/api/mule/createShortURL\"\\n\",\"stream\":\"stdout\",\"time\":\"2023-01-19T08:14:32.361997271Z\"}\r"
}
5

Hello Wolfram,

Thanks for your reply.

My requirement: as you can see there are 3 different types of log patterns are there in single log file

Pattern 1:
{"log":"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:14:32.361997271Z"}

Pattern 2:
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.02674995Z"}

Pattern 3:
{"log":"{"level":"info","ts":"2023-01-19T14:17:36.672Z","logging-at":"middlewares/log.go:37","msg":"GIN-REQUEST-LOG","client-ip":"10.164.30.231","duration":5.47,"method":"POST","path":"/api/mule/createShortURL","status":200,"user_id":"","referrer":"","user_agent":"AHC/1.0","body_size":516,"service":"api-microservice","request_id":""}\n","stream":"stdout","time":"2023-01-19T14:17:36.672945579Z"}

Problem Statement:

I need to parse all logs and store each log field as separate field.

I tried below parsing logic :slight_smile:

Sample Logstash configuration for creating a simple

Beats -> Logstash -> Elasticsearch pipeline.

input {
beats {
port => 5044
}
}

filter {
grok {
match => { "message" => ""log":"%{LOGLEVEL:loglevel}: %{GREEDYDATA:Message}stream":"%{DATA:stream}","time":"%{TIMESTAMP_ISO8601:time}"" }
}
grok {
match => { "message" => ""log":"{\"level\":\"%{LOGLEVEL:loglevel}\",\"ts\":\"%{TIMESTAMP_ISO8601:timestamp}\",\"logging-at\":\"%{DATA:logging_at}\",\"msg\":\"%{DATA:Message}\",\"client-ip\":\"%{IP:client_ip}\",\"duration\":%{NUMBER:duration},\"method\":\"%{DATA:method}\",\"path\":\"%{PATH:path}\",\"status\":%{NUMBER:status},\"user_id\":\"%{DATA:user_id}\",\"referrer\":\"%{DATA:referrer}\",\"user_agent\":\"%{DATA:user_agent}\",\"body_size\":%{NUMBER:body_size},\"service\":\"%{DATA:service}\",\"request_id\":\"%{DATA:request_id}\"}\n","stream":"%{DATA:stream}","time":"%{TIMESTAMP_ISO8601:my_time}"" }
}
mutate {
remove_field => ["message"]
remove_field => ["tags"]
}
}

output {
stdout {codec => rubydebug}
elasticsearch {
hosts => ["http://localhost:9200"]
index => "urlshort777-%{+YYYY.MM.dd}"
}
}

But with above pattern only one pattern i.e pattern 3 gets ingested inside index and other are getting rejected, could you guys please help me with this use case where i have mix json logs.

6

Thanks @Rios for the reply but i am still facing issue while parsing all 3 log pattern.

7

The grok pattern which has been provided supports all three log types.

8

As your data is JSON I would use the json Codec first. This way, the third pattern does not need any grok parsing (just renaming fields) and the grok patterns for one and two are simpler to write and understand.

Also, do you have access on the applications writing them or are those three logs read from different Filebeat installations? In this case it would be a great help to add a field or a tag to differentiate between the logs.

9

Hello @Wolfram_Haussig i have no control over application logs hence will not be able to add tags to differentiate between logs.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.