# Not able to parse logs while parsing mix json objects

**URL:** <https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494>\
**Category:** Logstash\
**Created:** [January 19, 2023, 9:57am UTC](https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494 "2023-01-19T09:57:36Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![abhishek1111](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@abhishek1111](https://discuss.elastic.co/u/abhishek1111)\
**Post date:** [January 19, 2023, 9:57am UTC](https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494/1 "2023-01-19T09:57:36Z")

</div>

Could someone please help me with parsing below mix json logs tried multiple ways of parsing it, but nothing has helped.

Logs

{"log":"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:14:32.361997271Z"}  
{"log":"INFO: 2023/01/19 08:15:12 apikey\_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.02674995Z"}  
{"log":"DEBUG: 2023/01/19 08:15:12 authorise\_user\_action.go:54: AuthorizationHandler.AuthoriseWrite: TeamID :mule \n","stream":"stdout","time":"2023-01-19T08:15:12.027445654Z"}  
{"log":"INFO: 2023/01/19 08:15:12 apikey\_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.027456218Z"}  
{"log":"INFO: 2023/01/19 08:15:12 authorise\_user\_action.go:66: AuthorizationHandler.AuthoriseWrite: tokenAuth: \u0026{e21163f1-b2ae-4db7-8bef-0a3967944e3f kambagiriswamy.v} \n","stream":"stdout","time":"2023-01-19T08:15:12.028027946Z"}  
{"log":"DEBUG: 2023/01/19 08:15:12 authorise\_user\_action.go:152: AuthorizationHandler.policyEnforcer: Performing Action From kambagiriswamy.v for Team mule for Data mule\_data Action write\n","stream":"stdout","time":"2023-01-19T08:15:12.029332059Z"}  
{"log":"DEBUG: 2023/01/19 08:15:12 authorise\_user\_action.go:160: policyEnforcer: enforcer ok: true \n","stream":"stdout","time":"2023-01-19T08:15:12.029341937Z"}  
{"log":"INFO: 2023/01/19 08:15:12 url\_controller.go:155: URLController.CreateShortURL: Initiating call to create short URL\n","stream":"stdout","time":"2023-01-19T08:15:12.029344603Z"}  
{"log":"INFO: 2023/01/19 08:15:12 apikey\_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.029347866Z"}  
{"log":"INFO: 2023/01/19 08:15:12 apikey\_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.029895955Z"}  
{"log":"DEBUG: 2023/01/19 08:15:12 url\_controller.go:180: URLController.CreateShortURL: Creating Short URL with Input Data {LongURL:[IDFC FIRST Bank](https://qa-opt.idfcfirstbank.com/dl/51be397e-3c9b-4fb8-9ae5-876f624da472) Domain: ShortKey: Title: Description: Expiration:0 MetaData: Classifier: ExpirationPolicy: ExpirationType: ClicksAllowed:0 IsExpiring:false}\n","stream":"stdout","time":"2023-01-19T08:15:12.03044091Z"}  
{"log":"INFO: 2023/01/19 08:15:12 url\_service.go:57: URLService.CreateShortURL: Initiating call to create Short URL\n","stream":"stdout","time":"2023-01-19T08:15:12.03044949Z"}  
{"log":"INFO: 2023/01/19 08:15:12 links\_repo.go:98: models.CreateLink: Initiating call to create link\n","stream":"stdout","time":"2023-01-19T08:15:12.03103815Z"}  
{"log":"INFO: 2023/01/19 08:15:12 url\_service.go:117: URLService.CreateShortURL: Created Short URL for l url= [IDFC FIRST Bank](https://qa-opt.idfcfirstbank.com/dl/51be397e-3c9b-4fb8-9ae5-876f624da472) short key = [idfcfrtest.com/h4my1y](http://idfcfrtest.com/h4my1y)\n","stream":"stdout","time":"2023-01-19T08:15:12.031605237Z"}  
{"log":"INFO: 2023/01/19 08:15:12 analytics\_repo.go:64: models.CreateAnalytics: Initiating call to create Analytics\n","stream":"stdout","time":"2023-01-19T08:15:12.031609722Z"}  
{"log":"INFO: 2023/01/19 08:15:12 url\_service.go:129: URLService.CreateShortURL: Created Initial Analytics short key = [idfcfrtest.com/h4my1y](http://idfcfrtest.com/h4my1y)\n","stream":"stdout","time":"2023-01-19T08:15:12.032152652Z"}  
{"log":"{"level":"info","ts":"2023-01-19T08:15:12.032Z","logging-at":"middlewares/log.go:37","msg":"GIN-REQUEST-LOG","client-ip":"10.164.30.231","duration":5.55,"method":"POST","path":"/api/mule/createShortURL","status":200,"user\_id":"","referrer":"","user\_agent":"AHC/1.0","body\_size":511,"service":"api-microservice","request\_id":""}\n","stream":"stdout","time":"2023-01-19T08:15:12.032236067Z"}  
{"log":"[GIN] 2023/01/19 - 08:15:12 | 200 | 5.596469ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:15:12.032241259Z"}

---

<div class="post-metadata">

**Author:** ![abhishek1111](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@abhishek1111](https://discuss.elastic.co/u/abhishek1111)\
**Post date:** [January 19, 2023, 5:38pm UTC](https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494/2 "2023-01-19T17:38:16Z")

</div>

@here could someone please help me here.

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [January 20, 2023, 7:01am UTC](https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494/3 "2023-01-20T07:01:07Z")

</div>

Hi,

Welcome to the Elastic Forum 🎉

Please be patient and wait for an answer, you should not expect response times of less than 24 hours here.

Also, your post does not really contain much useful information:

- What ways of parsing did you try already?
- What problems occured on parsing?
- What do you want to achive?

And please try to format your code with 3 backticks like this:

```json
{"log":"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:14:32.361997271Z"}
{"log":"INFO: 2023/01/19 08:15:12 apikey_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.02674995Z"}

```

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [January 20, 2023, 11:50am UTC](https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494/4 "2023-01-20T11:50:56Z")

</div>

It looks as simply but not. I haven't split 3rd type, you can use dissect or csv

```auto
filter {

  grok {
    break_on_match => false
    match => {
        "message" => [ "^{\"log\":\"%{DATA:logmsg}\",\"stream\"",
        "\"stream\":\"%{DATA:stream}\",\"time\"",
        "\"time\":\"%{TIMESTAMP_ISO8601:time}\"}"]
        }
  }
    date {
      match => ["time", "ISO8601"]
      target=> "time" # optionally
    }

   if ( [logmsg] =~ /^(INFO|DEBUG|ERROR|WARNING)/ ) {
    grok {
     match => { "logmsg" => ["%{LOGLEVEL:loglevel}: %{DATELOG:logtime} %{GREEDYDATA:details}"] }
     pattern_definitions => { "DATELOG" => "%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY} %{TIME}" } 
     remove_field => ["logmsg"]
    }

   date {
      match => ["logtime", "YYYY/MM/dd hh:mm:ss"]
      timezone => "Europe/Berlin"
      target=> "logtime"
   }
  }
   else if ( [logmsg] =~ /^{/ ) {
     mutate{ gsub => ["logmsg","[\\n]",""] }
     json { source => "logmsg" }
     
	 date {
      match => ["ts", "ISO8601"]
      target=> "ts" # optionally
      remove_field => ["logmsg"]
    }
   }

    mutate { remove_field => ["log","event"] } # "message"

}

```

Result:

```auto
{
    "@timestamp" => 2023-01-20T11:44:10.938671200Z,
        "stream" => "stdout",
      "loglevel" => "DEBUG",
       "details" => "authorise_user_action.go:160: policyEnforcer: enforcer ok: true \\n",
          "time" => 2023-01-19T08:15:12.029Z,
       "message" => "{\"log\":\"DEBUG: 2023/01/19 08:15:12 authorise_user_action.go:160: policyEnforcer: enforcer ok: true \\n\",\"stream\":\"stdout\",\"time\":\"2023-01-19T08:15:12.029341937Z\"}\r",
       "logtime" => 2023-01-19T07:15:12.000Z
}
{
           "msg" => "GIN-REQUEST-LOG",
     "loggig-at" => "middlewares/log.go:37",
      "cliet-ip" => "10.164.30.231",
        "method" => "POST",
         "level" => "ifo",
       "message" => "{\"log\":\"{\"level\":\"info\",\"ts\":\"2023-01-19T08:15:12.032Z\",\"logging-at\":\"middlewares/log.go:37\",\"msg\":\"GIN-REQUEST-LOG\",\"client-ip\":\"10.164.30.231\",\"duration\":5.55,\"method\":\"POST\",\"path\":\"/api/mule/createShortURL\",\"status\":200,\"user_id\":\"\",\"referrer\":\"\",\"user_agent\":\"AHC/1.0\",\"body_size\":511,\"service\":\"api-microservice\",\"request_id\":\"\"}\\n\",\"stream\":\"stdout\",\"time\":\"2023-01-19T08:15:12.032236067Z\"}\r",
       "duratio" => 5.55,
          "path" => "/api/mule/createShortURL",
     "body_size" => 511,
      "referrer" => "",
    "@timestamp" => 2023-01-20T11:44:10.939668100Z,
        "stream" => "stdout",
       "user_id" => "",
       "service" => "api-microservice",
          "time" => 2023-01-19T08:15:12.032Z,
    "request_id" => "",
     "user_aget" => "AHC/1.0",
            "ts" => 2023-01-19T08:15:12.032Z,
        "status" => 200
}
{
    "@timestamp" => 2023-01-20T11:44:10.937675300Z,
        "logmsg" => "[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST \"/api/mule/createShortURL\"\\n",
        "stream" => "stdout",
          "time" => 2023-01-19T08:14:32.361Z,
       "message" => "{\"log\":\"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST \"/api/mule/createShortURL\"\\n\",\"stream\":\"stdout\",\"time\":\"2023-01-19T08:14:32.361997271Z\"}\r"
}

```

---

<div class="post-metadata">

**Author:** ![abhishek1111](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@abhishek1111](https://discuss.elastic.co/u/abhishek1111)\
**Post date:** [January 23, 2023, 9:27am UTC](https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494/5 "2023-01-23T09:27:13Z")

</div>

Hello Wolfram,

Thanks for your reply.

My requirement: as you can see there are 3 different types of log patterns are there in single log file

Pattern 1:  
{"log":"[GIN] 2023/01/19 - 08:14:32 | 200 | 5.595393ms | 10.164.30.231 | POST "/api/mule/createShortURL"\n","stream":"stdout","time":"2023-01-19T08:14:32.361997271Z"}

Pattern 2:  
{"log":"INFO: 2023/01/19 08:15:12 apikey\_repo.go:46: models.IsValidAPIKey: Initiating call to validate API Key\n","stream":"stdout","time":"2023-01-19T08:15:12.02674995Z"}

Pattern 3:  
{"log":"{"level":"info","ts":"2023-01-19T14:17:36.672Z","logging-at":"middlewares/log.go:37","msg":"GIN-REQUEST-LOG","client-ip":"10.164.30.231","duration":5.47,"method":"POST","path":"/api/mule/createShortURL","status":200,"user\_id":"","referrer":"","user\_agent":"AHC/1.0","body\_size":516,"service":"api-microservice","request\_id":""}\n","stream":"stdout","time":"2023-01-19T14:17:36.672945579Z"}

Problem Statement:

I need to parse all logs and store each log field as separate field.

I tried below parsing logic 🙂

# Sample Logstash configuration for creating a simple

# Beats -\> Logstash -\> Elasticsearch pipeline.

input {  
beats {  
port =\> 5044  
}  
}

filter {  
grok {  
match =\> { "message" =\> ""log":"%{LOGLEVEL:loglevel}: %{GREEDYDATA:Message}stream":"%{DATA:stream}","time":"%{TIMESTAMP\_ISO8601:time}"" }  
}  
grok {  
match =\> { "message" =\> ""log":"{\"level\":\"%{LOGLEVEL:loglevel}\",\"ts\":\"%{TIMESTAMP\_ISO8601:timestamp}\",\"logging-at\":\"%{DATA:logging\_at}\",\"msg\":\"%{DATA:Message}\",\"client-ip\":\"%{IP:client\_ip}\",\"duration\":%{NUMBER:duration},\"method\":\"%{DATA:method}\",\"path\":\"%{PATH:path}\",\"status\":%{NUMBER:status},\"user\_id\":\"%{DATA:user\_id}\",\"referrer\":\"%{DATA:referrer}\",\"user\_agent\":\"%{DATA:user\_agent}\",\"body\_size\":%{NUMBER:body\_size},\"service\":\"%{DATA:service}\",\"request\_id\":\"%{DATA:request\_id}\"}\n","stream":"%{DATA:stream}","time":"%{TIMESTAMP\_ISO8601:my\_time}"" }  
}  
mutate {  
remove\_field =\> ["message"]  
remove\_field =\> ["tags"]  
}  
}

output {  
stdout {codec =\> rubydebug}  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
index =\> "urlshort777-%{+YYYY.MM.dd}"  
}  
}

But with above pattern only one pattern i.e pattern 3 gets ingested inside index and other are getting rejected, could you guys please help me with this use case where i have mix json logs.

---

<div class="post-metadata">

**Author:** ![abhishek1111](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@abhishek1111](https://discuss.elastic.co/u/abhishek1111)\
**Post date:** [January 23, 2023, 9:35am UTC](https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494/6 "2023-01-23T09:35:10Z")

</div>

Thanks @Rios for the reply but i am still facing issue while parsing all 3 log pattern.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [January 23, 2023, 10:02am UTC](https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494/7 "2023-01-23T10:02:18Z")

</div>

The grok pattern which has been provided supports all three log types.

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [January 23, 2023, 11:26am UTC](https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494/8 "2023-01-23T11:26:37Z")

</div>

As your data is JSON I would use the [json](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json.html) Codec first. This way, the third pattern does not need any grok parsing (just renaming fields) and the grok patterns for one and two are simpler to write and understand.

Also, do you have access on the applications writing them or are those three logs read from different Filebeat installations? In this case it would be a great help to add a field or a tag to differentiate between the logs.

---

<div class="post-metadata">

**Author:** ![abhishek1111](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@abhishek1111](https://discuss.elastic.co/u/abhishek1111)\
**Post date:** [January 24, 2023, 8:17am UTC](https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494/9 "2023-01-24T08:17:08Z")

</div>

Hello @Wolfram_Haussig i have no control over application logs hence will not be able to add tags to differentiate between logs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2023, 8:17am UTC](https://discuss.elastic.co/t/not-able-to-parse-logs-while-parsing-mix-json-objects/323494/10 "2023-02-21T08:17:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.