Logstash- How to parse formatted JSON arrays in log files

Elastic Stack Logstash
1

Logstash version: 7.17.10
Elasticsearch version:7.17.10

The logs are located in /var/logs directory and the format is as below:

xxx.log

[
  {
    "t": "SYS",
    "dt": "2023-04-17 19:46:40.147 GMT-04:00",
    "c": "MenuSectionsViewController",
    "cpu": "cpu usage: 40%",
    "m": "dealloc",
    "msg": "",
    "mem": "total: 4177M used: 169M free: 2398M",
    "b": "8.25.1"
  },
  {
    "t": "SYS",
    "dt": "2023-04-17 19:46:40.154 GMT-04:00",
    "c": "OrderTicketView",
    "cpu": "cpu usage: 40%",
    "m": "dealloc",
    "msg": "",
    "mem": "total: 4177M used: 169M free: 2398M",
    "b": "8.25.1"
  }
]

This is my logstash.conf:

input {
  file {
    path => "/var/logs/*.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "^\["
      negate => true
      what => "previous"
    } 
  }
}

filter{
    json { source => message }
}


output {
  stdout {
    codec => rubydebug
  } 
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "device-logs-%{+YYYY.MM.dd}"
  }
}

I will get a JSON parse error based on the previous config. details:

Error parsing json {:source=>"message", :raw=>"[\n  {\n    \"t\": \"SYS\",\n    \"dt\": \"2023-04-17 19:46:40.147 GMT-04:00\",\n    \"c\": \"MenuSectionsViewController\",\n    \"cpu\": \"cpu usage: 40%\",\n    \"m\": \"dealloc\",\n    \"msg\": \"\",\n    \"mem\": \"total: 4177M used: 169M free: 2398M\",\n    \"b\": \"8.25.1\"\n  },\n  {\n    \"t\": \"SYS\",\n    \"dt\": \"2023-04-17 19:46:40.154 GMT-04:00\",\n    \"c\": \"OrderTicketView\",\n    \"cpu\": \"cpu usage: 40%\",\n    \"m\": \"dealloc\",\n    \"msg\": \"\",\n    \"mem\": \"total: 4177M used: 169M free: 2398M\",\n    \"b\": \"8.25.1\"\n  },\n  {\n    \"t\": \"SYS\",\n    \"dt\": \"2023-04-17 19:46:40.155 GMT-04:00\",\n    \"c\": \"TicketSideBackground\",\n    \"cpu\": \"cpu usage: 40%\",\n    \"m\": \"dealloc\",\n    \"msg\": \"\",\n    \"mem\": \"total: 4177M used: 169M free: 2398M\",\n    \"b\": \"8.25.1\"\n  }", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Array (start marker at [Source: (byte[])"[

As you know, I want to use Logstash to parse JSON arrays in log files and output each JSON object as a separate document to Elasticsearch. My expected log documents are as follows:

doc_1

{
  "_index": "device-logs-2023.07.28",
  "_type": "_doc",
  "_id": "VVqImokBM0y425kVF-bp",
  "_version": 1,
  "_score": 1,
  "_source": {
    "path": "/var/logs/2.log",
    "tags": [
      "multiline",
      "_jsonparsefailure"
    ],
    "@timestamp": "2023-07-28T03:25:57.330Z",
    "host": "2b9233260805",
    "@version": "1",
    "message": {
          "t":"SYS",
          "dt": "2023-04-17 19:46:40.147 GMT-04:00",
           "c": "MenuSectionsViewController"
            ....
       }

  }
...

But unfortunately, I have tried multiple configurations and still haven't found the right solution, so I hope everyone can help me out.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.