Merge Multiline json into single line json using codec multiline plugin

Elastic Stack Logstash
1

What pattern I need to apply in input plugin to merge multiline json?

I am using below multiline json:

{
"end-time": "2018-07-18T07:43:01",
"start-time": "2018-07-17T15:36:39",
"status": "SUCCESS",
"steps": [
{
"dependency-list": [
"ConfigureCoreNodes"
],
"end-time": "2018-07-17T16:19:18",
"log-file": "/var/log/platform/eos-iaas-install/step-log-files/ConfigureCoreServices/step-ConfigureCoreServices-20180717161206.log",
"name": "ConfigureCoreServices",
"start-time": "2018-07-17T16:12:07",
"status": "SUCCESS",
"tasks": [
{
"dependency-list": [
"SetupNTP"
],
"end-time": "2018-07-17T16:14:27",
"log-file": "/var/log/platform/eos-iaas-install/step-log-files/ConfigureCoreServices/task-SetupCoreProxy-20180717161359.log",
"name": "SetupCoreProxy",
"retry": {
"attempts": 0,
"interval": 0
},
"start-time": "2018-07-17T16:13:59",
"status": "SUCCESS",
"time-consumed": "0:00:28"
},
{
"dependency-list": [
"LifecyclePrimordialActivation"
],
"end-time": "2018-07-18T05:35:45",
"log-file": "/var/log/platform/eos-iaas-install/step-log-files/LifecycleRealActivation/step-LifecycleRealActivation-20180718043722.log",
"name": "LifecycleRealActivation",
"start-time": "2018-07-18T04:37:22",
"status": "SUCCESS",
"tasks": [
{
"dependency-list": [],
"end-time": "2018-07-18T05:35:44",
"log-file": "/var/log/platform/eos-iaas-install/step-log-files/LifecycleRealActivation/task-Activate-20180718043722.log",
"name": "Activate",
"retry": {
"attempts": 0,
"interval": 0
},
"start-time": "2018-07-18T04:37:22",
"status": "SUCCESS",
"time-consumed": "0:58:22"
}
],
"time-consumed": "0:58:23"
}
],
"time-consumed": "16:06:22"
}

Below is the logstash input file:
input {
file {
type => "eos-iaas-output"
path => "/tmp/iaas-install.output.json"
codec => multiline {
pattern => "}$"
negate => "true"
what => "next"
max_lines => 3000
}
sincedb_path => "/dev/null"
start_position => "beginning"
}
}

2

If you want to read the entire file as a single event I would use this

input {
    file {
        path => "/tmp/iaas-install.output.json"
        codec => multiline { pattern => "^Spalanzani" negate => true what => "previous" auto_flush_interval => 2 }
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}

That is, use a pattern that never matches and then auto_flush after reading the whole file.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.