Logstash fail to rewrite json files

Hi,
I want to send json files with filebeat to logstash and then logstash rewrite these json files locally.

Filebeat version: 8.1.2
Logstash version: 8.1.2

There is my filebeat conf:

filebeat.inputs:

- type: filestream

  paths:
    - /path/to/json/*

output.logstash:
  hosts: ["192.168.1.15:5044"]

There is my logstash conf:

input {
  beats {
    port => 5044
  }
}

filter {
  ruby {
    code => 'event.set("filename", event.get("[log][file][path]").split("/").last)'
  }
}

output {
  file {
    path => "/path/to/write/json/%{filename}"
    codec => line { format => "%{message}" }
  }
}

For example i send this json file to logstash with filebeat:

{"widget": {
    "debug": "on",
    "window": {
        "title": "Sample Konfabulator Widget",
        "name": "main_window",
        "width": 500,
        "height": 500
    },
    "image": { 
        "src": "Images/Sun.png",
        "name": "sun1",
        "hOffset": 250,
        "vOffset": 250,
        "alignment": "center"
    },
    "text": {
        "data": "Click Here",
        "size": 36,
        "style": "bold",
        "name": "text1",
        "hOffset": 250,
        "vOffset": 100,
        "alignment": "center",
        "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
    }
}}

And then logstash rewrite this json as:

    "debug": "on",
    "window": {
        "vOffset": 100,
}}
{"widget": {
    "text": {
        "data": "Click Here",
        "size": 36,
        "style": "bold",
        "name": "text1",
        "hOffset": 250,
        "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
    }
        "title": "Sample Konfabulator Widget",
        "name": "main_window",
        "width": 500,
        "height": 500
    },
    "image": { 
        "src": "Images/Sun.png",
        "name": "sun1",
        "hOffset": 250,
        "vOffset": 250,
        "alignment": "center"
    },
        "alignment": "center",

There is exactly the same number of character/line between those two files but the lines are not in same order, so the json becomes invalid.
Do you have a solution for this issue ?
Thanks.

Take a look at the pipeline.ordered option.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.