Logstash json line by line

Elastic Stack Logstash
1

Hello;

I have this logstash.conf file:

input {
exec {
command => "python3 news.py"
interval => 30
codec => "json"
}
}

output {
stdout { }
}

without the code => json I obtain this result:

"message" => "
{\n "authors": ,\n "id": "1",\n "publish_date": "2020-10-16 23:01:45+00:00",\n "title": "Edition du 17/10/2020"\n}

\n{\n "authors": ,\n "id": "2",\n "publish_date": "2020-11-26 10:00:13+00:00",\n "title": "Communication officielle"\n}

\n{\n "authors": ,\n "id": "3",\n "publish_date": "2020-11-26 09:56:14+00:00",\n "title": "Non remboursé"\n}
"

when I add codec => json

it returns me just the 1st element ie.
"authors": ,
"id": "1",
"publish_date": "2020-10-16 23:01:45+00:00",
"title": "Edition du 17/10/2020"

but I need all the row, one by one jsonified.
pliz how to do that?

2

Indeed, the codec will just consume the first JSON object. You could try

    # Remove newlines, because mutate+split will not match them
    mutate { gsub => [ "message", "\n", "" ] }
    # This removes the "}{"
    mutate { split => { "message" => "}{" } }
    split { field => "message" }
    # Add { at the start of the string and } at the end if not present
    if [message] =~ /^[^{]/ { mutate { gsub => [ "message", "^", "{" ] } }
    if [message] =~ /[^}]$/ { mutate { gsub => [ "message", "$", "}" ] } }
    json { source => "message" }
3

Thank you, please, where to add this code in the Filter {}?

4

Yes, add those lines inside a filter {} section of your logstash configuration.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.