Filebeat ignoring closing bracket {

Elastic Stack Beats
1

Hi Team,,

I have a multiline JSON message in a file , I used custom log integration to parse it with below multiline config.

multiline:
match: after
negate: true
pattern: '^{'

I was getting all the data except last closing } , due to which my logstash json filter is failing, can someone help please

2

Hi @Venkata_Raja,

Welcome back! Have you tried amending the regex pattern to include the closing bracket?

3

I am unable to get correct pattern to do that.
My log file contain single event only that will be rotated every 5 minutes. That single event contains 2500 lines under a single JSOn object comprising multiple child JSON objects.

Can you please help either with the pattern or with process on how to add } at the end of message field in logstash so that I can parse JSON

4

Can you share a sample log message that you want to match?

5

You need to share your logstash configuration.

But if you do not have the last } in your file, this means that there is no line break after it and filebeat didn't read this line.

What is the source of the data the filebeat is reading? Do you have any control over it to change what is being written in the file?

6

Yes , you are right, there is only single event in log file - it will rotate once it has new log entry,so just 1 log at a time and we don't have any line break or new line at the end. That is why agent is ignoring last line that is } .
I can't modify source file as some 3rd party vendor is producing it.
Any way to add } at the end of message value using logstash filters so that it can be parsed as JSON.

7

As mentioned, you need to share your Logstash configuration.

8

Hi Leandro,

As an overview, I am just getting data from beats input (agent integration), using JSON filter on message field (which is failing) and sending output to Elasticsearch
Thank you


input { 
  beats {
    port => 5044
  }
}
filter {
  json {
    source => "message"
  }
}
output { 
   elasticsearch { hosts => ["localhost:9200"] } 
}
9

You can try to add the following filter before your json filter.

    mutate {
        update => {
            "message" => "%{message}}"
        }
    }
1 Like
10

Thank you so much. It worked

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.