Duplicating message parts in Elastic dashboard

Elastic Stack Logstash
1

I had shipped log to ES with the integration of filebeat and logstash.
This is my logstash.conf

# Read input from filebeat by listening to port 5044 on which filebeat will send the data
input {
    beats {
	    type => "test"
        port => "5044"
    }
}
 
filter {
  #If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
  if [fields][level] == "info" {
    grok {
      match => {"message" => ["(?m)%{TIMESTAMP_ISO8601:timestamp} - %{NOTSPACE:orgCode} - %{GREEDYDATA:message}"]}
	  #break_on_match => true
      add_tag => ["cms-info"]
	  tag_on_failure => ["_grokparsefailure-cms-info"]
    }
 }
output {
   
  stdout {
    codec => rubydebug
  }
 
  # Sending properly parsed log events to elasticsearch
    elasticsearch {
    hosts => ["https://xxxxxxxx.us-east-1.aws.found.io:443"]
    index => "%{[fields][service-name]}-%{[fields][env]}-%{[fields][application]}-%{[fields][level]}-%{+yyyy.MM.dd}"
    user => "elastic"
    password => "xxxxxx"
  }
}

This is my log message

"2019-10-29 21:57:16.9884 - yyyyy - Generating [item settings cloudfront item links] for asset : 137909"

But in my ES board showsed duplicating message parts. like following

1
1.png1380×183 22.3 KB

How can I solve this?

2

You have not specified the overwrite parameter, so the rest of the message is appended to the message field instead of replacing it.

On an unrelated note, this looks like a pattern that could result in a lot of very small indices and shards. This can cause performance and stability problems down the line, so I would recommend you reconsider this approach. See this blog post for details.

1 Like
3

Thanks it works !!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.