Logstash keeps regenerating last event

Hello everyone.

I'm new to Elastic stak and I'm having some issues with my logstash pipeline.

I have the following code:

    input {
        beats {
    		id => "data-collector-filebeat-input"
            port => 5044
    		add_field => {"application" => "data-collector"}
        }
    }


    filter {
    	if [application] == "data-collector" {
    		mutate {
    			id => "data-collector-remove-fileds"
    			remove_field => ["agent", "ecs", "@version", "host", "log", "tags", "input", "container"]
    			rename => { "@timestamp" => "processTime" }
    		}

    		json {
    			id => "data-collector-get-message"
    			source => "message"
    			remove_field => ["message"]
    		}

    		date {
    			id => "data-collector-parse-start-date"
    			match => ["date_start", "YYYY/MM/dd HH:mm:ss"]
    		}
    	}
    }


    output {
    	if [application] == "data-collector" {
    		stdout {
    			id => "data-collector-stdout"
    			#codec => json_lines
    		}

    		elasticsearch  {
    			id => "data-collector-save-to-elasticsearch"
    			index => "data-%{+YYYY.MM.dd}"
    			template_name => "data"
    			template => "/opt/elk/logstash/logstash-7.5.2/templates/data.json"
    		}
    	}
    }

I send a message using filebeat input, after some time without sending new messages logstash keeps duplicating the last message and so writting to elasticsearch.

I'm using logstash 7.5.2 and Filebeat 7.5.2 as well.

Has anyone know what am I doing wrong?

Thanks you a lot

[EDIT] I'm adding some more information
I've been doing some more investigation and I'm going to give more detail.

I'm using logstash 7.5.2 and Filebeat 7.5.2 both installed using ".tar.gz" files in a CentOS 7 operating system.
When running directly both systems everything seems to work fine, this problem starts when I manually create both services (for filebeat and logstash) using "systemctl". To be more precise my handly created service for logstash is the one causing the problem.

Now muy question is: Is it recommended to install Filebeat and Logstash using repository packages and ".tar.gz" ones are to be used just for development propose?

Thanks you a lot

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.