Same message being sent twice

[xShadow]

Hi,

I am not sure if I should place this topic under logstash or elasticsearch but I was testing the function of sending logs from logstash to elasticsearch and I noticed that the same message appears twice in the elasticsearch.

For example, I have a csv file where I add one line of message after starting logstash. The output of the configuration file points to my elasticsearch instance (single-node cluster), where I have also set an indexing template to set number of replication to 0. The output in my elasticsearch index shows the same message twice, but with different id. Am I supposed to set something to prevent this from happening?

Thank you!

[Christian_Dahlqvist]

How exactly are you adding data to your file? A common mistake when testing is to edit the file using an editor, which typically creates a new file which is then renamed in the background. As this is a technically a new file with the same name, it will be reprocessed from the beginning. Make sure that you append to the existing file as that is what both Filebeat and Logstash expects.

[xShadow]

Thanks for the reply. I was testing using a text editor (gedit) at first, but I have tested it using
echo test >> test.csv after your reply and I'm still seeing two records in the index.

[Christian_Dahlqvist]

What does your config look like? How are you running Logstash? Are you restarting Logstash at any point or just leaving it running?

[xShadow]

Below is the configuration file that I am using for testing purposes.

input {
    file {
        path => "/opt/logstash/log/test.csv"
    }
}
 
output {
    elasticsearch {
        hosts => ["x.x.x.x"]
        index => "testing"
    }
    stdout {}
}

I'm not sure what you mean by how am I running Logstash, but I specified the configuration file as shown below.
./logstash -f /path/to/config/test.conf

I normally restart Logstash service after changing the configuration file in case of errors. i.e. I restarted the logstash service before doing what you mentioned earlier.