Duplicated logs from logstash after append logs

Elastic Stack Logstash
1

When my java application is updating the logs, logstash is re-reading the file from the begning and kibana is displaing twice the same file.
I tried with this configuration :
input { file { path => "/home/ubuntu/ELK/examples/LogsFromLog4j/*.log" type => "oldLogs" sincedb_path => "/home/ubuntu/ELK/logstash-2.1.1/sincedb_log4j/sincedb_log4j.db" } }

and this :
input { beats { type => beats port => 5044 } }
but in both cases I have duplicated logs.

Some tips ? What is wrong ?

2

It sounds like you're not really appending to the file but rather rewriting it. How, exactly, is the file being updated? It looks like you're using Log4j; which appender and how is it configured?

3

Hi,
on my test I opened the file and then modify it appending logs.

Does Logstash is considering like a new file?

4

on my test I opened the file and then modify it appending logs.

This doesn't answer the question "how, exactly, is the file updated". Details are crucial! What does the code look like.

Does Logstash is considering like a new file?

Possibly. You can crank up Logstash's logging by starting it with --verbose to get more clues about what's happening.

5

"on my test I opened the file and then modify it appending logs. "
Step 1 : mylogs.log -> empty
Step 2 : open file mylogs.log -> add new log in the file -> save file
<dialogue> <server>localhost</server> <duration>165</duration> <outgoing><qvs_message appstate="getAPPSTATE_0"></qvs_record></qvs_message></outgoing> </dialogue>
--> mylogs.log = 1 Log
logstash (or filebeats) is reading the file the I can verify it with kibana and I have 1 log
Step 3 : open file mylogs.log -> add new log in the file -> save file
<dialogue> <server>localhost</server> <duration>165</duration> <outgoing><qvs_message appstate="getAPPSTATE_1"></qvs_record></qvs_message></outgoing> </dialogue>
--> mylogs.log = 2 Log
logstash (or filebeats) is reading the file the I can verify it with kibana and I have 3 logs but Iextected 2 logs

I hope this is well detailed

6

Opening the file in a text editor will create a new file with a new inode. Don't do that. Make sure you append to the same file, e.g. like this:

echo "test log message" >> logfile.log
7

Much better, thank you.