# Logstash is loading duplicates **URL:** \ **Category:** Logstash\ **Created:** [June 21, 2018, 12:07am UTC](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773 "2018-06-21T00:07:21Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![databeata](https://avatars.discourse-cdn.com/v4/letter/d/5e9695/32.png) [@databeata](https://discuss.elastic.co/u/databeata)\ **Post date:** [June 21, 2018, 12:07am UTC](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773/1 "2018-06-21T00:07:21Z")
Create a log file in hdfs from hue with one record. ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/f/ffc118d79b4561c61cc43326a82f6beedd119959.png) Run nifi job to copy log file from hdfs to elk server folder. ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b6bb7f2df1feb98c9e6838d1454d63d3f3e62e52.jpg) Run logstash job. input { file { path =\> "/path/\*.log" type =\> "test" start\_position =\> beginning sincedb\_path =\> "/path/test" } } filter { if [type] == "test" { grok { match =\> { "message" =\> "%{GREEDYDATA:test}" } } } } output { if [type] == "test" { elasticsearch { hosts =\> "ip:9200" index =\> "test" } } else { stdout { codec =\> rubydebug } } } List index documents. curl -XGET 'ip:9200/test/\_search?q=\*&pretty&pretty' "message" : "APBaaN APBaaN\_BusinessPartner 07-05-2018 04:35:434 2018-05-07 06:41:04.844 941", "@timestamp" : "2018-06-22T19:48:42.326Z", Append 2nd row to hdfs log file. hdfs dfs -appendToFile - /path/test.log APBaaN APBaaN\_BusinessPartner 07-06-2018 05:35:434 2018-06-07 07:41:04.844 941 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/c/0cb8fba9a33b96f6164cda8965fa99491b50179a.jpg) List index documents. curl -XGET 'ip:9200/test/\_search?q=\*&pretty&pretty' "message" : "APBaaN APBaaN\_BusinessPartner 07-05-2018 04:35:434 2018-05-07 06:41:04.844 941", "@timestamp" : "2018-06-22T19:48:42.326Z", "message" : "APBaaN APBaaN\_BusinessPartner 07-05-2018 04:35:434 2018-05-07 06:41:04.844 941", "@timestamp" : "2018-06-22T20:28:41.471Z", "message" : "APBaaN APBaaN\_BusinessPartner 07-06-2018 05:35:434 2018-06-07 07:41:04.844 941", "@timestamp" : "2018-06-22T20:28:41.471Z", 1st and 2nd are duplicates. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 21, 2018, 5:24am UTC](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773/2 "2018-06-21T05:24:44Z")
Exactly how are you adding the new row to the file? If you are using an editor, this does not append, but generally results in a new file being created and renamed. Logstash will process this as a new file and send both lines. You should in this case see two entries in .sincedb. ---
**Author:** ![databeata](https://avatars.discourse-cdn.com/v4/letter/d/5e9695/32.png) [@databeata](https://discuss.elastic.co/u/databeata)\ **Post date:** [June 21, 2018, 2:39pm UTC](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773/3 "2018-06-21T14:39:20Z")
I am using an editor to add 2nd row to log file. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 21, 2018, 2:43pm UTC](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773/4 "2018-06-21T14:43:54Z")
That explains it then. ---
**Author:** ![databeata](https://avatars.discourse-cdn.com/v4/letter/d/5e9695/32.png) [@databeata](https://discuss.elastic.co/u/databeata)\ **Post date:** [June 21, 2018, 3:57pm UTC](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773/5 "2018-06-21T15:57:32Z")
if I don't use editor then how do I test adding 2nd row. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 21, 2018, 4:01pm UTC](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773/6 "2018-06-21T16:01:46Z")
If you are on Linux: `echo "Test log entry" >> /path/test.log` ---
**Author:** ![databeata](https://avatars.discourse-cdn.com/v4/letter/d/5e9695/32.png) [@databeata](https://discuss.elastic.co/u/databeata)\ **Post date:** [June 23, 2018, 4:45am UTC](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773/7 "2018-06-23T04:45:15Z")
There is no hdfs dfs -echo command. There is hdfs dfs -appendToFile command. I have included nifi job in my original post. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 23, 2018, 5:59am UTC](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773/8 "2018-06-23T05:59:06Z")
OK, you are trying to read from HDFS. That could indeed be a problem. I would recommend you read [this note about the file input plugin and remote file systems](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_reading_from_remote_network_volumes). The same applies to [Filebeat](https://www.elastic.co/guide/en/beats/filebeat/6.3/faq.html#filebeat-network-volumes). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 21, 2018, 5:59am UTC](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773/9 "2018-07-21T05:59:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.