# Elasticsearch remove docs of Suricata **URL:** \ **Category:** Logstash\ **Created:** [October 28, 2016, 12:57pm UTC](https://discuss.elastic.co/t/elasticsearch-remove-docs-of-suricata/64284 "2016-10-28T12:57:37Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![brunopsitech](https://avatars.discourse-cdn.com/v4/letter/b/eb9ed0/32.png) [@brunopsitech](https://discuss.elastic.co/u/brunopsitech)\ **Post date:** [October 28, 2016, 12:57pm UTC](https://discuss.elastic.co/t/elasticsearch-remove-docs-of-suricata/64284/1 "2016-10-28T12:57:37Z")
Hi, i am using Logstash to generate logs of IPS. But elasticsearch delete all docs except last one, i dont know why. My Logstash config: input { file { path =\> ["/var/log/suricata/eve.json"] codec =\> "json" type =\> "Suricata" } } filter { if [type] == "Suricata" { date { match =\> ["timestamp", "ISO8601"] } ruby { code =\> "if event['event\_type'] == 'fileinfo'; event['fileinfo']['type']=event['fileinfo']['magic'].to\_s.split(',')[0]; end;" } } } output { elasticsearch { action =\> "index" user =\> "elastic" password =\> "system" hosts =\> ["127.0.0.1"] document\_id =\> "suricata" # index =\> "logstash-%{+YYYY.MM.DD}" ``` } ``` } My log of suricata {"timestamp":"2016-10-28T10:45:14.000075-0200","flow\_id":904705990262178,"event\_type":"flow","src\_ip":"192.168.1.103","src\_port":23630,"dest\_ip":"64.4.23.165","dest\_port":40017,"proto":"UDP","flow":{"pkts\_toserver":1,"pkts\_toclient":1,"bytes\_toserver":73,"bytes\_toclient":548,"start":"2016-10-28T10:40:12.196002-0200","end":"2016-10-28T10:40:12.383161-0200","age":0,"state":"established","reason":"timeout"}} {"timestamp":"2016-10-28T10:45:14.000133-0200","flow\_id":1410309540351611,"event\_type":"flow","src\_ip":"192.168.1.103","src\_port":23630,"dest\_ip":"189.63.100.104","dest\_port":9828,"proto":"UDP","flow":{"pkts\_toserver":1,"pkts\_toclient":1,"bytes\_toserver":161,"bytes\_toclient":99,"start":"2016-10-28T10:40:13.724603-0200","end":"2016-10-28T10:40:13.745362-0200","age":0,"state":"established","reason":"timeout"}} {"timestamp":"2016-10-28T10:45:15.000059-0200","flow\_id":783012386910091,"event\_type":"flow","src\_ip":"192.168.1.103","src\_port":57029,"dest\_ip":"192.168.1.1","dest\_port":53,"proto":"UDP","app\_proto":"dns","flow":{"pkts\_toserver":1,"pkts\_toclient":1,"bytes\_toserver":80,"bytes\_toclient":148,"start":"2016-10-28T10:40:13.733067-0200","end":"2016-10-28T10:40:13.739907-0200","age":0,"state":"established","reason":"timeout"}} {"timestamp":"2016-10-28T10:45:15.000129-0200","flow\_id":1513588471434918,"event\_type":"flow","src\_ip":"192.168.1.103","src\_port":23630,"dest\_ip":"65.55.223.36","dest\_port":40016,"proto":"UDP","flow":{"pkts\_toserver":1,"pkts\_toclient":1,"bytes\_toserver":78,"bytes\_toclient":62,"start":"2016-10-28T10:40:13.724646-0200","end":"2016-10-28T10:40:13.864608-0200","age":0,"state":"established","reason":"timeout"}} {"timestamp":"2016-10-28T10:45:15.000155-0200","flow\_id":113706174996590,"event\_type":"flow","src\_ip":"fe80:0000:0000:0000:2f6f:0e61:a3b4:906a","src\_port":5353,"dest\_ip":"ff02:0000:0000:0000:0000:0000:0000:00fb","dest\_port":5353,"proto":"UDP","flow":{"pkts\_toserver":7,"pkts\_toclient":0,"bytes\_toserver":700,"bytes\_toclient":0,"start":"2016-10-28T10:44:27.024686-0200","end":"2016-10-28T10:44:43.144673-0200","age":16,"state":"new","reason":"timeout"}} {"timestamp":"2016-10-28T10:45:15.000187-0200","flow\_id":2086000237808930,"event\_type":"flow","src\_ip":"157.55.130.154","src\_port":40002,"dest\_ip":"192.168.1.103","dest\_port":23630,"proto":"UDP","flow":{"pkts\_toserver":1,"pkts\_toclient":1,"bytes\_toserver":68,"bytes\_toclient":53,"start":"2016-10-28T10:40:13.724258-0200","end":"2016-10-28T10:40:13.724351-0200","age":0,"state":"established","reason":"timeout"}} {"timestamp":"2016-10-28T10:45:15.000208-0200","flow\_id":1809902534712734,"event\_type":"flow","src\_ip":"179.185.64.35","src\_port":30818,"dest\_ip":"192.168.1.103","dest\_port":23630,"proto":"UDP","flow":{"pkts\_toserver":3,"pkts\_toclient":4,"bytes\_toserver":462,"bytes\_toclient":1617,"start":"2016-10-28T10:33:44.056734-0200","end":"2016-10-28T10:40:13.724586-0200","age":389,"state":"established","reason":"timeout"}} {"timestamp":"2016-10-28T10:45:15.000229-0200","flow\_id":419765544509584,"event\_type":"flow","src\_ip":"192.168.1.103","src\_port":5353,"dest\_ip":"224.0.0.251","dest\_port":5353,"proto":"UDP","flow":{"pkts\_toserver":7,"pkts\_toclient":0,"bytes\_toserver":560,"bytes\_toclient":0,"start":"2016-10-28T10:44:27.024720-0200","end":"2016-10-28T10:44:43.144699-0200","age":16,"state":"new","reason":"timeout"}} {"timestamp":"2016-10-28T10:45:15.000474-0200","flow\_id":1148778261883622,"event\_type":"flow","src\_ip":"54.245.98.57","src\_port":32099,"dest\_ip":"192.168.1.103","dest\_port":28432,"proto":"UDP","flow":{"pkts\_toserver":3,"pkts\_toclient":3,"bytes\_toserver":182,"bytes\_toclient":238,"start":"2016-10-28T10:44:41.502502-0200","end":"2016-10-28T10:44:44.506604-0200","age":3,"state":"new","reason":"timeout"}} In elasticsearch i have only 1 doc, all others are removed. Someone khow what i have to do for work as append mode ? ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [October 28, 2016, 1:11pm UTC](https://discuss.elastic.co/t/elasticsearch-remove-docs-of-suricata/64284/2 "2016-10-28T13:11:00Z")
> [@brunopsitech](#): > > document\_id =\> "suricata" You have assigned the same document id to every document. What that results in is each successive document being re-indexed over the top of the previous one, replacing it. ---
**Author:** ![brunopsitech](https://avatars.discourse-cdn.com/v4/letter/b/eb9ed0/32.png) [@brunopsitech](https://discuss.elastic.co/u/brunopsitech)\ **Post date:** [October 28, 2016, 2:25pm UTC](https://discuss.elastic.co/t/elasticsearch-remove-docs-of-suricata/64284/3 "2016-10-28T14:25:27Z")
Thanks, work fine. But now ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [October 31, 2016, 4:06pm UTC](https://discuss.elastic.co/t/elasticsearch-remove-docs-of-suricata/64284/4 "2016-10-31T16:06:02Z")
Was there more to your reply, @brunopsitech? It just ends. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:31am UTC](https://discuss.elastic.co/t/elasticsearch-remove-docs-of-suricata/64284/5 "2017-07-06T04:31:55Z")