# Issue about automatically created index
**URL:** https://discuss.elastic.co/t/issue-about-automatically-created-index/126059
**Category:** Elasticsearch
**Created:** [March 29, 2018, 9:09am UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059 "2018-03-29T09:09:26Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@dyl](https://discuss.elastic.co/u/dyl)
#### Post date: [March 29, 2018, 9:09am UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/1 "2018-03-29T09:09:26Z")
Hello ELK community !
I'll explain you a big problem, look at this :
**I empty the cache :**
> curl -XDELETE '[http://localhost:9200/\_all](http://localhost:9200/_all)'
Answer :
> {"acknowledged":true}
**I display my indexes:**
> curl -XGET 'localhost:9200/\_cat/indices?v&pretty'
Answer :
> health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
So ok, no data anymore on my Elasticsearch.
**Now :**
Configuration of my pipeline to parse my MalwareBytes logs:
> input {
> file {
> path =\> "/home/XXX/ malwarebytes/\*.csv"
> start\_position =\> "beginning"
> sincedb\_path =\> "/dev/null"
> }
> }
> filter {
> csv {
> separator =\> ";"
> columns =\> ["Name","Status","Category","Type","EndPoint","Group","Policy","Scanned At","Reported At","Affected Application"]
> }
> }
> output {
> elasticsearch {
> hosts =\> "[http://localhost:9200](http://localhost:9200)"
> **index =\> "malwarebytes-report"**
> }
> stdout {}
> }
Then , I send my report month by month to elasticsearch (report-jan for january, report-feb...)
For exemple, I send my report of January, including 28 recording :
**I launch my pipeline :**
bin/logstash -f /etc/logstash/conf.d/pipeline\_malwarebytes.conf --config.reload.automatic
This pipeline listening /home/XXX/malwarebytes/ currently.
I send my report with WinScp and then I display my indexes :
> [root@lrtstfpe1 malwarebytes]# curl -XGET 'localhost:9200/\_cat/indices?v&pretty'
> health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
> yellow open malwarebytes-report dSjLd3tdR5qEwtREcHUW0w 5 1 28 0 460b 460b
Everything is OK, my index is created and my 28 recordings are inside.. Perfect, but now, PROBLEM :
I send my 2nd report, which of mars, including 150 recording :
My pipeline still listening, I send my 2nd report with WinScp, I display my indexes and ... :
> [root@lrtstfpe1 malwarebytes]# curl -XGET 'localhost:9200/\_cat/indices?v&pretty'
> health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
> yellow open malwarebytes-report dSjLd3tdR5qEwtREcHUW0w 5 1 178 0 253.7kb 253.7kb
> yellow open logstash-2018.03.29 FLz2cD\_mQjW5-XnGjF0Twg 5 1 300 0 142.5kb 142.5kb
Wtf is that ? So ok my 150 new recording have been added at my index (150+28=178),but with **an other index was created**!? and why with the double of my datas (150x2=300).
Can anyone help me to try to understand what's happened and how to don't have anymore this index "logstash..." when I import my datas ?
Thx you so much !
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 29, 2018, 9:11am UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/2 "2018-03-29T09:11:37Z")
Please don't post pictures of text, they are difficult to read and some people may not be even able to see them 🙂
---
### Author:  [@dyl](https://discuss.elastic.co/u/dyl)
#### Post date: [March 29, 2018, 9:23am UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/3 "2018-03-29T09:23:41Z")
Ok I just modify screenshots by text 😉
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 29, 2018, 9:43am UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/4 "2018-03-29T09:43:42Z")
What's your Logstash config look like?
---
### Author:  [@dyl](https://discuss.elastic.co/u/dyl)
#### Post date: [March 29, 2018, 9:46am UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/5 "2018-03-29T09:46:13Z")
It's a basic config, by default : My file without comment lines looks like :
> [root@lrtstfpe1 logstash]# sed '/^#/d' logstash.yml
> path.data: /var/lib/logstash
> path.logs: /var/log/logstash
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 29, 2018, 9:47am UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/6 "2018-03-29T09:47:31Z")
What about the actual pipeline you defined./
---
### Author:  [@dyl](https://discuss.elastic.co/u/dyl)
#### Post date: [March 29, 2018, 9:52am UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/7 "2018-03-29T09:52:27Z")
What do you mean ? The config of my actual pipeline is above in my 1st msg
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 29, 2018, 10:18am UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/8 "2018-03-29T10:18:23Z")
Is that the only pipeline?
---
### Author:  [@dyl](https://discuss.elastic.co/u/dyl)
#### Post date: [March 29, 2018, 12:17pm UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/9 "2018-03-29T12:17:05Z")
Yes the only one, I launch this pipeline and then I transfer my reports into my directory /home/XXX/malwarebytes
---
### Author:  [@dyl](https://discuss.elastic.co/u/dyl)
#### Post date: [March 29, 2018, 1:04pm UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/10 "2018-03-29T13:04:52Z")
I tried the same operation again but adding the command:
> curl -XPOST 'http: // localhost: 9200 / \_cache / clear'
Following this, the index "logstash ..." is no longer created, but the data are doubled in my malwarebytes index, first import: 28 records (normal), second import (328 records 150 \* 2 + 28) instead of 178 records (150 + 28)
I do not understand anything anymore.
If I check my index during the parsing of logstash, at one moment it's good, but 5 secondes later, datas are double :
> **[root@lrtstfpe1 conf.d]**# curl -XGET 'localhost:9200/\_cat/indices?v&pretty'
> health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
> yellow open **malwarebytes-report** NNyWuGI4Q3ypVaiRndqVYw 5 1 **178** 0 170kb 170kb
> **[root@lrtstfpe1 conf.d]**# curl -XGET 'localhost:9200/\_cat/indices?v&pretty'
> health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
> yellow open **malwarebytes-report** NNyWuGI4Q3ypVaiRndqVYw 5 1 **328** 0 170kb 170kb
---
### Author:  [@dyl](https://discuss.elastic.co/u/dyl)
#### Post date: [March 29, 2018, 1:23pm UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/11 "2018-03-29T13:23:45Z")
This is the complete evolution of my index :
//Check after the first report sent to logstash :
> **[root@lrtstfpe1 conf.d]**# curl -XGET 'localhost:9200/\_cat/indices?v&pretty'
> health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
> yellow open malwarebytes-report PO4g6rKRTb6yuMDb7i-6sg 5 1 **28** 0 25.3kb 25.3kb
//Check after the 2nd report sent to Logstash (Both commands are sent to one second of interval) :
> **[root@lrtstfpe1 conf.d]**# curl -XGET 'localhost:9200/\_cat/indices?v&pretty'
> health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
> yellow open malwarebytes-report PO4g6rKRTb6yuMDb7i-6sg 5 1 **178** 0 263.3kb 263.3kb
> **[root@lrtstfpe1 conf.d]**# curl -XGET 'localhost:9200/\_cat/indices?v&pretty'
> health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
> yellow open malwarebytes-report PO4g6rKRTb6yuMDb7i-6sg 5 1 **328** 0 263.3kb 263.3kb
**PS : If my files are already in the directory before starting the pipeline, the number sent is good : 178.**
---
### Author:  [@dyl](https://discuss.elastic.co/u/dyl)
#### Post date: [April 1, 2018, 9:40pm UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/12 "2018-04-01T21:40:20Z")
Can somebody help me ?
Thx for all 🙂
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 29, 2018, 9:40pm UTC](https://discuss.elastic.co/t/issue-about-automatically-created-index/126059/13 "2018-04-29T21:40:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.