Issue about automatically created index

[dyl (bus)]

Hello ELK community !

I'll explain you a big problem, look at this :

I empty the cache :

curl -XDELETE 'http://localhost:9200/_all'

Answer :

{"acknowledged":true}

I display my indexes:

curl -XGET 'localhost:9200/_cat/indices?v&pretty'

Answer :

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

So ok, no data anymore on my Elasticsearch.

Now :

Configuration of my pipeline to parse my MalwareBytes logs:

input {
file {
path => "/home/XXX/ malwarebytes/*.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ";"
columns => ["Name","Status","Category","Type","EndPoint","Group","Policy","Scanned At","Reported At","Affected Application"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "malwarebytes-report"
}
stdout {}
}

Then , I send my report month by month to elasticsearch (report-jan for january, report-feb...)

For exemple, I send my report of January, including 28 recording :

I launch my pipeline :
bin/logstash -f /etc/logstash/conf.d/pipeline_malwarebytes.conf --config.reload.automatic

This pipeline listening /home/XXX/malwarebytes/ currently.

I send my report with WinScp and then I display my indexes :

[root@lrtstfpe1 malwarebytes]# curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open malwarebytes-report dSjLd3tdR5qEwtREcHUW0w 5 1 28 0 460b 460b

Everything is OK, my index is created and my 28 recordings are inside.. Perfect, but now, PROBLEM :

I send my 2nd report, which of mars, including 150 recording :

My pipeline still listening, I send my 2nd report with WinScp, I display my indexes and ... :

[root@lrtstfpe1 malwarebytes]# curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open malwarebytes-report dSjLd3tdR5qEwtREcHUW0w 5 1 178 0 253.7kb 253.7kb
yellow open logstash-2018.03.29 FLz2cD_mQjW5-XnGjF0Twg 5 1 300 0 142.5kb 142.5kb

Wtf is that ? So ok my 150 new recording have been added at my index (150+28=178),but with an other index was created !? and why with the double of my datas (150x2=300).

Can anyone help me to try to understand what's happened and how to don't have anymore this index "logstash..." when I import my datas ?

Thx you so much !

[warkolm (Mark Walkom)]

Please don't post pictures of text, they are difficult to read and some people may not be even able to see them

[dyl (bus)]

Ok I just modify screenshots by text

[warkolm (Mark Walkom)]

What's your Logstash config look like?

[dyl (bus)]

It's a basic config, by default : My file without comment lines looks like :

[root@lrtstfpe1 logstash]# sed '/^#/d' logstash.yml
path.data: /var/lib/logstash
path.logs: /var/log/logstash

[warkolm (Mark Walkom)]

What about the actual pipeline you defined./

[dyl (bus)]

What do you mean ? The config of my actual pipeline is above in my 1st msg

[warkolm (Mark Walkom)]

Is that the only pipeline?

[dyl (bus)]

Yes the only one, I launch this pipeline and then I transfer my reports into my directory /home/XXX/malwarebytes

[dyl (bus)]

I tried the same operation again but adding the command:

curl -XPOST 'http: // localhost: 9200 / _cache / clear'

Following this, the index "logstash ..." is no longer created, but the data are doubled in my malwarebytes index, first import: 28 records (normal), second import (328 records 150 * 2 + 28) instead of 178 records (150 + 28)

I do not understand anything anymore.

If I check my index during the parsing of logstash, at one moment it's good, but 5 secondes later, datas are double :

[root@lrtstfpe1 conf.d]# curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open malwarebytes-report NNyWuGI4Q3ypVaiRndqVYw 5 1 178 0 170kb 170kb
[root@lrtstfpe1 conf.d]# curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open malwarebytes-report NNyWuGI4Q3ypVaiRndqVYw 5 1 328 0 170kb 170kb

[dyl (bus)]

This is the complete evolution of my index :

//Check after the first report sent to logstash :

[root@lrtstfpe1 conf.d]# curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open malwarebytes-report PO4g6rKRTb6yuMDb7i-6sg 5 1 28 0 25.3kb 25.3kb

//Check after the 2nd report sent to Logstash (Both commands are sent to one second of interval) :

[root@lrtstfpe1 conf.d]# curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open malwarebytes-report PO4g6rKRTb6yuMDb7i-6sg 5 1 178 0 263.3kb 263.3kb

[root@lrtstfpe1 conf.d]# curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open malwarebytes-report PO4g6rKRTb6yuMDb7i-6sg 5 1 328 0 263.3kb 263.3kb

PS : If my files are already in the directory before starting the pipeline, the number sent is good : 178.

[dyl (bus)]

Can somebody help me ?

Thx for all