Duplicated indices

Elastic Stack Logstash
1

Hello,

When I do a GET _cat/indices, I see that there is an index created twice for my data log:

yellow open .kibana             RE2BXwgDSnuWmXlFvEn9BA 1 1      4 0  35.3kb  35.3kb
yellow open carxx-2017.07.02    yAMp8AB1ROOMP9V-U1-Rsg 5 1 172610 0 131.9mb 131.9mb
yellow open carxx-2017.07.01    UbZ-2x3DQZ2sAki67NVh9w 5 1  65567 0  19.7mb  19.7mb
yellow open filebeat-2017.08.01 dDe7TI9sSmyGFi7F1O4rRw 5 1 123845 0    46mb    46mb
yellow open filebeat-2017.08.02 zysFh1O5Sg29zwcS6kvlGg 5 1  62100 0  48.7mb  48.7mb

The "carxx" one is the data index that I want and not the filebeat one (which are duplicated data). How can I get only the presence of "carxx" index?

Here is my logstash beats configuration:

  input {
        beats {
                port => "5044"
        }
}


filter {

        if [fields][type] == "carxx" {
		
                grok {
                        patterns_dir => ["/etc/logstash/conf.d/patterns/"]
                        match => { "message" => "%{ID:id}%{CARXXTIMESTAMP:carxxtimestamp}[%|$]%{TYPE:typetx}[%|$]%{NNUTILISATEUR:nnutilisateur}%{CODETX:codetx}%{NNDOSSIER:nndossier}%{IOUT:iout}%{HEUREREPONSE:heurereponse}%{INS:ins}%{IIN:iin}%{GREEDYDATA}" }
                }
				
				mutate {
						add_field => {
							"fulltx" => "%{typetx}%{codetx}"
						}		
					
						remove_field => [ "message" ]
				}

                date {
                        match => [ "carxxtimestamp", "YYMMddHHmmss" ]
                }

        } 
		
}


output {

			elasticsearch {
					hosts => ["192.168.1.21:9200"]
					index => "%{[fields][type]}-%{+YYYY.MM.dd}"
			}
			
}

Plus, [fields][type] == carxx is also present in the filebeat-* data discovering, so I can't do a filter on it in my logstash conf...

Thanks

2

What does the filebeat config look like?

3

Here is my filebeat.yml:

filebeat.prospectors:
- input_type: log
  paths:
    - /opt/carxx/carxx_test
  fields:
    type: carxx
- input_type: log
  paths:
    - /opt/xml_log/xml_test.log
  fields:
    type: txlog
fields:
  env: staging
output.elasticsearch:
  hosts: ["192.168.1.21:9200"]
  template.name: "filebeat"
  template.path: "filebeat.template.json"
output.logstash:
  hosts: ["192.168.1.21:5044"]
logging.level: debug
4

That's why :slight_smile:
You only want one of those.

6

Hmmmm, I have re-created the index with the new configuration, and I still have a "filebeat-*" index created...

yellow open carxx-2017.07.02    uPT8_RSWSYm1iuAa8Lpc4w 5 1 7799 0 3.4mb 3.4mb
yellow open .kibana             8WTr3fypTVm0GP0emXD2UA 1 1    1 0 3.1kb 3.1kb
yellow open filebeat-2017.08.02 1FFlGMQZRmKWFPKv5oHCbA 5 1 6144 0   4mb   4mb
7

somebody? :confused:

8

The mapping is not the same for data, one is log and the other is doc. I don't understand why.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.