# Noob index management - naming / cleanup
**URL:** https://discuss.elastic.co/t/noob-index-management-naming-cleanup/148032
**Category:** Logstash
**Created:** [September 11, 2018, 1:41am UTC](https://discuss.elastic.co/t/noob-index-management-naming-cleanup/148032 "2018-09-11T01:41:39Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)
#### Post date: [September 11, 2018, 1:41am UTC](https://discuss.elastic.co/t/noob-index-management-naming-cleanup/148032/1 "2018-09-11T01:41:39Z")
Ive setup a new beat-\>logstash-\>elastic system and have started collecting data from a number of sources (files, metrics, etc). Looking through the indexes available via Kibana Im seeing nearly a hundred entries marked 'index-' (EG "filebeat-2018.08.10") (same for "metricbeat").
- Is this by design? Do I want / need a new index every day?
Looking through 'filebeat.reference.yml' it appears that these are generated by elasticsearch. I don't send data directly there but rather through logstash... Or do I have this configured incorrectly? I have the elasticsearch entries in 'filebeat.yml' commented out. There's certainly data being sent in so I assume the 'logstash' entries are at least accurate.
- Lots of spurious entries are appearing in the indexes (EG "#033.31;1mbuilds#033.0;m.keyword")
--\> where are these coming from and how do I eliminate them?
---
### Author:  [@AquaX](https://discuss.elastic.co/u/AquaX)
#### Post date: [September 11, 2018, 1:48am UTC](https://discuss.elastic.co/t/noob-index-management-naming-cleanup/148032/2 "2018-09-11T01:48:21Z")
By default the elasticsearch output filter will put its documents in the logstash-%{+YYYY.MM.dd} [Docs Link](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index)
This is done on purpose in order to help you clean up old data. When you create an index pattern inside Kibana you can then define your dashboard to use logstash-\* pattern so that it picks up all of the indexes.
---
### Author:  [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)
#### Post date: [September 11, 2018, 11:18am UTC](https://discuss.elastic.co/t/noob-index-management-naming-cleanup/148032/3 "2018-09-11T11:18:12Z")
Im confused as Im sending all my data via logstash. Or are you referring to the logstash-\>elastic output?
---
### Author:  [@AquaX](https://discuss.elastic.co/u/AquaX)
#### Post date: [September 11, 2018, 2:54pm UTC](https://discuss.elastic.co/t/noob-index-management-naming-cleanup/148032/4 "2018-09-11T14:54:14Z")
Yes, sorry.
I meant:
output {
elasticsearch {}
}
This will automatically generate the index name logstash-%YYYYMMdd unless you define another index name
---
### Author:  [@A\_B](https://discuss.elastic.co/u/A_B)
#### Post date: [September 11, 2018, 3:35pm UTC](https://discuss.elastic.co/t/noob-index-management-naming-cleanup/148032/5 "2018-09-11T15:35:30Z")
> [@ethrbunny](#):
>
> (EG "filebeat-2018.08.10") (same for "metricbeat").
That sounds very much like beats is shipping directly to Elasticsearch. What does e.g. your filebeat output config look like?
> Looking through 'filebeat.reference.yml' it appears that these are generated by elasticsearch.
Elasticsearch indices are created by beats or Logstash
> Do I want / need a new index every day?
Having daily indices, at least for logs, help with house keeping. It is much easier to delete whole indices based on time than it is to delete documents from indices based on time. Usually for logs you set some sort of retention period unless you have unlimited storage capacity.
You could go less granular and have weekly indices as well, especially if your log volume is low.
---
### Author:  [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)
#### Post date: [September 11, 2018, 4:13pm UTC](https://discuss.elastic.co/t/noob-index-management-naming-cleanup/148032/6 "2018-09-11T16:13:56Z")
My config:
```
filebeat.inputs:
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
exclude_files: ['.gz$']
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#================================ Outputs =====================================
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["logstash.otech:5044"]
```
I use puppet to push out this same config to all hosts sending logs.
---
### Author:  [@A\_B](https://discuss.elastic.co/u/A_B)
#### Post date: [September 12, 2018, 7:07am UTC](https://discuss.elastic.co/t/noob-index-management-naming-cleanup/148032/7 "2018-09-12T07:07:04Z")
Have you restarted the filebeat service? With `reload.enabled: false` you will have to do that manually.
---
### Author:  [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)
#### Post date: [September 12, 2018, 10:10am UTC](https://discuss.elastic.co/t/noob-index-management-naming-cleanup/148032/8 "2018-09-12T10:10:52Z")
Yes. I believe the indexes are being generated by logstash though:
(from /etc/logstash/conf.d/beats.conf)('input' and 'filter' sections removed)
```
output {
elasticsearch {
hosts => "10.xx.xx.xx:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 10, 2018, 10:11am UTC](https://discuss.elastic.co/t/noob-index-management-naming-cleanup/148032/9 "2018-10-10T10:11:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.