# Index logic documentation?

**URL:** <https://discuss.elastic.co/t/index-logic-documentation/134481>\
**Category:** Elasticsearch\
**Created:** [June 4, 2018, 7:34pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481 "2018-06-04T19:34:44Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![brandondash](https://avatars.discourse-cdn.com/v4/letter/b/76d3ee/32.png) [@brandondash](https://discuss.elastic.co/u/brandondash)\
**Post date:** [June 4, 2018, 7:34pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/1 "2018-06-04T19:34:44Z")

</div>

I am looking for a writeup on when/how Elasticsearch decides to create new indices.

I went to list all indexes on my single node ELK stack server expecting to find exactly two: the one being fed by Logstash and the one created by Kibana. What I found was literally dozens of logstash indexes split by date, all open, and with no real rhyme nor reason to when they were created. (the kibana one was there also so at least I got that part right).

Where can I read up on this so I can stop it (or at least control it) ?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 4, 2018, 7:37pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/2 "2018-06-04T19:37:09Z")

</div>

> [@brandondash](#):
>
> I am looking for a writeup on when/how Elasticsearch decides to create new indices.

The only time Elasticsearch will create an index is if it is asked to do so.  
That may be from explicit request via a mapping, or implicitly if a create request is made for an index that doesn't exist.

> [@brandondash](#):
>
> I went to list all indexes on my single node ELK stack server expecting to find exactly two: the one being fed by Logstash and the one created by Kibana. What I found was literally dozens of logstash indexes split by date, all open, and with no real rhyme nor reason to when they were created.

What does `_cat/indices?v` show?

---

<div class="post-metadata">

**Author:** ![brandondash](https://avatars.discourse-cdn.com/v4/letter/b/76d3ee/32.png) [@brandondash](https://discuss.elastic.co/u/brandondash)\
**Post date:** [June 4, 2018, 7:40pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/3 "2018-06-04T19:40:29Z")

</div>

That _is_ what \_cat/indices?v shows

All yellow (expected since I am single node)  
All open (?)  
All from logstash (they all follow the same naming convention "logstash-date")  
All pri 5  
All rep 1  
Doc count is all over the place - as small as 11k, as large as 27k  
Store size all over the place - 74mb to 105mb

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 4, 2018, 7:43pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/4 "2018-06-04T19:43:58Z")

</div>

> [@brandondash](#):
>
> All from logstash (they all follow the same naming convention "logstash-date")

Then that's what's causing them to be created.

---

<div class="post-metadata">

**Author:** ![brandondash](https://avatars.discourse-cdn.com/v4/letter/b/76d3ee/32.png) [@brandondash](https://discuss.elastic.co/u/brandondash)\
**Post date:** [June 4, 2018, 7:47pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/5 "2018-06-04T19:47:01Z")

</div>

OK so logstash is causing them to be created. Presumably each index is holding a unique subset of the overall data I actually ingest. Since the names don't give me any context clues, how do I determine which data ends up in which index? I can tell you I didn't actively ask for any indices to be created. Whatever happened was triggered by logstash at a date AFTER I plugged in the pipeline.

Better yet, how can I tell logstash to give me an index name that is actually meaningful?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 4, 2018, 7:48pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/6 "2018-06-04T19:48:05Z")

</div>

You'd need to share your Logstash config I think, it'll help us understand what's going on.

---

<div class="post-metadata">

**Author:** ![brandondash](https://avatars.discourse-cdn.com/v4/letter/b/76d3ee/32.png) [@brandondash](https://discuss.elastic.co/u/brandondash)\
**Post date:** [June 4, 2018, 7:53pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/7 "2018-06-04T19:53:43Z")

</div>

The obvious guess is that a new index request happens every time the Logstash thread is restarted. Is there a way to tell Logstash NOT to ask for a new index and instead feed into the latest current index?

I am happy to post my pipeline, but it isn't very exciting.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 4, 2018, 7:55pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/8 "2018-06-04T19:55:08Z")

</div>

> [@brandondash](#):
>
> The obvious guess is that a new index request happens every time the Logstash thread is restarted.

That's unlikely.

If you can post your config it'll help immensely.

---

<div class="post-metadata">

**Author:** ![brandondash](https://avatars.discourse-cdn.com/v4/letter/b/76d3ee/32.png) [@brandondash](https://discuss.elastic.co/u/brandondash)\
**Post date:** [June 4, 2018, 8:05pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/10 "2018-06-04T20:05:45Z")

</div>

```
input {
    file {
        type => "bbb-web"
        path => [
            "/data/logstash/logs/*conf*{{ logstash_path_qualifier }}/bbb-web.log"
        ]
    }

    file {
        type => "freeswitch-log"
        path => [
            "/data/logstash/logs/*conf*{{ logstash_path_qualifier }}/freeswitch-log.log"
        ]
        codec => multiline {
            pattern => "%{SYSLOGTIMESTAMP} %{HOSTNAME} freeswitch-log: %{TIMESTAMP_ISO8601} "
            negate => "true"
            what => "previous"
            multiline_tag => "freeswitch_multiline"
        }
    }

    file {
        type => "freeswitch-master"
        path => [
            "/data/logstash/logs/*conf*{{ logstash_path_qualifier }}/freeswitch-master.log"
        ]
    }    
    
    file {
        type => "chatdb-mysql-audit"
        path => [
            "/data/logstash/logs/*chatdb*{{ logstash_path_qualifier }}/mysql-audit.log"
        ]
    }

    file {
        type => "confdb-mysql-audit"
        path => [
            "/data/logstash/logs/*confdb*{{ logstash_path_qualifier }}/mysql-audit.log"
        ]
    }

    file {
        type => "nginx-alb"
        path => [
            "/data/logstash/logs/*alb*{{ logstash_path_qualifier }}/nginx-access.log",
            "/data/logstash/logs/*alb*{{ logstash_path_qualifier }}/nginx-error.log"
        ]
    }

    file {
        type => "nginx-conference"
        path => [
            "/data/logstash/logs/*conf*{{ logstash_path_qualifier }}/nginx-access.log",
            "/data/logstash/logs/*conf*{{ logstash_path_qualifier }}/nginx-error.log"
        ]
    }

    file {
        type => "nginx-portal"
        path => [
            "/data/logstash/logs/*portal*{{ logstash_path_qualifier }}/nginx-access.log",
            "/data/logstash/logs/*portal*{{ logstash_path_qualifier }}/nginx-error.log"
        ]
    }

    file {
        type => "openfire-error"
        path => [
            "/data/logstash/logs/*chat*{{ logstash_path_qualifier }}/openfire-error.log"
        ]
    }

    file {
        type => "openfire-info"
        path => [
            "/data/logstash/logs/*chat*{{ logstash_path_qualifier }}/openfire-info.log"
        ]
    }

}

filter {

    if "bbb-web" in [path] {
        grok { 
            patterns_dir => ["/etc/logstash/patterns"]
            match => { "message" => "%{BBB_WEB}" }
        }    
    }

    if "freeswitch-log" in [path] {
        if "freeswitch_multiline" in [tags] {
            # If we find a multiline entry, strip out the recurring prefixes that occur mid-line
            mutate { 
                gsub => [
                    "message",
                    # NOTE - gsub does not recognize predefined grok patterns, so we have to hand enter them
                    "\n\b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b +(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?!<[0-9])(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9])(?::(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))(?![0-9]) \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b) freeswitch-log: ",
                    " "
                ]
            }
        }
        grok { 
            patterns_dir => ["/etc/logstash/patterns"]
            match => { "message" => "%{FREESWITCH_LOG}" }
        }    
    }
    
    if "freeswitch-master" in [path] {
        grok { 
            patterns_dir => ["/etc/logstash/patterns"]
            match => { "message" => "%{FREESWITCH_MASTER}" }
        }    
    }

    if "mysql-audit" in [path] {
        grok { 
            patterns_dir => ["/etc/logstash/patterns"]
            match => { "message" => "%{MYSQL_AUDIT}" }
        }    
    }
    
    if "nginx-access" in [path] {
        grok { 
            patterns_dir => ["/etc/logstash/patterns"]
            match => { "message" => "%{NGINX_ACCESS}" }
        }    
    }

    if "nginx-error" in [path] {
        grok { 
            patterns_dir => ["/etc/logstash/patterns"]
            match => { "message" => "%{NGINX_ERROR}" }
        }
    }

    if "openfire-error" in [path] {
        grok { 
            patterns_dir => ["/etc/logstash/patterns"]
            match => { "message" => "%{OPENFIRE_ERROR}" }
        }    
    }

    if "openfire-info" in [path] {
        grok { 
            patterns_dir => ["/etc/logstash/patterns"]
            match => { "message" => "%{OPENFIRE_INFO}" }
        }    
    }

    if "Guest" in [full_name] { mutate { add_tag => "guest_user" } }
    if " FOO " in [full_name] { mutate { add_tag => "FOO" } }
    if " BAR " in [full_name] { mutate { add_tag => "BAR" } }
    if " BAZ " in [full_name] { mutate { add_tag => "BAZ" } }

}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
    }
}
```

---

<div class="post-metadata">

**Author:** ![brandondash](https://avatars.discourse-cdn.com/v4/letter/b/76d3ee/32.png) [@brandondash](https://discuss.elastic.co/u/brandondash)\
**Post date:** [June 4, 2018, 8:23pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/11 "2018-06-04T20:23:59Z")

</div>

... and the logstash configuration proper (in case you wanted that too):

```
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
config.reload.automatic: true
path.logs: /var/log/logstash
```

---

<div class="post-metadata">

**Author:** ![azhar](https://avatars.discourse-cdn.com/v4/letter/a/74df32/32.png) [@azhar](https://discuss.elastic.co/u/azhar)\
**Post date:** [June 4, 2018, 9:02pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/12 "2018-06-04T21:02:30Z")

</div>

Hi,

From the elasticsearch output plugin documentation, logstash defaults to index name `"logstash-%{+YYYY.MM.dd}"` if not provided explicitly.

Refer to [this](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index) link for more details.

Hence, if you don't want new indexes to be created every day, you will need to set the index name explicitly in the pipeline config.

```
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "<index_name>"
    action => "index"
  }
}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 4, 2018, 10:16pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/13 "2018-06-04T22:16:15Z")

</div>

Why do you want a single big index?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 2, 2018, 10:16pm UTC](https://discuss.elastic.co/t/index-logic-documentation/134481/14 "2018-07-02T22:16:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.