# Logstash doesn't create new indexes **URL:** \ **Category:** Logstash\ **Created:** [July 29, 2017, 6:50pm UTC](https://discuss.elastic.co/t/logstash-doesnt-create-new-indexes/95047 "2017-07-29T18:50:08Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Omoidashita](https://avatars.discourse-cdn.com/v4/letter/o/e495f1/32.png) [@Omoidashita](https://discuss.elastic.co/u/Omoidashita)\ **Post date:** [July 29, 2017, 6:50pm UTC](https://discuss.elastic.co/t/logstash-doesnt-create-new-indexes/95047/1 "2017-07-29T18:50:08Z")
am a new user of ELK 5.5.1 (Elasticsearch, Logstash, Kibana). I am making a monitoring server using ELK with Ubuntu 16.4. I have for the moment two sources of data, Netflow from my rooter and Collectd from my server. Per default all the data comme in Logstash perfectly and goes out in Elastic in the same index "logstash-%{YYYY.MM.DD}". The data flow works fine but Kibana can't map both dataflow in a unique index because the type of data is different for the some fields. That's why I try to send the dataflow in two differents index. From Kibana I installed X-pack and setup a new user named "logstash\_internal" with the role "logstash\_writer" which have all privileges (Cluster Privileges =\> all, Index Privileges =\> \*, Privileges =\> all). I made the following config file for Logstash to push the data in two new indexes: ``` input { udp { port => 25826 buffer_size => 1452 codec => collectd { } } udp { port => 1734 codec => netflow { versions => [5, 9] } type => netflow } } output { if ( [type] == "netflow" ) { elasticsearch { hosts => ["localhost:9200"] user => logstash_internal password => logstashpwd index => "lg-OpenWrt-%{+YYYY.MM.dd}" } } else { elasticsearch { hosts => ["localhost:9200"] user => logstash_internal password => logstashpwd index => "lg-Monitor-%{+YYYY.MM.dd}" } } } ``` But Elasticsearch doesn't make new index. And when I check here: [http://127.0.0.1:9200/\_cat/indices?v](http://127.0.0.1:9200/_cat/indices?v) and in Timelion in Kibana, the dataflow is not received anymore and the indexes "lg-OpenWrt-%{+YYYY.MM.dd}" and "lg-Monitor-%{+YYYY.MM.dd}" don't exist. ``` health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open .monitoring-es-6-2017.07.29 LZMojNYGRDuDw4GCwzBF8w 1 1 14385 180 10.2mb 10.2mb yellow open logstash-2017.07.29 lCQ-WltYRpiLIeCGBi900A 5 1 3217 0 1mb 1mb yellow open .monitoring-kibana-6-2017.07.29 xYhmEpjwRLu0jTFa1N_ldA 1 1 762 0 438.9kb 438.9kb yellow open .monitoring-es-6-2017.07.28 DcwVvdwcSxatRtZUnwxURQ 1 1 7305 162 5mb 5mb yellow open .watcher-history-3-2017.07.29 uAq4UMt2QZqoATDx29N79Q 1 1 639 0 551kb 551kb yellow open .watcher-history-3-2017.07.28 kVhig4-VQrmN4apudXHd3A 1 1 455 0 515.8kb 515.8kb yellow open .triggered_watches qJnmD7XdQOitkFHLOkjj_g 1 1 0 0 48.1kb 48.1kb yellow open .monitoring-logstash-6-2017.07.28 qQVLWxtWQd-ber_3-2UVRw 1 1 135 0 239.1kb 239.1kb green open .security k-p9fCvjQjK_MpQ9Y85mfg 1 0 8 0 29.5kb 29.5kb yellow open .monitoring-logstash-6-2017.07.29 zFQSFH51QYKTlTcKmzdmow 1 1 378 0 336.4kb 336.4kb yellow open logstash-2017.07.28 hAieuJgwSi26nMS9t_zHZw 5 1 1071 0 366.6kb 366.6kb yellow open .monitoring-alerts-6 ROR1eoOZTqeVt0QC6aEZPg 1 1 1 0 6.2kb 6.2kb yellow open .monitoring-kibana-6-2017.07.28 IRYOmymfTniNraqLKyfleA 1 1 392 0 249.1kb 249.1kb yellow open .watches NBepeMe7Quuva1VtQXu4SA 1 1 4 0 20kb 20kb yellow open .kibana Y_hBJIPESReGwuWw-ekfbA 1 1 1 0 3.8kb 3.8kb ``` Does anyone know why Logstash doesn't make the two indexes? Many thanks for any advise you can give me. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 29, 2017, 7:10pm UTC](https://discuss.elastic.co/t/logstash-doesnt-create-new-indexes/95047/2 "2017-07-29T19:10:24Z")
My guess would be that the `logstash_internal` user does not have sufficient privileges to create anew indices. An example of what a Logstash indexing role can look like is available [here](https://www.elastic.co/guide/en/x-pack/current/logstash.html). Replace the `logstash-*` index pattern with something that matches your indices. ---
**Author:** ![Omoidashita](https://avatars.discourse-cdn.com/v4/letter/o/e495f1/32.png) [@Omoidashita](https://discuss.elastic.co/u/Omoidashita)\ **Post date:** [July 29, 2017, 7:47pm UTC](https://discuss.elastic.co/t/logstash-doesnt-create-new-indexes/95047/3 "2017-07-29T19:47:34Z")
Many thank Chritian, Actually I set the full privileges to **logstash\_internal** by changing the prvileges of the role **logstash\_writer**. Here is what I have set. Do you think there is an issue? **GET \_xpack/security/user/logstash\_internal/** return =\> ``` { "logstash_internal": { "username": "logstash_internal", "roles": [ "logstash_system", "logstash_writer", "logstash_reader" ], "full_name": "Internal Logstash User", "email": null, "metadata": {}, "enabled": true } } ``` **GET \_xpack/security/role/logstash\_writer/** return =\> ``` { "logstash_writer": { "cluster": [ "manage_index_templates", "monitor", "all" ], "indices": [ { "names": [ "logstash-*", "lg-Monitor-*", "lg-OpenWrt-*", "*" ], "privileges": [ "write", "delete", "create_index", "delete_index", "index", "manage", "create", "all" ] } ], "run_as": [ "kibana", "logstash_internal", "logstash_system" ], "metadata": {}, "transient_metadata": { "enabled": true } } } ``` ---
**Author:** ![Omoidashita](https://avatars.discourse-cdn.com/v4/letter/o/e495f1/32.png) [@Omoidashita](https://discuss.elastic.co/u/Omoidashita)\ **Post date:** [July 29, 2017, 8:13pm UTC](https://discuss.elastic.co/t/logstash-doesnt-create-new-indexes/95047/4 "2017-07-29T20:13:09Z")
Actually, per default when I don't specify the "index" field in the logstash config file, logstash request to Elasticsearch to make a new daily index "logstash-%{YYYY-MM-DD}" without issue. This is what Elasticsearch returns: `[2017-07-29T17:36:24,470][INFO][o.e.c.m.MetaDataCreateIndexService] [GZiWe8m] [logstash-2017.07.29] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_]` And if I put " **index =\> "logstash-%{+YYYY.MM.dd}**" in the logstash's config file (which is equivalent to the default value), Timelion in Kibana confirms the reception of the dataflow. But, when I change the "index" field with a new index, Elasticsearch doesn't show anything new Index creation in the terminal. ---
**Author:** ![Omoidashita](https://avatars.discourse-cdn.com/v4/letter/o/e495f1/32.png) [@Omoidashita](https://discuss.elastic.co/u/Omoidashita)\ **Post date:** [July 29, 2017, 9:53pm UTC](https://discuss.elastic.co/t/logstash-doesnt-create-new-indexes/95047/5 "2017-07-29T21:53:45Z")
I finally found the issue. I tried **PUT /lg-OpenWrt-2017.07.29?pretty** from Kibana and it returned: ``` { "error": { "root_cause": [ { "type": "invalid_index_name_exception", "reason": "Invalid index name [lg-OpenWrt-2017.07.29], must be lowercase", "index_uuid": "_na_", "index": "lg-OpenWrt-2017.07.29" } ], "type": "invalid_index_name_exception", "reason": "Invalid index name [lg-OpenWrt-2017.07.29], must be lowercase", "index_uuid": "_na_", "index": "lg-OpenWrt-2017.07.29" }, "status": 400 } ``` So I understood I have to edit my index name with lower case only. Now it works as expected. Case closed. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 26, 2017, 9:53pm UTC](https://discuss.elastic.co/t/logstash-doesnt-create-new-indexes/95047/6 "2017-08-26T21:53:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.