Logstash doesn't create new indexes

Elastic Stack Logstash
1

am a new user of ELK 5.5.1 (Elasticsearch, Logstash, Kibana).

I am making a monitoring server using ELK with Ubuntu 16.4.

I have for the moment two sources of data, Netflow from my rooter and Collectd from my server.

Per default all the data comme in Logstash perfectly and goes out in Elastic in the same index "logstash-%{YYYY.MM.DD}".

The data flow works fine but Kibana can't map both dataflow in a unique index because the type of data is different for the some fields.

That's why I try to send the dataflow in two differents index.

From Kibana I installed X-pack and setup a new user named "logstash_internal" with the role "logstash_writer" which have all privileges (Cluster Privileges => all, Index Privileges => *, Privileges => all).

I made the following config file for Logstash to push the data in two new indexes:

input {
 udp {
  port => 25826
  buffer_size => 1452
  codec => collectd { }
 }
 udp {
  port => 1734
  codec => netflow {
   versions => [5, 9]
  }
  type => netflow
 }
}

output {
 if ( [type] == "netflow" ) {
  elasticsearch {
   hosts => ["localhost:9200"]
   user => logstash_internal
   password => logstashpwd
   index => "lg-OpenWrt-%{+YYYY.MM.dd}"
  }
 } else {
  elasticsearch {
   hosts => ["localhost:9200"]
   user => logstash_internal
   password => logstashpwd
   index => "lg-Monitor-%{+YYYY.MM.dd}"
  }
 }
}

But Elasticsearch doesn't make new index. And when I check here: http://127.0.0.1:9200/_cat/indices?v and in Timelion in Kibana, the dataflow is not received anymore and the indexes "lg-OpenWrt-%{+YYYY.MM.dd}" and "lg-Monitor-%{+YYYY.MM.dd}" don't exist.

health status index                             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .monitoring-es-6-2017.07.29       LZMojNYGRDuDw4GCwzBF8w   1   1      14385          180     10.2mb         10.2mb
yellow open   logstash-2017.07.29               lCQ-WltYRpiLIeCGBi900A   5   1       3217            0        1mb            1mb
yellow open   .monitoring-kibana-6-2017.07.29   xYhmEpjwRLu0jTFa1N_ldA   1   1        762            0    438.9kb        438.9kb
yellow open   .monitoring-es-6-2017.07.28       DcwVvdwcSxatRtZUnwxURQ   1   1       7305          162        5mb            5mb
yellow open   .watcher-history-3-2017.07.29     uAq4UMt2QZqoATDx29N79Q   1   1        639            0      551kb          551kb
yellow open   .watcher-history-3-2017.07.28     kVhig4-VQrmN4apudXHd3A   1   1        455            0    515.8kb        515.8kb
yellow open   .triggered_watches                qJnmD7XdQOitkFHLOkjj_g   1   1          0            0     48.1kb         48.1kb
yellow open   .monitoring-logstash-6-2017.07.28 qQVLWxtWQd-ber_3-2UVRw   1   1        135            0    239.1kb        239.1kb
green  open   .security                         k-p9fCvjQjK_MpQ9Y85mfg   1   0          8            0     29.5kb         29.5kb
yellow open   .monitoring-logstash-6-2017.07.29 zFQSFH51QYKTlTcKmzdmow   1   1        378            0    336.4kb        336.4kb
yellow open   logstash-2017.07.28               hAieuJgwSi26nMS9t_zHZw   5   1       1071            0    366.6kb        366.6kb
yellow open   .monitoring-alerts-6              ROR1eoOZTqeVt0QC6aEZPg   1   1          1            0      6.2kb          6.2kb
yellow open   .monitoring-kibana-6-2017.07.28   IRYOmymfTniNraqLKyfleA   1   1        392            0    249.1kb        249.1kb
yellow open   .watches                          NBepeMe7Quuva1VtQXu4SA   1   1          4            0       20kb           20kb
yellow open   .kibana                           Y_hBJIPESReGwuWw-ekfbA   1   1          1            0      3.8kb          3.8kb

Does anyone know why Logstash doesn't make the two indexes?

Many thanks for any advise you can give me.

2

My guess would be that the logstash_internal user does not have sufficient privileges to create anew indices. An example of what a Logstash indexing role can look like is available here. Replace the logstash-* index pattern with something that matches your indices.

3

Many thank Chritian,

Actually I set the full privileges to logstash_internal by changing the prvileges of the role logstash_writer.

Here is what I have set. Do you think there is an issue?

GET _xpack/security/user/logstash_internal/

return =>

{
  "logstash_internal": {
    "username": "logstash_internal",
    "roles": [
      "logstash_system",
      "logstash_writer",
      "logstash_reader"
    ],
    "full_name": "Internal Logstash User",
    "email": null,
    "metadata": {},
    "enabled": true
  }
}

GET _xpack/security/role/logstash_writer/

return =>

{
  "logstash_writer": {
    "cluster": [
      "manage_index_templates",
      "monitor",
      "all"
    ],
    "indices": [
      {
        "names": [
          "logstash-*",
          "lg-Monitor-*",
          "lg-OpenWrt-*",
          "*"
        ],
        "privileges": [
          "write",
          "delete",
          "create_index",
          "delete_index",
          "index",
          "manage",
          "create",
          "all"
        ]
      }
    ],
    "run_as": [
      "kibana",
      "logstash_internal",
      "logstash_system"
    ],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }
  }
}
4

Actually, per default when I don't specify the "index" field in the logstash config file, logstash request to Elasticsearch to make a new daily index "logstash-%{YYYY-MM-DD}" without issue.

This is what Elasticsearch returns:

[2017-07-29T17:36:24,470][INFO ][o.e.c.m.MetaDataCreateIndexService] [GZiWe8m] [logstash-2017.07.29] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_]

And if I put "index => "logstash-%{+YYYY.MM.dd}" in the logstash's config file (which is equivalent to the default value), Timelion in Kibana confirms the reception of the dataflow.

But, when I change the "index" field with a new index, Elasticsearch doesn't show anything new Index creation in the terminal.

5

I finally found the issue.

I tried PUT /lg-OpenWrt-2017.07.29?pretty from Kibana and it returned:

{
  "error": {
    "root_cause": [
      {
        "type": "invalid_index_name_exception",
        "reason": "Invalid index name [lg-OpenWrt-2017.07.29], must be lowercase",
        "index_uuid": "_na_",
        "index": "lg-OpenWrt-2017.07.29"
      }
    ],
    "type": "invalid_index_name_exception",
    "reason": "Invalid index name [lg-OpenWrt-2017.07.29], must be lowercase",
    "index_uuid": "_na_",
    "index": "lg-OpenWrt-2017.07.29"
  },
  "status": 400
}

So I understood I have to edit my index name with lower case only.

Now it works as expected.

Case closed.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.