New index is not created

Hi. I have Filebeat + ELK stack. My Logstash has a pretty standard configuration:

input {
  beats {
    port => 5044

filter {
  grok {
  	match => {
  		"message" => "^(?<log_date>\d{4}-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d)\d?\|(?<level>WARN|INFO|DEBUG|ERROR|FATAL)\|%{GREEDYDATA:message}"
  date {
    match => ["log_date", "yyyy-MM-dd HH:mm:ss.SSS"]
    timezone => "+01:00"
    remove_field => ["log_date"]

output {
  elasticsearch {
    hosts => ["elastic:9201", "elastic:9202"]

Yes, I have 2 instances of ES on a single machine, but they use different directories. For some reason indices are not created daily I have 2 indices. Both created the 28th of December. New docs are added to the second one.
Does someone know what I did wrong?


You can specify indices like this in your output

output {
  elasticsearch {
    hosts => ["elastic:9201", "elastic:9202"]
    index = >  "foobar"

You should reffer to : documentation

if you want something daily you should try something like this

index = > "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

I don't wanna specify it manually. AFAIK default one should work as I expect. I suppose that it is related to ILM, for some reason, Logstash is not able to figure out the format.

Maybe, you should try logstash log level debug and see what's happening.

Nothing realy interesting. It uses default index. To be sure I dropped index template and LS recreated it. It still writes to old index.

[2020-01-16T14:16:46,488][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@index = "logstash-%{+YYYY.MM.dd}"
[2020-01-16T14:16:46,488][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@manage_template = true
[2020-01-16T14:16:46,489][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@template_name = "logstash"
[2020-01-16T14:16:46,489][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@template_overwrite = false

I have also added the second file output plugin and I see that timestamp is properly formatted. What date does the ES plugin use for index names?

Logs related to ILM:

[2020-01-16T14:16:46,497][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_enabled = "auto"
[2020-01-16T14:16:46,497][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_rollover_alias = "logstash"
[2020-01-16T14:16:46,498][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_pattern = "{now/d}-000001"
[2020-01-16T14:16:46,498][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_policy = "logstash-policy"

I used custom index format and used lowercase y instead of uppercase and it fixed issue. Other peoples struggles with it also Github

Oh i forgot that, ineed no caps in index names

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.