New index is not created

Elastic Stack Elasticsearch
1

Hi. I have Filebeat + ELK stack. My Logstash has a pretty standard configuration:

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
  	match => {
  		"message" => "^(?<log_date>\d{4}-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d)\d?\|(?<level>WARN|INFO|DEBUG|ERROR|FATAL)\|%{GREEDYDATA:message}"
  	}
  }
  date {
    match => ["log_date", "yyyy-MM-dd HH:mm:ss.SSS"]
    timezone => "+01:00"
    remove_field => ["log_date"]
  }
}

output {
  elasticsearch {
    hosts => ["elastic:9201", "elastic:9202"]
  }
}

Yes, I have 2 instances of ES on a single machine, but they use different directories. For some reason indices are not created daily I have 2 indices. Both created the 28th of December. New docs are added to the second one.
Does someone know what I did wrong?

2

Hi

You can specify indices like this in your output

output {
  elasticsearch {
    hosts => ["elastic:9201", "elastic:9202"]
    index = >  "foobar"
  }
}

You should reffer to : documentation

if you want something daily you should try something like this

index = > "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
3

I don't wanna specify it manually. AFAIK default one should work as I expect. I suppose that it is related to ILM, for some reason, Logstash is not able to figure out the format.

4

Maybe, you should try logstash log level debug and see what's happening.

5

Nothing realy interesting. It uses default index. To be sure I dropped index template and LS recreated it. It still writes to old index.

[2020-01-16T14:16:46,488][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@index = "logstash-%{+YYYY.MM.dd}"
[2020-01-16T14:16:46,488][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@manage_template = true
[2020-01-16T14:16:46,489][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@template_name = "logstash"
[2020-01-16T14:16:46,489][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@template_overwrite = false

I have also added the second file output plugin and I see that timestamp is properly formatted. What date does the ES plugin use for index names?

Edit:
Logs related to ILM:

[2020-01-16T14:16:46,497][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_enabled = "auto"
[2020-01-16T14:16:46,497][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_rollover_alias = "logstash"
[2020-01-16T14:16:46,498][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_pattern = "{now/d}-000001"
[2020-01-16T14:16:46,498][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_policy = "logstash-policy"
6

I used custom index format and used lowercase y instead of uppercase and it fixed issue. Other peoples struggles with it also Github

7

Oh i forgot that, ineed no caps in index names

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.