Writing events to different indexes through a variable

Elastic Stack Logstash
1

Hi everyone

I did not find a suitable topic in the community, I decided to create a topic.
Has anyone tried this configuration ?

output { **
** elasticsearch { **
** index => "%{[some_field][sub_field]}-%{+YYYY.MM.dd}" } }

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html№

Can't run with parameter ilm_policy

And here's another error in the logstash

Invalid index name [], must be lowercase", "index_uuid"=>"na", "index"=>""}}}}

How to fix this ?

2

Index names can't have uppercase characters in the name, all letters needs to be lowercase.

You can use ILM, but you would need to set it up in the index template, if you use ilm_enabled as true or ilm_rollover_alias you won't be able to use variable substituion in the index name as explained in the documentation.

This does not means that you cannot use ILM, you can, but need to configure it in the index template, not in the logstash output.

3

Thanks a lot for your answer !

If in my configuration Field1

index => "%{[Field1]}-%{+YYYY.MM.dd}" } }

If in my configuration the Field1 is taken from events which contains values = index names. Should I convert this field to lower case ?

{
"lowercase": {
"field": "Field1"
}
}

Is it possible to do this process in Output ? So that the register of field values is not changed in the events themselves already in the elastic.

Please, show where ILM is configured through the index template ?

4

No, you need to apply a lowercase filter using mutate in the filter block, it is not possible to do it in the output, also, the documentation you shared is from the lowercase processor for ingest pipelines, which run in elasticsearch, what you need is to use mutate and apply the lowercase option.

Check this part of the mutate documentation.

You will need something like this:

    filter {
      mutate {
        lowercase => [ "fieldname" ]
      }
    }

You need to create an index template according to the documentation, them on the settings of the template you can set both the lifecycle policy and the rollover alias, both are index settings.

You would need something like this in your index template:

    "template": {
      "settings": {
          "index" : {
              "lifecycle": {
                  "name": "policy-name"
              }
          }
      }
    }
5

Hi Leandrojmp

Thanks again for your help.
But something is still missing for me (

My config

filter {
.....
....
........

mutate {
  lowercase => [ "[Properties][app]" ]
    }

}

output {
  elasticsearch {
    hosts => [**********]
    user => "*****"
    password => "********"
    index => "%{[Properties][app]}-%{+YYYY.MM.dd}"
    template_name => "lsnext-ft"
  }
}

I also commented on the template (template_name), but the same thing.
This field Properties.app goes nested by JSON.

Always throws errors.

.
.
.
[2022-10-17T22:31:53,725][ERROR][logstash.outputs.elasticsearch][lsn-ft] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"%{[Properties][app]}-2022.10.17", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x28886b95>], :response=>{"index"=>{"_index"=>"%{[Properties][app]}-2022.10.17", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [%{[Properties][app]}-2022.10.17], must be lowercase", "index_uuid"=>"_na_", "index"=>"%{[Properties][app]}-2022.10.17"}}}}
[2022-10-17T22:31:53,726][ERROR][logstash.outputs.elasticsearch][lsn-ft] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"%{[Properties][app]}-2022.10.17", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x1c47a1c9>], :response=>{"index"=>{"_index"=>"%{[Properties][app]}-2022.10.17", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [%{[Properties][app]}-2022.10.17], must be lowercase", "index_uuid"=>"_na_", "index"=>"%{[Properties][app]}-2022.10.17"}}}}
[2022-10-17T22:31:53,726][ERROR][logstash.outputs.elasticsearch][lsn-ft] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"%{[Properties][app]}-2022.10.17", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x5a633a56>], :response=>{"index"=>{"_index"=>"%{[Properties][app]}-2022.10.17", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [%{[Properties][app]}-2022.10.17], must be lowercase", "index_uuid"=>"_na_", "index"=>"%{[Properties][app]}-2022.10.17"}}}}
[2022-10-17T22:31:53,726][ERROR][logstash.outputs.elasticsearch][lsn-ft] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"%{[Properties][app]}-2022.10.17", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x6002e645>], :response=>{"index"=>{"_index"=>"%{[Properties][app]}-2022.10.17", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [%{[Properties][app]}-2022.10.17], must be lowercase", "index_uuid"=>"_na_", "index"=>"%{[Properties][app]}-2022.10.17"}}}}
.
.
.

Maybe mutate doesn't work with nested fields ?

6

Mutate works with nested fields, your issue here is that the field [Properties][app] does not exist in your document and since the field does not exist, logstash can not get the value for it and will use the literal string in the index option as the index name.

So logstash is trying to create an index with the literal name of %{[Properties][app]}-%{+YYYY.MM.dd}, which will not work since you have an uppercase letter.

You need to make sure that the field exists before using it in the index option.

You could add this conditional in your output:

output {
  if [Properties][app] {
    elasticsearch {
      hosts => [**********]
      user => "*****"
      password => "********"
      index => "%{[Properties][app]}-%{+YYYY.MM.dd}"
      template_name => "lsnext-ft"
    }
  }
}

This way Logstash will only try to output if the field Properties.app exists.

7

Cooool :slight_smile:
This is how it works !
Thanks !

I'm sorry, can I ask one more of my old branches Painless script - counting equal values in a field ?
Maybe you know painless script well

8

Hi Leandrojmp.
If still it is possible, I will ask on the subject of this branch ?

I had an ilm_pattern configured. Both the day and the number of the index were taken into account.

ilm_pattern => "{now/d}-000001"

After configuratio the index parametr, only day counts.

index => "%{[Properties][app]}-%{+YYYY.MM.dd}"

I tried setting

index => "%{[Properties][app]}-%{+YYYY.MM.dd}-%{+000001}"

Or so

index => "%{[Properties][app]}-%{+YYYY.MM.dd-000001}"

But for some reason, the countdown from 000001 is not conducted, all indexes are prefixed with 000001 even though the date varies.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.