Logstash output to Elasticsearch name index based on field?

Hello. Im trying to name a index based on a field but it doesnt seem to work:

output
{

elasticsearch {

hosts => ["localhost"]
user => ["elastic"]
password => ["pass"]

index => "winlogbeat-server-winapplication-%{[event][provider]}-%{+yyyy.MM.dd}"

    }

}

As you see, Im trying even a generic one and it does not produce the index.

How can I do this?

Tried index => "winlogbeat-mssql-winapplication-%{[_source][event][provider]}-%{+yyyy.MM.dd}"
but no dice either

Amazing. I dont understand why but:

filter {

mutate {

    lowercase => [ "[event][provider]" ]

}

}

I tried this because same thing happens to the hostname and I need to lowercase it.

Why?

Yes elasticsearch allow only lowercase name for indexes

Index name must be lowercase, cannot begin with an underscore, and cannot contain commas

Is there a way to instead of

filter {

mutate {

    lowercase => [ "[event][provider]" ]

}

}

Add a additional field called [eventproviderlower], insert the value of [event][provider] and use that? Its mostly not to mutate or touch the logs in any way shape or form (compilance and such)

Thanks

You can use the mutate filter to create a new field, lowercase it and use it in your output, you can use a [@metadata] field for it, the [@metadata] is only present in your pipeline, it will not be in your final document.

You need something like this:

mutate {
    copy => { "[event][provider]" => "[@metadata][eventlower]" }
}
mutate {
    lowercase => [ "[@metadata][eventlower]" ]
}

You need two different mutates because of the order which the filters are executed internally by mutate.

And in your output you need to use %{[@metadata][eventlower]}

Thanks a lot :slight_smile: Great idea

This is what you mean correct

mutate {
copy => { "[host][hostname]" => "[@metadata][hostlower]" }
}
mutate {
lowercase => [ "[@metadata][hostlower]" ]
}

output
{

elasticsearch {

hosts => ["localhost"]
user => ["elastic"]
password => ["4234"]

index => "winlogbeat-winapplication-%{[@metadata][hostlower]}-%{+yyyy.MM.dd}"

    }

}

Because I just implemented it and it doesnt seem to be working

(I used hostname as another example)

No ideas?

What exactly does that mean?

The config seems right, do you have anything in the logs? Share your full pipeline and a sample log, your problem could be in other parts of your pipeline.

Figured out the issue.

I had to open and close a filter section:

filter
{
mutate {
copy => { "[host][hostname]" => "[@metadata][hostlower]" }
}
mutate {
lowercase => [ "[@metadata][hostlower]" ]
}

}