Logstash output to Elasticsearch name index based on field?

Elastic Stack Logstash
#1

Hello. Im trying to name a index based on a field but it doesnt seem to work:

output
{

elasticsearch {

hosts => ["localhost"]
user => ["elastic"]
password => ["pass"]

index => "winlogbeat-server-winapplication-%{[event][provider]}-%{+yyyy.MM.dd}"

    }

}

As you see, Im trying even a generic one and it does not produce the index.

How can I do this?

#2

Tried index => "winlogbeat-mssql-winapplication-%{[_source][event][provider]}-%{+yyyy.MM.dd}"
but no dice either

#3

Amazing. I dont understand why but:

filter {

mutate {

    lowercase => [ "[event][provider]" ]

}

}

I tried this because same thing happens to the hostname and I need to lowercase it.

Why?

#4

Yes elasticsearch allow only lowercase name for indexes

Index name must be lowercase, cannot begin with an underscore, and cannot contain commas

#5

Is there a way to instead of

filter {

mutate {

    lowercase => [ "[event][provider]" ]

}

}

Add a additional field called [eventproviderlower], insert the value of [event][provider] and use that? Its mostly not to mutate or touch the logs in any way shape or form (compilance and such)

Thanks

#6

You can use the mutate filter to create a new field, lowercase it and use it in your output, you can use a [@metadata] field for it, the [@metadata] is only present in your pipeline, it will not be in your final document.

You need something like this:

mutate {
    copy => { "[event][provider]" => "[@metadata][eventlower]" }
}
mutate {
    lowercase => [ "[@metadata][eventlower]" ]
}

You need two different mutates because of the order which the filters are executed internally by mutate.

And in your output you need to use %{[@metadata][eventlower]}

#7

Thanks a lot :slight_smile: Great idea

#8

This is what you mean correct

mutate {
copy => { "[host][hostname]" => "[@metadata][hostlower]" }
}
mutate {
lowercase => [ "[@metadata][hostlower]" ]
}

output
{

elasticsearch {

hosts => ["localhost"]
user => ["elastic"]
password => ["4234"]

index => "winlogbeat-winapplication-%{[@metadata][hostlower]}-%{+yyyy.MM.dd}"

    }

}

Because I just implemented it and it doesnt seem to be working

(I used hostname as another example)

#9

No ideas?

#10

What exactly does that mean?

#11

The config seems right, do you have anything in the logs? Share your full pipeline and a sample log, your problem could be in other parts of your pipeline.

#12

Figured out the issue.

I had to open and close a filter section:

filter
{
mutate {
copy => { "[host][hostname]" => "[@metadata][hostlower]" }
}
mutate {
lowercase => [ "[@metadata][hostlower]" ]
}

}