Multiple Elasticsearch Indices in Logstash output

Elastic Stack Elasticsearch
1

Hi all,

Currently I'm using Elasticsearch auto clear index script which find all the indexes with format
logstash-%{+YYYY.MM.dd} and clear all the indices older than 10 days, but for some reason I need to exclude a specific log type.

So I tried to change the elasticsearch's _index value in logstash, I have found a way to do it, but it made elasticsearch to generate new node in cluster which decrease the performance, so I don't want to use it.
output {
if [Type] == "IndexType1" {
elasticsearch {
host => localhost
cluster => elasticsearch
index => "IndexType1-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
host => localhost
cluster => elasticsearch }
}

So I have tried another way on Goolge like below to filter the new field on output but so far not success:
filter {
if [Type] == "IndexType1" {
grok {
match => { "message" => "%{Pattern}" }
mutate {
add_field => { "IndexType" => "IndexType1" }
}
}
}
output {
elasticsearch {
host => localhost
cluster => elasticsearch
index => "logstash%{IndexType}-%{+YYYY.MM.dd}"
}

But it's not working the _index value displayed on Kibana still not changed. Even the field IndexType was created.

I don't know if I did it correctly.
I'm new to logstash, elasticsearch and kibana.So would you please help me to correct the code or advice me if there's any other solutions.

2

So I tried to change the elasticsearch's _index value in logstash, I have found a way to do it, but it made elasticsearch to generate new node in cluster which decrease the performance, so I don't want to use it.

What do you mean by "generate new node in cluster"? Creating an additional index each day will impact performance but I doubt it makes a very big impact. Any method of having different retention policies for different documents is going to impact performance in some way.

3

Hi magnusbaeck,
It's mean currenly my elasticsearch cluster has 2 hosts and 4 nodes, when I add the config:
if [Type] == "IndexType1" {
elasticsearch {
host => localhost
cluster => elasticsearch
index => "IndexType1-%{+YYYY.MM.dd}"
}

Two more nodes appear in the cluster. I don't want that.
Is there any other way to use dynamic indices instead?

4

Indexes and cluster nodes are completely unrelated concepts. I don't know why you're talking about indexes here.

If you don't want Logstash to show up as a cluster node, don't use the node protocol but instead the transport protocol or HTTP. Use the protocol option to change this. See also Transport Client Versus Node Client in the Definitive Guide.

5

Thank so much for clarifying!

Actually my main question is simple: Is there any other way to change the _index default value

Instead of using "index => "logstash%{IndexType}-%{+YYYY.MM.dd}" in the output sector?

6

Not that I'm aware of. Why would you want to do that?

7

Hi magnusbaeck,

As describe from begin: Currently I'm using Elasticsearch auto clear index script which find all the indexes base on format
logstash-%{+YYYY.MM.dd}
and clear all older than 10 days, but I want to exclude a specific log type.
Ex: I have apache log, nginx log, postfix log. Now I want to change the _indexe logstash-%{+YYYY.MM.dd} value of postfix log so it wont be clean by the script.

8

Yes, but what's the problem with setting the index option? If it's not working it's because you're not setting the IndexType field correctly. Looking at your previously posted configuration,

filter {
  if [Type] == "IndexType1" {
    grok {
      match => { "message" => "%{Pattern}" }
    mutate {
      add_field => { "IndexType" => "IndexType1" }
    } 
  }
}
output {
  elasticsearch { 
    host => localhost
    cluster => elasticsearch 
    index => "logstash%{IndexType}-%{+YYYY.MM.dd}"
  }
}

I'd start by changing [Type] to [type].

I'd also change the index name pattern to logstash-%{IndexType}-%{+YYYY.MM.dd} (note hyphen after "logstash") to make sure that the index template still applies (I think it by default applies to indexes whose names match logstash-*).

9

Sorry for slow reply!

Thank you so much for your advice!
I will try your method with logstash-%{IndexType}-%{+YYYY.MM.dd}, hope it should affect on the IndexType1 only.

10

Hi magnus,

Changing to logstash-%{IndexType}-%{+YYYY.MM.dd} didn't work also.
Seems like there's no other way to change the _index.
Thank for all your helps

11

Please show the configuration you're using. Also, please prove that the IndexType field is getting the correct value, e.g. by using the stdout output to dump the contents of a message at the end of the pipeline.

12

Thank you magnusbaeck!
The issue solved when I upgrade to new logstash.