Create Dynamic Index in Elasticsearch

Elastic Stack Logstash
1

I am trying to configure logstash to create dynamic indexes in Elasticsearch. Here are my configuration

input{
  kafka
  {
    bootstrap_servers => ["192.168.56.2:9092,192.168.56.23:9092,192.168.56.24:9092"]
    topics => ["logstash", "qafilebeat", "prfilebeat", "app-error-logs"]
    auto_offset_reset => latest
    decorate_events => true
    group_id => "logstash_indexer"
  }
}
 
filter {
  json {
    source => "message"
  }
}
output {
    elasticsearch {
      hosts => ["192.168.56.5"]
    }
}

Here are sample logs

{"log":"Nov 06, 2017 6:50:12 PM org.glassfish.jersey.internal.Errors logErrors\n","stream":"stderr","attrs":{"APPLICATION":"app1","DATACENTER":"ny","ENVIRONMENT":"qa12","com.cluster":"qa12-app1","com.container-name":"app1-2_33_1","time":"2017-11-06T18:50:12.578546934Z"}
{"log":"WARNING: The following warnings have been detected: WARNING: The (sub)resource method postTaxeneConfig in com.app1.server.taxene.config.connector.jersey.TaxeneConfigResource contains empty path annotation.\n","stream":"stderr","attrs":{"APPLICATION":"app1","DATACENTER":"NY","ENVIRONMENT":"qa12","com.cluster":"qa12-app1"},"time":"2017-11-06T18:50:12.57858747Z"}

​It is creating only logstash-* index in Elasticsearch. But I want to create -* where Environment could be prod or qa, Application be app1, app2 etc and cluster be a name.

Any help would be highly appreciated​.

Cheers
Ferdous Shibly

2

Example:

elasticsearch {
  ...
  index => "%{[attrs][ENVIRONMENT]}-%{[attrs][APPLICATION]}-%{+YYYY.MM.dd}"
}
3

Thanks Magnus Bäck, it is working. I have one more quire. How can I send logs to different Elasticsearch cluster based upon fields? I want to send qa application logs to qa Elasticsearch cluster and prod logs to prod Elasticsearch cluster.

like

output {
  if [attrs][ENVIRONMENT] =~ /(qa|dv|ee|pe)\d+/ {
    elasticsearch {
      hosts => ["192.168.56.5"]
      index => "%{[attrs][ENVIRONMENT]}%{[attrs][APPLICATION]}-%{+YYYY.MM.dd}"
    }
  }else {
    elasticsearch {
      hosts => ["192.168.56.10"]
      index => "%{[attrs][ENVIRONMENT]}%{[attrs][APPLICATION]}-%{+YYYY.MM.dd}"
    }
  }
}
4

Yeah, that's what it would look like.

1 Like
5

If you are doing this then it probably makes sense to change your indices to monthly and/or reduce the shard count from the default 5.

1 Like
6

Magnus,

How can I check whether the field is emply or not? like in this case, if [attrs][APPLICATION] is empty or not present or if [attrs][APPLICATION] is generic, how can I send the logs to qageneric index other with it will create appropriate indexes?

output {
  if [attrs][ENVIRONMENT] =~ /(qa*|dv*|ee*|pe*)\d+/ {
    if [attrs][APPLICATION] =~ /generic/ or ![attrs][APPLICATION] {
      elasticsearch {
        hosts => ["192.168.56.5"]
        index => "%{[attrs][ENVIRONMENT]}-generic-%{+YYYY.MM.dd}"
      }
    }
  }
}
7 
if [attrs][APPLICATION] =~ /generic/ or ![attrs][APPLICATION] {

This should work except that the negation operator is not rather than !.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.