How to create multiple separate index using single conf file in logstash through filebeat?

Elastic Stack Logstash
1

Hi,

I want to create separate indices based on the condition of the logs, for example if my log consist of api1 then it should create index named "api1" and if it consist api2 then create another index named "api2".
Please see the below log and conf for the reference

Log :
xx.xxx.x.x j.d [2023-06-12 10:15:20] "GET /api1/aa/1" 200 278
xx.xxx.x.x j.d [2023-06-12 10:16:21] "GET /api2/aa/1" 201 165
xx.xxx.x.x j.d [2023-06-12 10:17:21] "GET /api3/aa/1" 201 175

Conf :

input {
  beats {
    port => xxxx
  }
}

filter {
      grok {
      match => { "message" => "%{IP: } %{USERNAME: } \[%{TIMESTAMP_ISO8601:timestamp}\] \"%{WORD:http_method} %{PATH: path} HTTP/%{NUMBER: }\" %{NUMBER: } %{NUMBER: }" }
    }
    if [api_path] =~ ^/api1 {
      mutate {
        add_field => { "[@metadata][index_name]" => "api1-logs-%{+YYYY.MM.dd}" }
      }
    } else if [api_path] =~ ^/api2 {
      mutate {
        add_field => { "[@metadata][index_name]" => "api2-logs-%{+YYYY.MM.dd}" }
      }
    } else {
      mutate {
        add_field => { "[@metadata][index_name]" => "other-api-logs-%{+YYYY.MM.dd}" }
      }
    }
}

output {
  if [@metadata][index_name] {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    index => "%{[@metadata][index_name]}"
    user => "######"
    password => "########"
	ssl => true
	ssl_certificate_verification => false
	cacert => "xxx\xx\certs\http_ca.crt""
  }
  }
}
2

Hi @KRISHNA_KUMAR1

welcome to the community.

Before proceeding with the help, could you inform the version you are using? Have you validated that the conditions are valid through stdout?

To test, you can put in your file:

output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

another attempt is to make the conditional in the output, in this documentation there is an example:

Example: Set up Filebeat modules to work with Kafka and Logstash | Logstash Reference [8.8] | Elastic

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.