Multiple filebeat input in logstash but how to create different set of index for them

Elastic Stack Logstash
1

I have two set of environment consider as A and B. I have configured A env in logstash.conf and its working as expected but now I have to add B in the same file but I have to create different index for the A and B. How to achieve it?

I have gone through above link but the index is not working with the same.

Below is the logstash.conf I am using

<
input {
beats {
port => "5044"
}
}
filter {
if [fields][log_type] in [ "apache_access" , "apache-access" ] and [fields][application] == "application" and [fields][env] == "A" {
grok {
match => [
"message" , "%{IP:access-ip1} %{IP:access-ip2} - - [%{NOTSPACE:access-timestamp} +%{INT}] "%{WORD:access-httpmethod} %{NOTSPACE:access-request} %{WORD:access-protocol}/%{NUMBER:access-protocolversion}" %{INT:access-status} %{INT:access-responsesize} %{INT:access-responsetime} "-" "%{WORD} %{WORD} %{NOTSPACE}" [ %{WORD} %{WORD} %{WORD}= %{INT:access-responsetimeinmicrosec}%{GREEDYDATA}"
]
overwrite => [ "message" ]
}

if [fields][log_type] in [ "apache_access" , "apache-access" ] and [fields][application] == "application"  and [fields][env] == "B" {
grok {
           "message" , "%{IP:access-ip1} %{IP:access-ip2} \- \- \[%{NOTSPACE:access-timestamp} \+%{INT}\] \"%{WORD:access-httpmethod} %{NOTSPACE:access-request} %{WORD:access-protocol}/%{NUMBER:access-protocolversion}\" %{INT:access-status} %{INT:access-responsesize} %{INT:access-responsetime} \"\-\" \"%{WORD} %{WORD} %{NOTSPACE}\" \[ %{WORD} %{WORD} %{WORD}\= %{INT:access-responsetimeinmicrosec}%{GREEDYDATA}"
    overwrite => [ "message" ]
}
  }

}

output {
if [fields][log_type] in [ "apache_access" , "apache-access" ] and [fields][application] == "application" and [fields][env] == "A" {
elasticsearch {
...
index => "A"
}
} else {
elasticsearch {
...
index => "B"
}
}
}
/>

2

You can use interpolation in the index setting of the Elasticsearch output - meaning that some field or metadata field in your event holds part of the index name. So those events from "A" could have a field (add_field in beats input) called say index_suffix with value "a" and events from "B" have the same field with value "b".

index => "logstash-%{[index_suffix]}"

3

Thanks for your reply guyboertje.

Can you or someone please suggest how can we implement below scenario (master slave in logstash):

Create one A.conf for env A, create B.conf for env B with required filter and output details. And then will create one master.conf which will be having input configuration of logstash along with source of both env.conf over there. Is it possible to have this kind of configuration? and how can we achieve this?

4

If you point -f at a directory then logstash will concatente all the files in it to build the configuration. So you might have a set of files....

01input.conf
10env-qa.conf
20env-dev.conf

The input would be common and all the processing and output for qa/dev would have to be conditional based on tags or some other field.

5

Thanks Badger,

I tried that and got successful as well.

One more query

  1. Do we need different port in input tag to use multiple indexes? like below
    <
  • pipeline.id: my-pipeline_1
    path.config: "/etc/path/to/A.config"
  • pipeline.id: my-other-pipeline
    path.config: "/etc/different/path/B.cfg"

/>
and then

<

#A.cfg
input { beats { port => 5044 } }
filter { dissect { ... } }
output { elasticsearch { IP:port} 
index A}
#B.cfg
input { tcp { port => 5045 } }
filter { grok { ... } }
output { elasticsearch  { ... } 
index B}

/>

6

You cannot have two pipelines listening on the same port, so yes, you need to use different ports.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.