Sending data from same source for different type of beat agents to logstash

Elastic Stack Logstash
1

I have got these beat agents installed on a machine :

  1. winlogbeat
  2. metricbeat

Trying to send data to same logstash instance. Wondering, how logstash would differentiate the data from these 2 inputs and send output to elasticsearch?

2

Hi Ravi,

Beats adds some fields of its own as well. Have a look at

https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-beat.html

3

So in the configuration for winlogbeat and metricbeat, I need to specify these?
eg:
for metricbeat
beat.name: metricbeat in metricbeat.yml
for winlogbeat
beat.name: winlogbeat in winlogbeat.yml

4

Not really. Beats does it for you.

Do you wish to send it to Logstash for any preprocessing?

In that case you can use tags in your FileBeat config.
https://www.elastic.co/guide/en/beats/filebeat/6.2/configuration-filebeat-options.html#_literal_tags_literal

As for metricbeat, that will send data directly to elasticsearch.

5

Yes, I want to send both metricbeat and winlogbeat to logstash.

6

Hi Ravi,

You can use the beat.name field to apply conditional logic and process events in Logstash.

And this is correct. You will have to specify the beat name.

7

Thank you. I would also need logstash to send output to ES with different index for both beats.
How the configuration for logstash would like in the .conf file for input and output?

8

Hi Nachiket,

Could you please update on the below?

Thanks,
Ravi

9

Hi Ravi,

You could use the conditional to separate the two streams.

if [beat][name] == "xyz" {
 mutate {
  add_field => { "indice" => "xyz" }
 }
}
else {
 mutate {
  add_field => {"indice" => "abc"}
 }
}

You can then use the indice field in your elasticsearch output.

10

Thank you Nachiket.

so the logstash.conf should look like this?
input {
beats {
port => 5044
}
}

if [beat][name] == "metricbeat" {
mutate {
add_field => { "indice" => "metricbeat" }
}
}
else {
mutate {
add_field => {"indice" => "winlogbeat"}
}
}

output {
elasticsearch {
hosts => ["172.31.1.10:9200"]
index => "metricbeat-%{+YYYY.MM.dd}"
}
}

output {
elasticsearch {
hosts => ["172.31.1.10:9200"]
index => "winlogbeat-%{+YYYY.MM.dd}"
}
}

11

Hi Nachiket,

I am using filter after input in logstash.conf. Does this looks good?
input {
beats {
port => 5044
}
}

filter {
if [beat][name] == "metricbeat" {
mutate {
add_field => { "indice" => "metricbeat"}
}
}
else {
mutate {
add_field => {"indice" => "winlogbeat"}
}
}
}

output {
elasticsearch {
hosts => ["172.31.1.10:9200"]
index => "metricbeat-%{+YYYY.MM.dd}"
}
}

output {
elasticsearch {
hosts => ["172.31.1.10:9200"]
index => "winlogbeat-%{+YYYY.MM.dd}"
}
}

Regards,
Ravi

12

This config will create a single index for both filebeat and metricbeat. Is that what was intended?

To create two indices, please use the indice variable we created in the elasticsearch output.

output {
  elasticsearch {
    hosts => ["172.31.1.10:9200"]
    index => "%{indice}-%{+YYYY.MM.dd}"
  }
}
13

No, I need to create 2 indices, one for each beat type.

Here is my updated config:

input {
beats {
port => 5044
}
}

filter {
if [beat][name] == "metricbeat" {
mutate {
add_field => { "indice" => "metricbeat"}
}
}
else {
mutate {
add_field => {"indice" => "winlogbeat"}
}
}
}

output {
elasticsearch {
hosts => ["172.31.1.10:9200"]
index => "%{indice}-%{+YYYY.MM.dd}"
}
}

output {
elasticsearch {
hosts => ["172.31.1.10:9200"]
index => "%{indice}-%{+YYYY.MM.dd}"
}
}

14

As all events now contain a parameter with the index prefix, you should only have a single elasticsearch output.

15

Hi Christian,

I want to see separate ES output for both beat types. Is this possible?

Thanks,
Ravi

16

Why? If you want that you have to put conditionals around the outputs as well in order to avoid duplicates.

17

I am sending different types of beat data, so would like to see ES output with different index.

18

The way you have set it up now you can send data to different indices using a single output, which probably is more efficient. Why does this not work for you??

19

I am ok if it can send data to different indices using a single output. Then my config should be like this?

input {
beats {
port => 5044
}
}

filter {
if [beat][name] == "metricbeat" {
mutate {
add_field => { "indice" => "metricbeat"}
}
}
else {
mutate {
add_field => {"indice" => "winlogbeat"}
}
}
}

output {
elasticsearch {
hosts => ["172.31.1.10:9200"]
}
}

20

Your output block should still look like this, but you only need one.