Getting error while running logstash with multiple beats data

Elastic Stack Logstash
1

Currently I am sending metricbeat and winlogbeat data from same source to same logstash instance A. Now I would like to send filebeat data from another source to same logstash instance A.

When I restart logstash, am getting error
Error- [FATAL][logstash.runner ] Logstash could not b
e started because there is already another instance using the configured data di
rectory. If you wish to run multiple instances, you must change the "path.data"
setting

Here is my logstash configuration:
input {
beats {
port => 5044
}
}

filter {
if [beat][name] == "metricbeat" {
mutate {
add_field => { "indice" => "metricbeats"}
}
}
if [beat][name] == "winlogbeat" {
mutate {
add_field => { "indice" => "winbeats"}
}
}
if [beat][name] == "filebeat" {
mutate {
add_field => { "indice" => "filebeats"}
}
}
}

output {
elasticsearch
{
hosts => ["172.31.1.10:9200"]
index => "%{indice}-%{+YYYY.MM.dd}"
}
}

Please advise.

2

The error message indicates that you're trying to run two concurrent instances of Logstash. How are you restarting Logstash? After you get the error message, is there an old Logstash instance running?

3

I have one logstash instance running.

I use the below command once I make the changes to configuration file.

sudo systemctl restart logstash.service

cd /usr/share/logstash
sudo bin/logstash -f /etc/logstash/conf.d/logstash.conf

4

Also, if logstash configuration looks good?

5

I... don't understand. You run systemctl restart logstash.service but you also start Logstash in your terminal with bin/logstash -f ...?

6

I did restart but I found no changes to the output. so then I did bin/logstash -f ...

7

Well, if Logstash is running you can't start another instance with the same data directory. Stop the daemonized Logstash instance or configure either instance to use another data directory.

8

I am not sure if I understood. Do I need to stop the logstash service and then update the logstash.conf file?

9

No. Just don't run two Logstash instances at the same time. If you have to do that they need to use different data directories.

10

I am currently running only one logstash instance and I want to run only one instance of logstash.

Initially we were sending data for metricbeat and winlogbeat to this logstash instance. Later we decided to also send data for filebeat to the same logstash instance.

Wondering if I want to update the logstash configuration file to include filebeat with separate index, can I do it without stopping the logstash service and if I can use this configuration?

input {
beats {
port => 5044
}
}

filter {
if [beat][name] == "metricbeat" {
mutate {
add_field => { "indice" => "metricbeats"}
}
}
if [beat][name] == "winlogbeat" {
mutate {
add_field => { "indice" => "winbeats"}
}
}
if [beat][name] == "filebeat" {
mutate {
add_field => { "indice" => "filebeats"}
}
}
}

output {
elasticsearch
{
hosts => ["172.31.1.10:9200"]
index => "%{indice}-%{+YYYY.MM.dd}"
}
}

11

Wondering if I want to update the logstash configuration file to include filebeat with separate index, can I do it without stopping the logstash service

Yes, if you have automatic config reloading enabled it'll get picked up automatically. Otherwise you need to restart Logstash.

and if I can use this configuration?

That configuration looks reasonable.

12

I tried sending metricbeat and winlogbeat data from source A to logstash instance B and filebeat data from source C to logstash instance B. I cannot see the winlogbeat and filebeat data sent to elasticsearch through logstash instance B.

Here is my configuration on logstash instance B:

input {
beats {
port => 5044
}
}

filter {
if [beat][name] == "metricbeat" {
mutate {
add_field => { "indice" => "metricbeat"}
}
}
if [beat][name] == "winlogbeat" {
mutate {
add_field => { "indice" => "winlogbeat"}
}
}
if [beat][name] == "filebeat" {
mutate {
add_field => { "indice" => "filebeat"}
}
}
}

output {
elasticsearch
{
hosts => ["172.31.1.10:9200"]
index => "%{indice}-%{+YYYY.MM.dd}"
}
}

what shall I be doing to fix this?

13

Now I can see the output with index for metricbeat and filebeat as
metricbeat-2018.09.03
filebeat-2018.09.03
the above looks fine.

But for winlogbeat, I can see output with index as
%{indice}-2018.09.03

Any idea why I am getting the output as this?

14

What does the [beat][name] field contain in the Winlogbeat case? The indice field is clearly not being set.

15

I am trying to filter the data sending from winlogbeat. Shall I use [type] == wineventlog?

16

Use whatever is unique to the Winlogbeat messages. I don't know what your events looks like.

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.