Logstash not working for multiple instances

Elastic Stack Logstash
1

Hi All,

I am getting below while trying to run second instance of logstash-

Error- [FATAL][logstash.runner ] Logstash could not b
e started because there is already another instance using the configured data di
rectory. If you wish to run multiple instances, you must change the "path.data"
setting.

Please suggest.
Version-Logstash 5.4.2

Regards,
Prateek

2

Hello @Prateek_Kshtriya:

Might I ask which is your use case and why do you need to run several instances of Logstash on the same server?

ANyway as the error mentions if you create a new conf file where you use a different path.data, you shoudl be able to do so (please keep in mind though that only one will be able to listen for *beats traffic, unless you specify a different port for each instance.

3

Hey Javier,

I am trying to post the perfmon counter data from windows cluster machines. For that I have created different config files, which I am calling on command line using "logstash -f xyz.conf".

Regards,
Prateek

4

Please also suggest how do we define Path.data for each different instance as I have not well versed with the current version and had not faced such issue with earlier versions.

Regards,
Prateek

5

Hello @Prateek_Kshtriya:
Unless you want to store each windows cluster data on a different ES server, you should be able to do this using a single instance of LS .
As I said before, you should be able to run several instances as long as each config file has a different path.data configuration.

If this were not your case, please share some (at least two) config files(probably uploading them to pastebin or so, hiding/changing any sensitive data they might have.

Regards

6

PFB the sample config file, the counter and path are different in the other configs corresponding to machines but data is getting indexed at same index name.

input {
file {
path => "//remoteMachine/PerfmonData/*/DataCollector01.csv"

    start_position => "beginning"
  }

}

filter {
csv {
columns => ["IST", "DiskTime", "AvailableMBytes", "ProcessorTime", "PhysicalDiskIdleTime", "IdleTime"]
separator => ","
add_field => { "MachineName" => "RabbitMQ" }
}
grok {

        match => { "message" => "%{DATESTAMP:timestamp}" }
    }
	date {
        match => [ "timestamp", "MM/dd/yy HH:mm:ss" ]
    }
		
mutate {

convert => [ "DiskTime", "integer" ]
convert => [ "AvailableMBytes", "integer" ]
convert => [ "ProcessorTime", "integer" ]
convert => [ "PhysicalDiskIdleTime", "integer" ]
convert => [ "IdleTime", "integer" ]
}

}

output {
elasticsearch{
action => "index"
index => "resourcemetrics-%{+YYYY.MM.dd}"
}
}

7

PFB the sample config file, the counter and path are different in the other configs corresponding to machines but data is getting indexed at same index name.

Yes, because configuration files aren't independent. Logstash has a single event pipeline that processes all events. If you don't want all outputs to receive all events you have to add conditionals.

The path.data setting that the original error message talks about can e.g. be overridden with the --path.data command line option.

8

There are different config files corresponding to each cluster machine, it is just they are posting to same index.
This is something which was getting executed with earlier versions ,so is there a change which is implemented in the current version to have different Path.data for having instance to post at same index.

The independent configs are supposed to perform below activity-

Config 1--Pick the data from machine 1 and post it to Index 1
Config 2-- Pick the data from machine 2 and post it to Index 1

Regards,
Prateek

9

This is something which was getting executed with earlier versions ,so is there a change which is implemented in the current version to have different Path.data

Yes, it was a change in Logstash 5.2 or thereabouts.

for having instance to post at same index.

Whether the instances are posting to the same ES index has nothing to do with this.

10

It worked using --path.data at command line. But I am not getting "How two different config files are not independent ?"

Regards,
Prateek

11

But I am not getting "How two different config files are not independent ?"

Having two files, a.conf and b.conf, in /etc/logstash/conf.d is exactly equivalent to concatenating them inte a single file and letting that be the only file in /etc/logstash/conf.d. All filters and outputs defined in all configuration files are applied to all events coming from all configuration files.

1 Like
12

In earlier versions it was like each individual config files are treated as independent but with latest version of Logstash reads multiple config files as only one file.

So if it is required to read different config files as independent then --path.data is required to be passed explicitly in command line while executing the config

Regards,
Prateek

13

In earlier versions it was like each individual config files are treated as independent but with latest version of Logstash reads multiple config files as only one file.

No. Logstash has always worked as I described but the requirement to not share data directory is new.

So if it is required to read different config files as independent then --path.data is required to be passed explicitly in command line while executing the config

Yes, two instances of Logstash can't share the same data directory.

14

Now it is clear to me . Thank you .

Regards,
Prateek

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.