I created two different droplets in Digitalocen and install nginx and filebeat on those two droplets.I configured my filebeat.yml file as below:
VM1 filebeat.yml:_
filebeat.prospectors:
- type: log
paths:- /var/log/nginx/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/ .yml
reload.enabled: true
- /var/log/nginx/*.log
setup.template.settings:
index.number_of_shards: 3
output.logstash:
hosts: ["134.209.153.109:5045"]
VM2 filebeat.yml:_
filebeat.prospectors:
- type: log
paths:- /var/log/nginx/ .log
filebeat.config.modules:
path: ${path.config}/modules.d/ .yml
reload.enabled: true
- /var/log/nginx/ .log
setup.template.settings:
index.number_of_shards: 3
output.logstash:
hosts: ["134.209.153.109:5044"]
"After this I installed ELK in another Droplet and configured Logstash like below"
My Logstash vm1.conf file:
input {
beats {
port => 5045
}
}
filter {
if [fileset][module] == "nginx" {
if [fileset][name] == "access" {
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}""] }
remove_field => "message"
}
output {
elasticsearch {
hosts => ["134.209.153.109:9200"]
manage_template => false
index => "vm1_nginx-1"
}
}
My Logstash vm2.conf file:
input {
beats {
port => 5044
}
}
filter {
if [fileset][module] == "nginx" {
if [fileset][name] == "access" {
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}""] }
remove_field => "message"
}
output {
elasticsearch {
hosts => ["134.209.153.109:9200"]
manage_template => false
index => "vm2_nginx-1"
}
}
I configured my file and whhen i run logstash i am getting the error like below
[FATAL] 2020-03-09 05:19:25.972 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[ERROR] 2020-03-09 05:19:25.988 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Can you please help me out how to solve the problem.How to change path.setting ( " If you wish to run multiple instances, you must change the "path.data" setting")
Thanks in advance