Hi,
I am using logstash to push files from the ec2 instance (/media/ephemeral0/logs/yarn/*.gz) to S3 folders. This instance is part of a spark cluster. However the same configuration (for logstash) works fine on the master node but does not work on the slave nodes.
i have checked the perms on the folder and the files and they have read perms for logstash user.
this is my logstash.conf file content
input{
file {
path => "/media/ephemeral0/logs/yarn/.gz"
sincedb_path => "/usr/share/logstash/sincedb1"
start_position => "beginning"
codec => "gzip_lines"
}
file {
path => "/media/ephemeral0/logs/spark/.log"
sincedb_path => "/usr/share/logstash/sincedb2"
start_position => "beginning"
}
file {
path => "/media/ebs2/logs/yarn/*.log"
sincedb_path => "/usr/share/logstash/sincedb3"
start_position => "beginning"
}
}
output {
s3{
bucket => "Bucketname"
size_file => 20480
prefix => "spark/%{host}"
tags => ["spark"]
region => "us-west-2"
upload_queue_size => 10
time_file => 1
encoding => "gzip"
validate_credentials_on_root_bucket => false
canned_acl => "authenticated-read"
}
}
I checked the sincedb1 file and see the following entries... not sure if 0 means no offsetbyte being read.
8782113 1 10537472 0
8782117 1 10537472 0
8782127 1 10537472 0
8782455 1 10537472 0
8782512 1 10537472 0
No Errors seen in /var/log/logstash/logstash-plain.log
below is the output...
[2017-10-04T19:07:03,572][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent.
[2017-10-04T19:07:03,584][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2017-10-04T19:07:47,797][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-10-04T19:07:47,800][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-10-04T19:07:51,430][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000}
[2017-10-04T19:07:51,659][INFO ][logstash.pipeline ] Pipeline main started
[2017-10-04T19:07:51,727][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
can anyone please help me identify the issue.
updated logging level to debug and the output of the log is below
[2017-10-06T04:49:57,984][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100123.log.gz: old size 0, new size 21263
[2017-10-06T04:49:57,984][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100319.log.gz: old size 0, new size 15057
[2017-10-06T04:49:57,984][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100111.log.gz: old size 0, new size 20857
[2017-10-06T04:49:57,985][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100422.log.gz: old size 0, new size 20555
[2017-10-06T04:49:57,985][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100509.log.gz: old size 0, new size 22055
[2017-10-06T04:49:57,986][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100201.log.gz: old size 0, new size 20822
[2017-10-06T04:49:57,986][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100304.log.gz: old size 0, new size 14753
[2017-10-06T04:49:57,986][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100410.log.gz: old size 0, new size 19653
[2017-10-06T04:49:57,987][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100419.log.gz: old size 0, new size 26496
[2017-10-06T04:49:57,987][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100114.log.gz: old size 0, new size 20923
[2017-10-06T04:49:57,987][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100208.log.gz: old size 0, new size 21505
[2017-10-06T04:49:57,988][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100521.log.gz: old size 0, new size 21938
[2017-10-06T04:49:57,988][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100110.log.gz: old size 0, new size 21901
[2017-10-06T04:49:57,988][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100322.log.gz: old size 0, new size 19900
[2017-10-06T04:49:57,989][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100106.log.gz: old size 0, new size 21218
[2017-10-06T04:49:57,989][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100107.log.gz: old size 0, new size 21227
[2017-10-06T04:49:57,989][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100408.log.gz: old size 0, new size 19734
[2017-10-06T04:49:57,990][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100402.log.gz: old size 0, new size 20054
[2017-10-06T04:49:57,990][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100313.log.gz: old size 0, new size 14524
[2017-10-06T04:49:57,991][DEBUG][logstash.inputs.file ] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100215.log.gz: old size 0, new size 13928
still nothing changed.
Thanking in advance
Naresh