# Files not being pushed - even though the pipeline has been successfully created **URL:** \ **Category:** Logstash\ **Created:** [October 4, 2017, 7:06pm UTC](https://discuss.elastic.co/t/files-not-being-pushed-even-though-the-pipeline-has-been-successfully-created/102750 "2017-10-04T19:06:07Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![naresh.komma](https://avatars.discourse-cdn.com/v4/letter/n/958977/32.png) [@naresh.komma](https://discuss.elastic.co/u/naresh.komma)\ **Post date:** [October 4, 2017, 7:06pm UTC](https://discuss.elastic.co/t/files-not-being-pushed-even-though-the-pipeline-has-been-successfully-created/102750/1 "2017-10-04T19:06:08Z")
Hi, I am using logstash to push files from the ec2 instance (/media/ephemeral0/logs/yarn/\*.gz) to S3 folders. This instance is part of a spark cluster. However the same configuration (for logstash) works fine on the master node but does not work on the slave nodes. i have checked the perms on the folder and the files and they have read perms for logstash user. this is my logstash.conf file content input{ file { path =\> "/media/ephemeral0/logs/yarn/_.gz" sincedb\_path =\> "/usr/share/logstash/sincedb1" start\_position =\> "beginning" codec =\> "gzip\_lines" } file { path =\> "/media/ephemeral0/logs/spark/_.log" sincedb\_path =\> "/usr/share/logstash/sincedb2" start\_position =\> "beginning" } file { path =\> "/media/ebs2/logs/yarn/\*.log" sincedb\_path =\> "/usr/share/logstash/sincedb3" start\_position =\> "beginning" } } output { s3{ bucket =\> "Bucketname" size\_file =\> 20480 prefix =\> "spark/%{host}" tags =\> ["spark"] region =\> "us-west-2" upload\_queue\_size =\> 10 time\_file =\> 1 encoding =\> "gzip" validate\_credentials\_on\_root\_bucket =\> false canned\_acl =\> "authenticated-read" } } I checked the sincedb1 file and see the following entries... not sure if 0 means no offsetbyte being read. 8782113 1 10537472 0 8782117 1 10537472 0 8782127 1 10537472 0 8782455 1 10537472 0 8782512 1 10537472 0 No Errors seen in /var/log/logstash/logstash-plain.log below is the output... [2017-10-04T19:07:03,572][WARN][logstash.runner] SIGTERM received. Shutting down the agent. [2017-10-04T19:07:03,584][WARN][logstash.agent] stopping pipeline {:id=\>"main"} [2017-10-04T19:07:47,797][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} [2017-10-04T19:07:47,800][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} [2017-10-04T19:07:51,430][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>8, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>1000} [2017-10-04T19:07:51,659][INFO][logstash.pipeline] Pipeline main started [2017-10-04T19:07:51,727][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} can anyone please help me identify the issue. updated logging level to debug and the output of the log is below [2017-10-06T04:49:57,984][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100123.log.gz: old size 0, new size 21263 [2017-10-06T04:49:57,984][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100319.log.gz: old size 0, new size 15057 [2017-10-06T04:49:57,984][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100111.log.gz: old size 0, new size 20857 [2017-10-06T04:49:57,985][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100422.log.gz: old size 0, new size 20555 [2017-10-06T04:49:57,985][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100509.log.gz: old size 0, new size 22055 [2017-10-06T04:49:57,986][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100201.log.gz: old size 0, new size 20822 [2017-10-06T04:49:57,986][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100304.log.gz: old size 0, new size 14753 [2017-10-06T04:49:57,986][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100410.log.gz: old size 0, new size 19653 [2017-10-06T04:49:57,987][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100419.log.gz: old size 0, new size 26496 [2017-10-06T04:49:57,987][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100114.log.gz: old size 0, new size 20923 [2017-10-06T04:49:57,987][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100208.log.gz: old size 0, new size 21505 [2017-10-06T04:49:57,988][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100521.log.gz: old size 0, new size 21938 [2017-10-06T04:49:57,988][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100110.log.gz: old size 0, new size 21901 [2017-10-06T04:49:57,988][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100322.log.gz: old size 0, new size 19900 [2017-10-06T04:49:57,989][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100106.log.gz: old size 0, new size 21218 [2017-10-06T04:49:57,989][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100107.log.gz: old size 0, new size 21227 [2017-10-06T04:49:57,989][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100408.log.gz: old size 0, new size 19734 [2017-10-06T04:49:57,990][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100402.log.gz: old size 0, new size 20054 [2017-10-06T04:49:57,990][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100313.log.gz: old size 0, new size 14524 [2017-10-06T04:49:57,991][DEBUG][logstash.inputs.file] each: file grew: /media/ephemeral0/logs/yarn/yarn-yarn-nodemanager-ip-10-20-19-30.log.2017100215.log.gz: old size 0, new size 13928 still nothing changed. Thanking in advance Naresh ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 1, 2017, 7:06pm UTC](https://discuss.elastic.co/t/files-not-being-pushed-even-though-the-pipeline-has-been-successfully-created/102750/2 "2017-11-01T19:06:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.