Logstash.conf doesnt work when run as service but runs otherwise

Darshan_v_Shah · May 22, 2019, 5:54pm

I want to listen to beats on port 5000 and also push a local file to ES. In process of debugging, I have commented the rest and just running :

input { file { path=> "/home/ec2-user/URLTestOutput/out" codec => "json"} }

#mutate{ dissect {"mapping"=>" }  }

output {


if "/home/ec2-user/URLTestOutput/out" in [path]
 {
 stdout { codec => rubydebug }
 #elasticsearch {
 # hosts => ["http://x.x.x.x:9200"]
 # index => "index-url-status-logs-%{+YYYY.MM.dd}"
 }
#}
}

This when run with sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/url.conf & runs correctly but when run with sudo systemctl start logstash does not output anything.

Why so?

jpcarey · May 22, 2019, 11:13pm

I think the stdout will be redirected to journald when ran as a systemd unit. Check sudo journalctl -u logstash.service -f

Darshan_v_Shah · May 23, 2019, 7:16am

Hi jpcarey@,

Thanks for quick reply.

Unfortunately, I am still not being able to view any output in it
sudo journalctl -u logstash.service -f

-- Logs begin at Mon 2019-05-20 07:04:21 UTC. -- May 22 18:53:43 ip-172-31-15-198.us-east-2.compute.internal systemd[1]: Stopped logstash. May 23 06:57:36 ip-172-31-15-198.us-east-2.compute.internal systemd[1]: Started logstash. May 23 06:57:58 ip-172-31-15-198.us-east-2.compute.internal logstash[3164]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties May 23 06:57:59 ip-172-31-15-198.us-east-2.compute.internal logstash[3164]: [2019-05-23T06:57:59,633][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.7.2"} May 23 06:58:07 ip-172-31-15-198.us-east-2.compute.internal logstash[3164]: [2019-05-23T06:58:07,910][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} May 23 06:58:08 ip-172-31-15-198.us-east-2.compute.internal logstash[3164]: [2019-05-23T06:58:08,361][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_234b2f01f0dadf9e1ebf40e5956ec908", :path=>["/home/ec2-user/URLTestOutput/out"]} May 23 06:58:08 ip-172-31-15-198.us-east-2.compute.internal logstash[3164]: [2019-05-23T06:58:08,399][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3f9145c6 run>"} May 23 06:58:08 ip-172-31-15-198.us-east-2.compute.internal logstash[3164]: [2019-05-23T06:58:08,487][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} May 23 06:58:08 ip-172-31-15-198.us-east-2.compute.internal logstash[3164]: [2019-05-23T06:58:08,509][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections May 23 06:58:08 ip-172-31-15-198.us-east-2.compute.internal logstash[3164]: [2019-05-23T06:58:08,818][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

jpcarey · May 23, 2019, 3:56pm

Hmm. I also couldn't get stdout initially. I did a quick test with CentOS Linux release 7.6.1810 & rpm installed logstash 6.7.2.

logstash.conf

cat /etc/logstash/conf.d/logstash.conf
input {
  file {
    path => "/tmp/URLTestOutput"
    codec => "json"
    start_position => "beginning"
  }
}
output {
  stdout { codec => rubydebug }
}

After I forced systemd StandardOutput=journal, I could see the output in journalctl.

# /etc/systemd/system/logstash.service.d/override.conf
[Service]
StandardOutput=journal
StandardError=journal

(hint: sudo systemctl edit logstash.service to add the override listed above, then use sudo systemctl daemon-reload)

May 23 15:49:41 localhost.localdomain systemd[1]: Started logstash.
May 23 15:49:41 localhost.localdomain logstash[5091]: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
May 23 15:50:09 localhost.localdomain logstash[5091]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
May 23 15:50:11 localhost.localdomain logstash[5091]: [2019-05-23T15:50:11,081][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.7.2"}
May 23 15:50:25 localhost.localdomain logstash[5091]: [2019-05-23T15:50:25,685][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
May 23 15:50:27 localhost.localdomain logstash[5091]: [2019-05-23T15:50:27,122][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_35b0b3db3268e34aa137a3db09ad6995", :path=>["/tmp/URLTestOutput"]}
May 23 15:50:27 localhost.localdomain logstash[5091]: [2019-05-23T15:50:27,236][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1807ea48 run>"}
May 23 15:50:27 localhost.localdomain logstash[5091]: [2019-05-23T15:50:27,511][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
May 23 15:50:27 localhost.localdomain logstash[5091]: [2019-05-23T15:50:27,715][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
May 23 15:50:29 localhost.localdomain logstash[5091]: [2019-05-23T15:50:29,088][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
May 23 15:50:29 localhost.localdomain logstash[5091]: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
May 23 15:50:29 localhost.localdomain logstash[5091]: {
May 23 15:50:29 localhost.localdomain logstash[5091]: "@version" => "1",
May 23 15:50:29 localhost.localdomain logstash[5091]: "message" => "the first one",
May 23 15:50:29 localhost.localdomain logstash[5091]: "host" => "localhost.localdomain",
May 23 15:50:29 localhost.localdomain logstash[5091]: "path" => "/tmp/URLTestOutput",
May 23 15:50:29 localhost.localdomain logstash[5091]: "@timestamp" => 2019-05-23T15:50:29.074Z,
May 23 15:50:29 localhost.localdomain logstash[5091]: "foo" => "bar"
May 23 15:50:29 localhost.localdomain logstash[5091]: }
May 23 15:50:29 localhost.localdomain logstash[5091]: {
May 23 15:50:29 localhost.localdomain logstash[5091]: "@version" => "1",
May 23 15:50:29 localhost.localdomain logstash[5091]: "message" => "the second one",
May 23 15:50:29 localhost.localdomain logstash[5091]: "host" => "localhost.localdomain",
May 23 15:50:29 localhost.localdomain logstash[5091]: "path" => "/tmp/URLTestOutput",
May 23 15:50:29 localhost.localdomain logstash[5091]: "@timestamp" => 2019-05-23T15:50:29.223Z,
May 23 15:50:29 localhost.localdomain logstash[5091]: "foo" => "bar2"
May 23 15:50:29 localhost.localdomain logstash[5091]: }

logger · May 24, 2019, 11:10am

If you had first started logstash with sudo then all generated files will have root permissions.

When you start the service logstash wont be able to read and write on those files

Edit: so Check all permissions.
/var/lib/logstash
/usr/share/logstash (if you updated Plugins as root)
/etc/logstash

system · June 21, 2019, 11:10am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.