If I start the logstash service using service or systemctl, it starts there are no errors started successfully but logs dosen't get published there in kibana.
If i use " bin/logstash -f /etc/logstash/conf.d/logstash.conf " then logs are publishing to kibana.
I tried with this solution provided here Logstash service doesn't output logs, command line does . Then also no luck.
Any help would be appreciated
The content of logstash.yml, pipelines.yml and logstash.service as well as the logs of Logstash might help to find the problem.
HI Jenni,
Kindly find out the details below
path.data: /var/lib/logstash
pipeline.ordered: auto
path.logs: /var/log/logstash
path.config: "/etc/logstash/conf.d/*.conf"
[Unit]
Description=logstash
[Service]
Type=simple
User=logstash
Group=logstash
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/usr/share/logstash/config"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
TimeoutStopSec=infinity
[Install]
WantedBy=multi-user.target
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2020-07-09 11:51:32 UTC; 51min ago
Main PID: 20795 (java)
Tasks: 49 (limit: 4915)
CGroup: /system.slice/logstash.service
└─20795 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,406][WARN ][logstash.outputs.amazonelasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_versi
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,431][INFO ][logstash.outputs.amazonelasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::AmazonElasticSearch", :hosts=>["https://vpc-logerror-vilj5v
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,457][INFO ][logstash.outputs.amazonelasticsearch][main] Using mapping template from {:path=>nil}
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,497][INFO ][logstash.outputs.amazonelasticsearch][main] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60002, "settings"=>{"inde
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,548][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pip
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,041][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sinc
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,086][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,244][INFO ][filewatch.observingtail ][main][08f9a378b558a96db1555616c42d9ed88ef9b26655c2be176bc35c226b2e5572] START, creating Discoverer, Watch with file and sincedb co
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,248][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,718][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Also the files inside /etc/logstash/conf.d/
It should be input, filter and output parts in the config..
actually i did that also giving the /etc/logstash/conf.d/ in alogstash.service path setting and also in pipeline.yml but nothing changed
Your path.settings should be /etc/logstash if your logstash.yml is in that folder. If it doesn't work, please post the full log.
Hi jenni ,
As you suggested i changed and now
[Unit]
Description=logstash
[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
# When stopping, how long to wait before giving up and sending SIGKILL?
# Keep in mind that SIGKILL on a process can cause data loss.
TimeoutStopSec=infinity
[Install]
WantedBy=multi-user.target
And my logstash.yml is in the path /etc/logstash
logstash execution logs is as follows
logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2020-07-09 11:51:32 UTC; 27s ago
Main PID: 20795 (java)
Tasks: 50 (limit: 4915)
CGroup: /system.slice/logstash.service
└─20795 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,406][WARN ][logstash.outputs.amazonelasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_versi
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,431][INFO ][logstash.outputs.amazonelasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::AmazonElasticSearch", :hosts=>["https://vpc-logerror-vilj5v
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,457][INFO ][logstash.outputs.amazonelasticsearch][main] Using mapping template from {:path=>nil}
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,497][INFO ][logstash.outputs.amazonelasticsearch][main] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60002, "settings"=>{"inde
Jul 09 11:51:56 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:56,548][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pip
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,041][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sinc
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,086][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,244][INFO ][filewatch.observingtail ][main][08f9a378b558a96db1555616c42d9ed88ef9b26655c2be176bc35c226b2e5572] START, creating Discoverer, Watch with file and sincedb co
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,248][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Jul 09 11:51:58 ip-10-0-0-123 logstash[20795]: [2020-07-09T11:51:58,718][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
lines 1-18/18 (END)
Eventhough im not successful in publishing the logs to kibana.
But if i use this command bin/logstash -f /etc/logstash/conf.d/logstash.conf yes im able to push the logs but as a service not able to.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.