I have quite a few logstash conf files written out that have to be kicked off manually at the moment. Is there a way to cause them to run automatically?
I need to basically tie what I bring in to a dashboard that constantly updates without me having to go in the back end and manually run it.
Thank you
Can you provide more context about this? How are you running Logstash?
You can Logstash as a service, I would say that this is the recommended way to run it.
To run them right now I am going to /logstash/bin and running the logstash -f path/to.conf --path.settings /etc/elogstash to make it run. I have logstash running as a service and it has a schedule inside of the conf file but it does not start on its own.
Please provide context, you need to share your configuration, it is impossible to provide any help without it.
If you have a plugin that has a schedule, Logstash should executer it on the defined schedule, you just need to service to be running.
It is configured to run the input every 2 minutes, what is the issue here?
Is the service running? You said that you run it both manually and as a service, this can conflict with each other, so you should let the service running.
I have to run the configuration manually as it is not recognizing the schedule inside of the conf file. logstash is running as a service.
Do you have any log from Logstash that confirms that?
When you run it manually you run using which user?
It is confusing when you say that Logstash is running as a service, but you also run it manually, how this works? Do you stop the service? If logstash is correctly running as a service and you try to run it manually, it will not run and says that there is an instance already running.
Can you stop the service and start it again to get fresh logs and share them?
I dont stop the service I run the command i put in the original post to force it to run. If I let it run on its own it gets an error code when running sudo journalctl -u logstash -f
nd the reset counter just increments.
When I run it manually I run as root
This may cause a lot of issues when you try to run it as a service as Logstash creates some files that would now be owned by the root user and not the logstash user.
From your description I suspect that your Logstash service is not even running, probably because you run it as a root in the past and now the service cannot start.
Please, try to stop and start the service and share the logs you get.
I think that the log file and data path may now be owned by the root user and the service cannot start because the logstash user does not have permissions, you would need to fix those permissions.
All my conf files are set as logstash:logstash with 640 on them
What else would I need to check?
Probably the log file on /var/log/logstash/ and some data files in /var/lib/logstash.
This is described in the error that the service generates, can you restart the service to get fresh logs as asked?
You will need to check on /var/log/messages or /var/log/syslog for the issue that is making the service to not start.
Thge main error I am seeing at the moment is
[FATAL] 2025-03-05 08:03:00.440 [main] Logstash - Logstash stopped processing because of an error: (LoadError) no such file to load -- logstash/outputs/elasticsearch
This logs does not make any sense, is this the only log? Something is not right
Did you edit the logstash service?
Share the full content of the following files: /etc/logstash/logstash.yml and /etc/logstash/pipelines.yml.
Also, you need to share all logs or it is pretty complicated to troubleshoot with partial information.
Please run systemctl stop logstash than systemctl start logstash and check the /var/log/messages or /var/log/syslog for every mention to logstash and share them.
The pipelines.yml looks ok, but none of those pipelines are running if the Logstash service is not running.
It is not clear what caused the error, you need to share more log lines as asked
But as mentioned, running Logstash as root can mess up the permissions and make it not possible to run as a service without fixing it.
I recommend that you check the permissions for /var/lib/logstash, where everything needs to be owned by the logstash user and the logstash group and also /var/log/logstash, where the path needs to owned by the logstash user and the files inside as well.
After doing what you said in /var/lib/logstash I have seemed to get past every error of it trying to run on its own besides this one.
Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
You need to share the full log, the lines that came before and after, with partial lines is not possible to troubleshoot.
Please, when sharing log lines share the full log error.
I am unable to share my full logs so I will have to find an alternative.
Thank you
This may be again related to permissions, in this case permissions for the temp dir.
Check this part of the documentation.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.