Logstash manages to send data to elasticsearch only in debugging mode

Hello everyone,

I am currently trying a basic test setup with apache logs on logstash, but I have a problem, my data is received on kibana/elasticsearch only when I run the following command:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/apache.conf

And so the problem is that if I stop this command and logstash is running normally, I have no errors but no data appears, and another weird thing, when I'm in debugging mode, it seems to me from the tutorials I've seen that I'm supposed to see the data in real time, but that's not the case.

If anyone has an idea.. Btw I'm on version 8.7

Here is my logstash configuration:

root@srv-elk:/etc/logstash/conf.d# cat /etc/logstash/conf.d/apache.conf | grep ^[^#]
input {
        file {
                 path => "/var/log/apache2/access.log"
                 #start_position => "beginning"
                 #ignore_older => 0
                 #sincedb_path => "NUL"
                 #delimiter => "\r"
filter {
        grok {
                #patterns_dir => [ "/etc/logstash/patterns.d"]
                match => [ "message" , "%{COMBINEDAPACHELOG}" ]
        date {
                match => ["timestamp","dd/MMM/yyyy:HH:mm:ss Z" ]
output {
    elasticsearch {
         hosts => ["https://localhost"]
         user => "elastic"
         password => "ozPpuZ0=ypPMUHJLkobf"
         index => "apache-%{+YYYY.MM.dd}"
         ssl => true
         cacert => "/http_ca.crt"

And for elasticsearch:

root@srv-elk:/etc/logstash/conf.d# cat /etc/elasticsearch/elasticsearch.yml | grep ^[^#]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
discovery.type: single-node
network.host: ""
http.port: 9200
discovery.seed_hosts: [""]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
  enabled: true
  keystore.path: certs/http.p12
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
http.host: "localhost"

When you run LS form the command line, it will run, in your case, as root user as a process. You can run as a background process with & at the end of command. Recommended mode is a service mode for continuous running.

Follow next steps:

  1. Change in elasticsearch.yml and restart:
    network.host: [ _site_ , _local_]
    Or just set to Leandro explained here.
    Restart elasticsearch.
  2. Edit apache.conf on Linux should be: sincedb_path => "/dev/null" - this means sincedb is runtime mode, keep log read tracking until restart process, not permanent on disk. The disk mode is used in the production mode, when you need to track logs read.
  3. Check log permissions, root user might take ownership, so run: chown -R logstash:logstash /var/log/logstash/
  4. Run as the service: sudo systemctl start logstash.service
    If is not enabled: systemctl enable logstash.service and most likely: systemctl daemon-reload

Thank you for your answer, unfortunately it did not work, always the same problem. No errors anywhere, the only message I didn't mention is this one (last line) :

[INFO ] 2023-04-18 11:21:49.216 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:536: warning: already initialized constant Manticore::Client::StringEntity

But it didn't seem very useful to me.

Seeing you talk about permission made me think I wasn't sure if logstash had read rights to the apache log file, I gave it permissions:

sudo usermod -aG adm logstash

And it works ! ^^'

I had already made this command but I uninstalled logstash from my server in the meantime and suddenly it no longer had the rights!

Sorry it was a noob error x)
I always find it weird that it didn't get any error about that...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.