Logstash Service is active but still data is not passing to Elastic search

Elastic Stack Logstash
1

Hi, I am new to Logstash and Elastic search. I am currently going through documents and trying to fix a issue. Our logstash service is running active but data is not passing to Elasticsearch.

"logstash.version"=>"7.17.7" running in linux-x86_64 system.

status check command response.

systemctl status logstash.service

● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2023-05-02 05:22:26 CDT; 1 day 2h ago
 Main PID: 24647 (java)
   CGroup: /system.slice/logstash.service
           └─24647 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.e...

May 03 07:47:42  logstash[24647]: [2023-05-03T07:47:42,437][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
May 03 07:47:42  logstash[24647]: [2023-05-03T07:47:42,804][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
May 03 07:47:42  logstash[24647]: [2023-05-03T07:47:42,804][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
May 03 07:47:43  logstash[24647]: [2023-05-03T07:47:43,355][DEBUG][filewatch.sincedbcollection][main][daba0a116d1dbc345e97f78aa5ce67934a38b26e4efccabfe7df376857451c55] writi...t write = 15)
May 03 07:47:47  logstash[24647]: [2023-05-03T07:47:47,437][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
May 03 07:47:47  logstash[24647]: [2023-05-03T07:47:47,806][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
May 03 07:47:47  logstash[24647]: [2023-05-03T07:47:47,806][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
May 03 07:47:52  logstash[24647]: [2023-05-03T07:47:52,437][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
May 03 07:47:52  logstash[24647]: [2023-05-03T07:47:52,807][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
May 03 07:47:52  logstash[24647]: [2023-05-03T07:47:52,807][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}

In log file /var/log/logstash/logstash-plain.log there is no error shown.

But If I run it manually with this command data is passing to Elasticsearch /usr/share/logstash/bin/logstash -f "/etc/logstash/conf.d/wflserver_es.conf"

2

Hello, welcome to our community!

It might be worth checking the configuration file to see if there are any issues, as it doesn't seem to be a problem with the service itself... So, it's possible that there might be a configuration issue (syntax, missing fields, incorrect output etc) causing the data not to be passed to Elasticsearch.

Structure of a config file.

Configuration examples.

3

Hi Priscilla_parodi,

Thanks for taking your time and replying for this issue.

Below, I have pasted my config file wflserver_es.conf

input {
 file {
        path => ["log/wflserver.log"]
        start_position=>["beginning"]
        sincedb_path => ["/dev/null"]
        tags=>["twist"]
   }
}

filter {
    if "twist" in [tags]{

        grok {
            break_on_match=>true
            match=>{"message"=>"<log l='%{WORD:line}' d='%{INT:epoch}' cn='&PWPTM;' cv='&p;.%{WORD:workflow}.%{DATA:server},%{DATA:twistpid}.InputFiles.&m;' n='%{DATA:filename}' t='%{WORD:filetype}'><msg>Start with"}
            tag_on_failure=>[]
            add_tag=>"JobStatus"
            add_field=>["tool", "InputFiles"]
            add_field=>["action", "Start"]
            add_field=>["recieved_at", "%{@timestamp}"]
            add_field=>["recieved_from", "%{host}"]
        }

        grok {
            break_on_match=>true
            match=>{"message"=>"<log l='%{WORD:line}' d='%{INT:epoch}' cn='&PWPTM;' cv='&p;.%{WORD:workflow}.%{DATA:server},%{DATA:twistpid}.JobStatus.&m;' n='%{DATA:filename}'><msg>Job completed successfully</msg></log>"}
            tag_on_failure=>[]
            add_tag=>"JobStatus"
            add_field=>["action", "Done"]
            add_field=>["recieved_at", "%{@timestamp}"]
            add_field=>["recieved_from", "%{host}"]
        }

        grok {
            break_on_match=>true
            match=>{"message"=>"<log l='%{WORD:line}' d='%{INT:epoch}' cn='&PWPTM;' cv='&p;.%{WORD:workflow}.%{DATA:server},%{DATA:twistpid}.Defaults and Exceptions Check,%{WORD:setparam}.&m;' n='%{DATA:filename}'><msg>Rule %{INT:rule}: Expression: %{DATA:expression} failed. Rule skipped</msg></log>"}
            tag_on_failure=>[]
            add_tag=>"JobStatus"
            add_field=>["action", "Failed"]
            add_field=>["recieved_at", "%{@timestamp}"]
            add_field=>["recieved_from", "%{host}"]
        }
         if "JobStatus" not in [tags]{
            drop {}
        }

      date {
           match=>["epoch", "UNIX_MS"]
        }
    }

}
output {

    if "twist" in [tags] {
       # stdout { codec => rubydebug }
        elasticsearch {
          hosts=>["Myhost"]
          index=>"es_twist_log_index"                                                                                                                                                               
        }
    }
}

yesterday, after I debugged more. I found out it is happening due to permission issue. when I am running command manually, root user is considered. But logstash service is using logstash user and this user don't had the permission for /etc/logstash folder and /var/log/logstash folder. I have changed permission for this folder then service is started.

Today, morning once again service is stopped.

status response

# systemctl status -l logstash.service
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: failed (Result: signal) since Wed 2023-05-03 13:13:06 CDT; 13h ago
  Process: 22307 ExecStart=/usr/share/logstash/bin/logstash --path.settings /etc/logstash (code=killed, signal=KILL)
 Main PID: 22307 (code=killed, signal=KILL)

If I tried to restart it. I am getting error like below

Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2023-05-04 02:59:52.787 [main] runner - Starting Logstash {"logstash.version"=>"7.17.7", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.16+8 on 11.0.16+8 +indy +jit [linux-x86_64]"}
[INFO ] 2023-05-04 02:59:52.792 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2023-05-04 02:59:53.029 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2023-05-04 02:59:53.036 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[FATAL] 2023-05-04 02:59:53.038 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
4

Sure, thanks for sharing additional info.

Besides the warnings, if the Logstash service was running without issue before failing this morning, it's less likely that permissions are the issue now.

I would split it into two parts. One is the error that stopped Logstash, and the other is when you try to restart it.

When Logstash stopped, it was abruptly terminated, and it could be due to a system resource issue, such as running out of memory. If so, one thing you can try is to increase the amount of memory allocated to Logstash by modifying the jvm.options file. Look for the -Xms and -Xmx options and try increasing these values to see if it helps. It's also worth checking if there are any other processes or services that might be competing with Logstash for system resources.

When you try to restart it:
“Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.”

This could happen if there is already a running instance of Logstash, or if the previous instance (the first one) did not shut down properly. Or you can run multiple instances as suggested, changing the path.data setting.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.