Logstash not creating index in kibana

kiran80511 · August 9, 2021, 9:00am

This is my logstash plain log file


[2021-06-18T08:21:45,675][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2021-06-18T08:21:45,686][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.13.2", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[2021-06-18T08:21:46,858][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[2021-06-18T08:21:46,883][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2021-06-18T08:21:47,116][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-18T08:21:52,198][INFO ][logstash.runner          ] Logstash shut down.
[2021-06-18T08:21:52,216][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

This is my conf file

input {
udp {
port => 5044
type => "syslog"


}

}
filter{
      if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{USERNAME:hostname}%{DATA:syslog_program}:%{GREEDYDATA:syslog_message}" }
}
date {
        match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
     }

ruby {
        code => "
            fieldArray = event.get('syslog_message').split(' ');
            for field in fieldArray
                name = field.split('=')[0];
                value = field.split('=')[1];
                 if value =~ /[0-9]/
                   if value =~ /^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/
                      event.set(name, value)
                   else
                      if value =~ /[A-z]/
                        event.set(name, value)
                      else
                        event.set(name, value.to_i)
                      end
                   end
                else
                    event.set(name, value)

                end
            end
        "
    }
}

}
output {
  elasticsearch {
ilm_enabled => false
    hosts => ['https://elastic.sys:9200']
    cacert => '/etc/logstash/certs/ca.crt'
    user => 'logstash_writer'
    password => 'LiL14stC0M0BZw9iBAY9'
      index => "syslog"


  }
  stdout { codec => rubydebug }
}

Cad · August 9, 2021, 9:15am

kiran80511:

No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[2021-06-18T08:21:46,883][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.

Where is located your configuration file ?

Cad.

kiran80511 · August 9, 2021, 9:22am

[root@elastic conf.d]# ls
example.bak  syslog.bak  syslog.conf
[root@elastic conf.d]#

and path define in pipelines.yml

- pipeline.id: main
 path.config: "/etc/logstash/conf.d/*.conf"

Cad · August 9, 2021, 9:25am

What is the permissions of the sysolg.conf file ?

kiran80511 · August 9, 2021, 9:27am

-rw-r--r--. 1 root root 1323 Jul 22 14:36 syslog.conf

Cad · August 9, 2021, 9:50am

How did you run logstash ?
Did you try to restart the service ?

kiran80511 · August 9, 2021, 9:55am

sudo service logstash start
Yes I have already restart the service.
Logstash create index in kibana when I use HTTP but when I try ELK in HTTPS then not creating index in kibana.

kiran80511 · August 10, 2021, 10:38am

plz Any one can help ??

YuWatanabe · August 10, 2021, 12:59pm

Well . If it works with http but not https , then something should be logged in journal log.

Check log in elasticsearch. You may find some error related to certificate. Or check log for logstash.

kiran80511 · August 11, 2021, 6:29am

I have check logs of Elasticsearch ,there is no any error and in logstash log file last log is 18 june 2021 after that log is not created in logstash.
The log of logstash are

[2021-06-18T08:21:45,686][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.13.2", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[2021-06-18T08:21:46,858][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[2021-06-18T08:21:46,883][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2021-06-18T08:21:47,116][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-18T08:21:52,198][INFO ][logstash.runner          ] Logstash shut down.
[2021-06-18T08:21:52,216][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

Plz help me.

YuWatanabe · August 11, 2021, 7:51am

try

ls -l /etc/logstash/conf.d/syslog.conf

kiran80511 · August 11, 2021, 8:18am

-rw-r--r--. 1 root root 1323 Jul 22 14:36 /etc/logstash/conf.d/syslog.conf

YuWatanabe · August 11, 2021, 8:37am

Set this and restart logstash . See what happens.

- pipeline.id: main
 path.config: "/etc/logstash/conf.d/syslog.conf"

kiran80511 · August 12, 2021, 5:39am

I have changed as you said and restart the logstash but as I said earlier that after 18 june 2021 logs is not created in logstash log file so unable to check the logs.

YuWatanabe · August 12, 2021, 7:01am

I'd like to know if this message is cleared or not.

[2021-06-18T08:21:46,883][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.

If you are saying these log is not updated after the restart on Aug 12 , then something is wrong with your logstash installation. Next , you should check journal logs , e.g journalctl -b 0

[2021-06-18T08:21:45,675][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2021-06-18T08:21:45,686][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.13.2", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[2021-06-18T08:21:46,858][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[2021-06-18T08:21:46,883][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2021-06-18T08:21:47,116][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-18T08:21:52,198][INFO ][logstash.runner          ] Logstash shut down.
[2021-06-18T08:21:52,216][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

YuWatanabe · August 12, 2021, 7:04am

Also , you should restart logstash by sudo service logstash restart not sudo service logstash start. Stop then Start will be more solid instead of reloading the config.

sudo service logstash stop
sudo service logstash start

kiran80511 · August 13, 2021, 6:49am

Output of journalctl -b 0 -r

Aug 13 12:11:55 elastic.sys logstash[1508]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will 
Aug 13 12:11:54 elastic.sys logstash[1508]: Using bundled JDK: /usr/share/logstash/jdk
Aug 13 12:11:54 elastic.sys systemd[1]: Started logstash.
Aug 13 12:11:54 elastic.sys systemd[1]: Stopped logstash.
Aug 13 12:11:54 elastic.sys systemd[1]: logstash.service holdoff time over, scheduling restart.
Aug 13 12:11:54 elastic.sys systemd[1]: logstash.service failed.
Aug 13 12:11:54 elastic.sys systemd[1]: Unit logstash.service entered failed state.
Aug 13 12:11:54 elastic.sys systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
Aug 13 12:11:54 elastic.sys logstash[1445]: at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environmen
Aug 13 12:11:54 elastic.sys logstash[1445]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?]
Aug 13 12:11:54 elastic.sys logstash[1445]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?]
Aug 13 12:11:54 elastic.sys logstash[1445]: org.jruby.exceptions.SystemExit: (SystemExit) exit
Aug 13 12:11:54 elastic.sys logstash[1445]: [2021-08-13T12:11:54,637][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of 
Aug 13 12:11:54 elastic.sys logstash[1445]: bin/logstash --help
Aug 13 12:11:54 elastic.sys logstash[1445]: bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
Aug 13 12:11:54 elastic.sys logstash[1445]: bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
Aug 13 12:11:54 elastic.sys logstash[1445]: bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
Aug 13 12:11:54 elastic.sys logstash[1445]: bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE
Aug 13 12:11:54 elastic.sys logstash[1445]: bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
Aug 13 12:11:54 elastic.sys logstash[1445]: usage:
Aug 13 12:11:54 elastic.sys logstash[1445]: ERROR: Failed to read pipelines yaml file. Location: /etc/logstash/pipelines.yml

YuWatanabe · August 13, 2021, 7:52am

Are you sure you have your logstash installed properly ? You do not see these errors if it's installed correctly.

Aug 13 12:11:54 elastic.sys logstash[1445]: ERROR: Failed to read pipelines yaml file. Location: /etc/logstash/pipelines.yml

kiran80511 · August 18, 2021, 5:53am

yes I have installed properly but if error comes then I want to uninstall logstash completely.Please suggest how to uninstall logstash in centos 7?

Cad · August 18, 2021, 12:58pm

You have to put two space space before path.config. A single space also give me an error.

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"