# Everyone faced this issue? **URL:** \ **Category:** Logstash\ **Created:** [October 29, 2019, 2:22am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576 "2019-10-29T02:22:52Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 29, 2019, 2:22am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/1 "2019-10-29T02:22:53Z")
[2019-10-28T01:43:36,493][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} [2019-10-28T01:43:41,249][INFO][logstash.runner] Logstash shut down. [2019-10-28T01:44:13,990][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.4.1"} [2019-10-28T01:44:15,636][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, { at line 1, column 7 (byte 7) after input ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge\_state'"]} ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 29, 2019, 4:12am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/2 "2019-10-29T04:12:36Z")
You have an error in your logstash pipeline config. The error states that. > [@pbona](#): > > exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, { at line 1, column 7 (byte 7) after input ", : Of course we can not help unless you provide it. If you do please properly format it using the `` format button. ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 29, 2019, 11:36pm UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/3 "2019-10-29T23:36:03Z")
@stephenb `exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 1, column 7 (byte 7) after input ", :` Inside the logstash.yml file ? ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 12:09am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/4 "2019-10-30T00:09:48Z")
``` [root@elastic logstash]# cat conf.d/logstash.conf input { beats { port => 5044 ssl => true ssl_key => '/etc/logstash/config/certs/logstash.pkcs8.key' ssl_certificate => '/etc/logstash/config/certs/logstash.crt' } } output { elasticsearch { hosts => ["https://172.16.5.199:9200"] cacert => '/etc/logstash/config/certs/ca.crt' user => 'logstash_writer' password => ************ } } ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 12:11am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/5 "2019-10-30T00:11:03Z")
``` [root@elastic logstash]# cat pipelines.yml # This file is where you define your pipelines. You can define multiple. # For more information on multiple pipelines, see the documentation: # https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html - pipeline.id: main path.config: "/etc/logstash/conf.d/*.conf" ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 30, 2019, 12:15am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/6 "2019-10-30T00:15:50Z")
This feels like a mistake where you have somehow pointed path.config at your logstash.yml. Rather than the details of the logstash.conf it might be more helpful to post details of the overall configuration. ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 12:19am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/7 "2019-10-30T00:19:45Z")
``` [root@elastic logstash]# cat logstash.yml # Settings file in YAML # # Settings can be specified either in hierarchical form, e.g.: # # pipeline: # batch: # size: 125 # delay: 5 # # Or as flat keys: # # pipeline.batch.size: 125 # pipeline.batch.delay: 5 # # ------------ Node identity ------------ # # Use a descriptive name for the node: # node.name: logstash.local # # If omitted the node name will default to the machine's host name # # ------------ Data path ------------------ # # Which directory should be used by logstash and its plugins # for any persistent needs. Defaults to LOGSTASH_HOME/data # path.data: /var/lib/logstash # # ------------ Pipeline Settings -------------- # # The ID of the pipeline. # # pipeline.id: main # # Set the number of workers that will, in parallel, execute the filters+outputs # stage of the pipeline. # # This defaults to the number of the host's CPU cores. # # pipeline.workers: 2 # # How many events to retrieve from inputs before sending to filters+workers # # pipeline.batch.size: 125 # # How long to wait in milliseconds while polling for the next event # before dispatching an undersized batch to filters+outputs # # pipeline.batch.delay: 50 # # Force Logstash to exit during shutdown even if there are still inflight # events in memory. By default, logstash will refuse to quit until all # received events have been pushed to the outputs. # # WARNING: enabling this can lead to data loss during shutdown # # pipeline.unsafe_shutdown: false # # ------------ Pipeline Configuration Settings -------------- # # Where to fetch the pipeline configuration for the main pipeline # path.config: /etc/logstash/conf.d/*.conf # # Pipeline configuration string for the main pipeline # # config.string: # # At startup, test if the configuration is valid and exit (dry run) # # config.test_and_exit: false # # Periodically check if the configuration has changed and reload the pipeline # This can also be triggered manually through the SIGHUP signal # # config.reload.automatic: false # # How often to check if the pipeline configuration has changed (in seconds) # # config.reload.interval: 3s # # Show fully compiled configuration as debug log message # NOTE: --log.level must be 'debug' # # config.debug: false # # When enabled, process escaped characters such as \n and \" in strings in the # pipeline configuration files. # # config.support_escapes: false # ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 12:20am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/8 "2019-10-30T00:20:05Z")
``` # ------------ Module Settings --------------- # Define modules here. Modules definitions must be defined as an array. # The simple way to see this is to prepend each `name` with a `-`, and keep # all associated variables under the `name` they are associated with, and # above the next, like this: # # modules: # - name: MODULE_NAME # var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE # var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE # var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE # var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE # # Module variable names must be in the format of # # var.PLUGIN_TYPE.PLUGIN_NAME.KEY # # modules: # # ------------ Cloud Settings --------------- # Define Elastic Cloud settings here. # Format of cloud.id is a base64 value e.g. dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy # and it may have an label prefix e.g. staging:dXMtZ... # This will overwrite 'var.elasticsearch.hosts' and 'var.kibana.host' # cloud.id: # # Format of cloud.auth is: : # This is optional # If supplied this will overwrite 'var.elasticsearch.username' and 'var.elasticsearch.password' # If supplied this will overwrite 'var.kibana.username' and 'var.kibana.password' # cloud.auth: elastic: # # ------------ Queuing Settings -------------- # # Internal queuing model, "memory" for legacy in-memory based queuing and # "persisted" for disk-based acked queueing. Defaults is memory # # queue.type: memory # # If using queue.type: persisted, the directory path where the data files will be stored. # Default is path.data/queue # # path.queue: # # If using queue.type: persisted, the page data files size. The queue data consists of # append-only data files separated into pages. Default is 64mb # # queue.page_capacity: 64mb # # If using queue.type: persisted, the maximum number of unread events in the queue. # Default is 0 (unlimited) # # queue.max_events: 0 # # If using queue.type: persisted, the total capacity of the queue in number of bytes. # If you would like more unacked events to be buffered in Logstash, you can increase the # capacity using this setting. Please make sure your disk drive has capacity greater than # the size specified here. If both max_bytes and max_events are specified, Logstash will pick # whichever criteria is reached first # Default is 1024mb or 1gb # # queue.max_bytes: 1024mb # # If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint # Default is 1024, 0 for unlimited # # queue.checkpoint.acks: 1024 # # If using queue.type: persisted, the maximum number of written events before forcing a checkpoint # Default is 1024, 0 for unlimited # # queue.checkpoint.writes: 1024 # # If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page # Default is 1000, 0 for no periodic checkpoint. # # queue.checkpoint.interval: 1000 # ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 12:21am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/9 "2019-10-30T00:21:03Z")
``` # ------------ Dead-Letter Queue Settings -------------- # Flag to turn on dead-letter queue. # # dead_letter_queue.enable: false # If using dead_letter_queue.enable: true, the maximum size of each dead letter queue. Entries # will be dropped if they would increase the size of the dead letter queue beyond this setting. # Default is 1024mb # dead_letter_queue.max_bytes: 1024mb # If using dead_letter_queue.enable: true, the directory path where the data files will be stored. # Default is path.data/dead_letter_queue # # path.dead_letter_queue: # # ------------ Metrics Settings -------------- # # Bind address for the metrics REST endpoint # # http.host: "127.0.0.1" # # Bind port for the metrics REST endpoint, this option also accept a range # (9600-9700) and logstash will pick up the first available ports. # # http.port: 9600-9700 # # ------------ Debugging Settings -------------- # # Options for log.level: # * fatal # * error # * warn # * info (default) # * debug # * trace # # log.level: info path.logs: /var/log/logstash # # ------------ Other Settings -------------- # # Where to find custom plugins # path.plugins: [] # # ------------ X-Pack Settings (not applicable for OSS build)-------------- # # X-Pack Monitoring # https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.username: logstash_system xpack.monitoring.elasticsearch.password: ' ************' xpack.monitoring.elasticsearch.hosts: ['https://172.16.5.199:9200'] xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/config/certs/ca.crt ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 30, 2019, 2:38am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/10 "2019-10-30T02:38:51Z")
how are you starting logstash? you can test your pipeline file with the following and yes by eye it looks good. `./bin/logstash -t -f ./conf.d/logstash.conf` I tend to agree with badger the error says it is failing on the first line which seems to indicate it is reading the wrong file. you could also set `log.level: debug` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 2:44am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/11 "2019-10-30T02:44:29Z")
> [@stephenb](#): > > log.level: debug I have set log.level: debug ``` [root@elastic logstash]# systemctl status logstash -l ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-10-29 22:42:27 EDT; 56s ago Main PID: 18982 (java) CGroup: /system.slice/logstash.service └─18982 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.9.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.24.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash Oct 29 22:42:27 elastic.nbc.org.kh.local systemd[1]: logstash.service: main process exited, code=exited, status=143/n/a Oct 29 22:42:27 elastic.nbc.org.kh.local systemd[1]: Stopped logstash. Oct 29 22:42:27 elastic.nbc.org.kh.local systemd[1]: Unit logstash.service entered failed state. Oct 29 22:42:27 elastic.nbc.org.kh.local systemd[1]: logstash.service failed. Oct 29 22:42:27 elastic.nbc.org.kh.local systemd[1]: Started logstash. Oct 29 22:42:27 elastic.nbc.org.kh.local logstash[18982]: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Oct 29 22:43:19 elastic.nbc.org.kh.local logstash[18982]: Thread.exclusive is deprecated, use Thread::Mutex Oct 29 22:43:24 elastic.nbc.org.kh.local logstash[18982]: Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 30, 2019, 3:06am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/12 "2019-10-30T03:06:34Z")
> [@pbona](#): > > /usr/share/logstash/logs Whats is in the log file? Also you can start manually [via command line](https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html) to test if that works then it may have to do with the `systemctl` start Interesting I just noticed something.... You systemctl output says sending logs to `/usr/share/logstash/logs` but in your config it says `path.logs: /var/log/logstash` makes me think that the `systemctl` is not using the correct config. ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 3:30am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/13 "2019-10-30T03:30:19Z")
``` [root@elastic logstash]# /usr/share/logstash/bin/logstash -t -f ./conf.d/logstash.conf OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Thread.exclusive is deprecated, use Thread::Mutex WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN] 2019-10-29 22:46:26.124 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2019-10-29 22:46:28.208 [LogStash::Runner] Reflections - Reflections took 65 ms to scan 1 urls, producing 20 keys and 40 values Configuration OK [INFO] 2019-10-29 22:46:29.217 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 3:40am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/14 "2019-10-30T03:40:30Z")
@stephenb I have changed the path.logs: /usr/share/logstash/logs but it is stilll not working ``` [root@elastic logstash]# systemctl restart logstash [root@elastic logstash]# systemctl status logstash -l ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-10-29 23:35:26 EDT; 6s ago Main PID: 21172 (java) CGroup: /system.slice/logstash.service └─21172 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.9.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.24.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash Oct 29 23:35:26 elastic.nbc.org.kh.local systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE Oct 29 23:35:26 elastic.nbc.org.kh.local systemd[1]: Stopped logstash. Oct 29 23:35:26 elastic.nbc.org.kh.local systemd[1]: Unit logstash.service entered failed state. Oct 29 23:35:26 elastic.nbc.org.kh.local systemd[1]: logstash.service failed. Oct 29 23:35:26 elastic.nbc.org.kh.local systemd[1]: Started logstash. Oct 29 23:35:26 elastic.nbc.org.kh.local logstash[21172]: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 30, 2019, 3:41am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/15 "2019-10-30T03:41:01Z")
You need to add `-path.settings /etc/logstash` `/usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/logstash.conf --path.settings /etc/logstash` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 3:44am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/16 "2019-10-30T03:44:07Z")
``` [root@elastic logstash]# /usr/share/logstash/bin/logstash -t -f ./conf.d/logstash.conf --path.settings /etc/logstash OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Thread.exclusive is deprecated, use Thread::Mutex Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties [2019-10-29T23:43:30,715][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2019-10-29T23:43:32,578][INFO][org.reflections.Reflections] Reflections took 95 ms to scan 1 urls, producing 20 keys and 40 values Configuration OK [2019-10-29T23:43:33,411][INFO][logstash.runner] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 3:45am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/17 "2019-10-30T03:45:33Z")
> [@stephenb](#): > > path.settings /etc/logstash Add `path.settings /etc/logstash` to logstash.conf or logstash.yml ? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 30, 2019, 3:45am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/18 "2019-10-30T03:45:40Z")
Ok now run without the `-t` and it should run. This is looking like you `systemctl` start up script is not correct If it works cat `cat /etc/systemd/system/logstash.service` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 3:48am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/19 "2019-10-30T03:48:55Z")
``` [root@elastic logstash]# /usr/share/logstash/bin/logstash -f ./conf.d/logstash.conf --path.settings /etc/logstash OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Thread.exclusive is deprecated, use Thread::Mutex Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties [2019-10-29T23:47:33,677][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2019-10-29T23:47:33,713][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.4.1"} [2019-10-29T23:47:37,115][INFO][org.reflections.Reflections] Reflections took 44 ms to scan 1 urls, producing 20 keys and 40 values [2019-10-29T23:47:39,627][INFO][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_writer:xxxxxx@172.16.5.199:9200/]}} [2019-10-29T23:47:40,158][WARN][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@172.16.5.199:9200/"} [2019-10-29T23:47:40,249][INFO][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7} [2019-10-29T23:47:40,252][WARN][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [2019-10-29T23:47:40,328][INFO][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://172.16.5.199:9200"]} [2019-10-29T23:47:40,521][WARN][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [2019-10-29T23:47:40,552][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, :thread=>"#"} [2019-10-29T23:47:40,787][INFO][logstash.outputs.elasticsearch][main] Using default mapping template [2019-10-29T23:47:41,167][INFO][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} warning: thread "Ruby-0-Thread-5: :1" terminated with exception (report_on_exception is true): LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'https://172.16.5.199:9200/logstash' perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80 ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [October 30, 2019, 3:49am UTC](https://discuss.elastic.co/t/everyone-faced-this-issue/205576/20 "2019-10-30T03:49:10Z")
``` perform_request_to_url at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291 perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278 with_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373 perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277 Pool at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285 exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341 rollover_alias_exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359 maybe_create_rollover_alias at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91 setup_ilm at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10 setup_after_successful_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52 [2019-10-29T23:47:41,534][INFO][logstash.inputs.beats][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"} [2019-10-29T23:47:41,612][FATAL][logstash.runner] An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `rollover_alias_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91:in `maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:in `setup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in `block in setup_after_successful_connection'"]} [2019-10-29T23:47:41,748][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"} [2019-10-29T23:47:41,847][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2019-10-29T23:47:41,958][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit ``` [Next page](https://discuss.elastic.co/t/everyone-faced-this-issue/205576.md?page=2)