[root@elastic logstash]# systemctl start logstash
[root@elastic logstash]# systemctl status logstash
â— logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-11-04 02:37:04 EST; 49s ago
Main PID: 23874 (java)
CGroup: /system.slice/logstash.service
└─23874 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupa...
Nov 04 02:37:04 elastic.nbc.org.kh.local systemd[1]: logstash.service holdoff time over, scheduling restart.
Nov 04 02:37:04 elastic.nbc.org.kh.local systemd[1]: Stopped logstash.
Nov 04 02:37:04 elastic.nbc.org.kh.local systemd[1]: Started logstash.
Nov 04 02:37:04 elastic.nbc.org.kh.local logstash[23874]: Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC ...ease.
Nov 04 02:37:28 elastic.nbc.org.kh.local logstash[23874]: WARNING: An illegal reflective access operation has occurred
Nov 04 02:37:28 elastic.nbc.org.kh.local logstash[23874]: WARNING: Illegal reflective access by com.headius.backport9.modules.M...or.fd
Nov 04 02:37:28 elastic.nbc.org.kh.local logstash[23874]: WARNING: Please consider reporting this to the maintainers of com.hea...dules
Nov 04 02:37:28 elastic.nbc.org.kh.local logstash[23874]: WARNING: Use --illegal-access=warn to enable warnings of further ille...tions
Nov 04 02:37:28 elastic.nbc.org.kh.local logstash[23874]: WARNING: All illegal access operations will be denied in a future release
Hint: Some lines were ellipsized, use -l to show in full.
[root@elastic logstash]# less /var/log/logstash/logstash-plain.log
[2019-11-04T01:23:42,909][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2019-11-04T01:23:42,960][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2019-11-04T01:23:43,814][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.4.2"}
[2019-11-04T01:23:43,851][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"a6b62bd5-ca53-4339-936a-ef78e0065f7a", :path=>"/var/lib/logstash/uuid"}
[2019-11-04T01:23:44,961][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[2019-11-04T01:23:44,983][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2019-11-04T01:23:45,506][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-11-04T01:23:50,232][INFO ][logstash.runner ] Logstash shut down.
[2019-11-04T01:25:45,886][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.4.2"}
[2019-11-04T01:25:47,109][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[2019-11-04T01:25:47,118][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2019-11-04T01:25:47,609][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-11-04T01:25:52,553][INFO ][logstash.runner ] Logstash shut down.
[2019-11-04T01:26:43,565][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.4.2"}
[2019-11-04T01:26:44,831][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[2019-11-04T01:26:44,840][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2019-11-04T01:26:45,352][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-11-04T01:26:50,320][INFO ][logstash.runner ] Logstash shut down.
[2019-11-04T01:27:26,683][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.4.2"}
[2019-11-04T01:27:27,906][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[2019-11-04T01:27:27,936][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2019-11-04T01:27:28,419][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-11-04T01:27:33,406][INFO ][logstash.runner ] Logstash shut down.
[2019-11-04T01:28:08,659][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.4.2"}
[2019-11-04T01:28:09,772][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[root@elastic logstash]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-11-04 20:59:07.850 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-04 20:59:07.859 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.4.2"}
[INFO ] 2019-11-04 20:59:11.032 [Converge PipelineAction::Create<main>] Reflections - Reflections took 75 ms to scan 1 urls, producing 20 keys and 40 values
[INFO ] 2019-11-04 20:59:12.804 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_writer:xxxxxx@172.16.5.199:9200/]}}
[WARN ] 2019-11-04 20:59:13.801 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@172.16.5.199:9200/"}
[INFO ] 2019-11-04 20:59:14.148 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2019-11-04 20:59:14.151 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2019-11-04 20:59:14.205 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://172.16.5.199:9200"]}
[WARN ] 2019-11-04 20:59:14.426 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2019-11-04 20:59:14.570 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, :thread=>"#<Thread:0x185e0faa run>"}
[INFO ] 2019-11-04 20:59:14.944 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template
[INFO ] 2019-11-04 20:59:15.167 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[INFO ] 2019-11-04 20:59:15.639 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
warning: thread "Ruby-0-Thread-5: :1" terminated with exception (report_on_exception is true):
LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'https://172.16.5.199:9200/logstash'
perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80
perform_request_to_url at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291
perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278
with_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373
perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277
Pool at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285
exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341
rollover_alias_exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359
maybe_create_rollover_alias at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91
setup_ilm at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10
setup_after_successful_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52
403 is a Forbidden status. If you followed that blog post you generated passwords for various user accounts. You should have one of those configured on your elasticsearch output. It is not configured correctly.
[root@elastic ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-11-06 03:48:24.903 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-06 03:48:24.930 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.4.1"}
[INFO ] 2019-11-06 03:48:28.108 [Converge PipelineAction::Create<main>] Reflections - Reflections took 76 ms to scan 1 urls, producing 20 keys and 40 values
[INFO ] 2019-11-06 03:48:30.055 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_writer:xxxxxx@172.16.5.199:9200/]}}
[WARN ] 2019-11-06 03:48:30.876 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@172.16.5.199:9200/"}
[INFO ] 2019-11-06 03:48:31.448 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2019-11-06 03:48:31.451 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2019-11-06 03:48:31.485 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://172.16.5.199:9200"]}
[INFO ] 2019-11-06 03:48:31.590 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template
[WARN ] 2019-11-06 03:48:31.615 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2019-11-06 03:48:31.645 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, :thread=>"#<Thread:0x5651f062 run>"}
[root@elastic ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-11-07 02:08:42.589 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-07 02:08:42.599 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.4.1"}
[INFO ] 2019-11-07 02:08:45.772 [Converge PipelineAction::Create<main>] Reflections - Reflections took 69 ms to scan 1 urls, producing 20 keys and 40 values
[INFO ] 2019-11-07 02:08:47.501 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_writer:xxxxxx@172.16.5.199:9200/]}}
[WARN ] 2019-11-07 02:08:48.808 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@172.16.5.199:9200/"}
[INFO ] 2019-11-07 02:08:49.095 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2019-11-07 02:08:49.097 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2019-11-07 02:08:49.173 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://172.16.5.199:9200"]}
[WARN ] 2019-11-07 02:08:49.309 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2019-11-07 02:08:49.342 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, :thread=>"#<Thread:0x3fc12196 run>"}
[INFO ] 2019-11-07 02:08:49.370 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template
[INFO ] 2019-11-07 02:08:49.708 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
warning: thread "Ruby-0-Thread-5: :1" terminated with exception (report_on_exception is true):
LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'https://172.16.5.199:9200/logstash'
perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80
perform_request_to_url at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291
perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278
with_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373
perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277
Pool at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285
exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341
rollover_alias_exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359
maybe_create_rollover_alias at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91
setup_ilm at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10
setup_after_successful_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.