I am trying to ingest data from an API using http_poller plugin in logstash. Tried using some methods, however was it is throwing error like this.
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-08-12 09:56:00.939 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [ERROR] 2020-08-12 09:56:04.472 [LogStash::Runner] http_poller - Unknown setting 'sincedb_path' for http_poller [FATAL] 2020-08-12 09:56:04.495 [LogStash::Runner] runner - The given configuration is invalid. Reason: Something is wrong with your configuration. [ERROR] 2020-08-12 09:56:04.507 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
If I don't include sincedb_path attribute then it runs smoothly but no index is created on Elasticsearch end (trying to view index in Kibana)
Kindly guide.
This is my configuration file.
input {
http_poller {
urls => {
urlname => "https://randomuser.me/api"
}
request_timeout => 20
schedule => { every => "20s" }
codec => "line"
sincedb_path => "/var/log/logstash"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "monitorwebsite"
}
stdout {
codec => rubydebug
}
}