I'm new to logstash, and I've been trying to get a docker container working with logstash in order to get data from the stock market for a project. Unfortunately, primarily due to the fact that I have no experience with this tool, I have been unable to get any info from any website (trying to get info from DJIA | Dow Jones Industrial Average Overview | MarketWatch but I've tried a few others).
Following are some of my configuration files and the response after running, but I'm not sure at this point what I'm doing right/wrong, or where the problem lies.
dow.conf:
input { #http_poller { # urls => { # myresource => "https://www.marketwatch.com/investing/index/djia" # } # } #request_timeout => 60 #interval => 60 #codec => "json" # set this if the response is json formatted http { host => "https://www.marketwatch.com/investing/index/djia" id => "dow_id" } } output { file { path => /usr/share/logstash/data/DOW.log #codec => line { format => "custom format: %{message}"} } }
logstash.yml and pipelines.yml:
http.host: "https://www.marketwatch.com/investing/index/djia"
dockerfile:
FROM logstash:7.7.0 #OS is centos #needed for permissions USER root #install python3, pip3 RUN yum -y install epel-release && yum clean all RUN yum -y install python3-pip && yum clean all #pip install necessary libraries RUN pip3 install python3-logstash RUN bin/logstash-plugin install logstash-input-http_poller RUN bin/logstash-plugin install logstash-input-http #transfer necessary files to image RUN rm -f /usr/share/logstash/pipeline/logstash.conf ADD pipeline/ /usr/share/logstash/pipeline/ ADD config/ /usr/share/logstash/config/ #create necessary directories and files with needed permissions RUN touch /usr/share/logstash/data/DOW.log && chmod 777 /usr/share/logstash/data/DOW.log
response after 'logstash -f dow.conf':
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long) WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties [2020-06-13T17:16:04,100][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"} [2020-06-13T17:16:04,150][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"} [2020-06-13T17:16:04,849][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-06-13T17:16:04,863][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.7.0"} [2020-06-13T17:16:04,893][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"224578fc-fa55-4310-ac03-7b61fd02f7a8", :path=>"/usr/share/logstash/data/uuid"} [2020-06-13T17:16:06,452][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", [A-Za-z0-9_-], '\"', \"'\", [A-Za-z_], \"-\", [0-9], \"[\", \"{\" at line 19, column 12 (byte 387) after output {\r\n file {\r\n path => ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]} warning: thread "Api Webserver" terminated with exception (report_on_exception is true): SocketError: initialize: name or service not known initialize at org/jruby/ext/socket/RubyTCPServer.java:124 new at org/jruby/RubyIO.java:876 add_tcp_listener at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.3-java/lib/puma/binder.rb:229 add_tcp_listener at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/forwardable.rb:229 start_webserver at /usr/share/logstash/logstash-core/lib/logstash/webserver.rb:104 run at /usr/share/logstash/logstash-core/lib/logstash/webserver.rb:60 each at org/jruby/RubyRange.java:526 each_with_index at org/jruby/RubyEnumerable.java:1258 run at /usr/share/logstash/logstash-core/lib/logstash/webserver.rb:55 start_webserver at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:393 [2020-06-13T17:16:06,817][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SocketError) initialize: name or service not known