Http_poller input plugin issue


#1

I am trying to create a dashboard for our devops tools, e.g

  • GIT commits/pull requests
  • TeamCity Build Status
  • Sonar metrics
    Etc

I am trying to fetch these data via Logstash

I am using the input plugin, http_poller

Below is my config file

# contents of logstash\bin\logstash.config

# input
input {
  http_poller {
	urls => {
		'uat3' => "https://<myTeamCityServer>/app/rest/server/"
	}
	interval => 60
	codec => "json"
	truststore => "C:\DATA\Logstash\certfile\RootFile.jks" 
	#ssl_certificate_validation=>false
	}   
}
output {
  stdout {
    codec => rubydebug
  }
}

And this is the corresponding Console log


#2
 C:\DATA\Logstash\logstash-5.2.2\bin>logstash -f logstashAPI.config --debug
 ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
 Could not find log4j2 configuration at path /DATA/Logstash/logstash-5.2.2/config/log4j2.properties. Using default config which logs to console
 10:54:06.668 [LogStash::Runner] DEBUG logstash.runner - -------- Logstash Settings (* means modified) ---------
 10:54:06.684 [LogStash::Runner] DEBUG logstash.runner - node.name: "MyNode"
 10:54:06.684 [LogStash::Runner] DEBUG logstash.runner - *path.config: "logstashAPI.config"
 10:54:06.684 [LogStash::Runner] DEBUG logstash.runner - path.data: "C:/DATA/Logstash/logstash-5.2.2/data"
 10:54:06.684 [LogStash::Runner] DEBUG logstash.runner - config.test_and_exit: false
 10:54:06.684 [LogStash::Runner] DEBUG logstash.runner - config.reload.automatic: false
 10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - config.reload.interval: 3
 10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - metric.collect: true
 10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - pipeline.id: "main"
 10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - pipeline.workers: 4
 10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - pipeline.output.workers: 1
 10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - pipeline.batch.size: 125
 10:54:06.715 [LogStash::Runner] DEBUG logstash.runner - pipeline.batch.delay: 5
 10:54:06.715 [LogStash::Runner] DEBUG logstash.runner - pipeline.unsafe_shutdown: false
 10:54:06.731 [LogStash::Runner] DEBUG logstash.runner - path.plugins: []
 10:54:06.731 [LogStash::Runner] DEBUG logstash.runner - config.debug: false
 10:54:06.731 [LogStash::Runner] DEBUG logstash.runner - *log.level: "debug" (default: "info")
 10:54:06.731 [LogStash::Runner] DEBUG logstash.runner - version: false
 10:54:06.731 [LogStash::Runner] DEBUG logstash.runner - help: false
 10:54:06.746 [LogStash::Runner] DEBUG logstash.runner - log.format: "plain"
 10:54:06.746 [LogStash::Runner] DEBUG logstash.runner - http.host: "xxxx.0.0.1"
 10:54:06.746 [LogStash::Runner] DEBUG logstash.runner - http.port: yyyy..yyyy
 10:54:06.746 [LogStash::Runner] DEBUG logstash.runner - http.environment: "production"
 10:54:06.746 [LogStash::Runner] DEBUG logstash.runner - queue.type: "memory"
 10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.page_capacity: 262144000
 10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.max_bytes: 1073741824
 10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.max_events: 0
 10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.checkpoint.acks: 1024
 10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.checkpoint.writes: 1024
 10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.checkpoint.interval: 1000
 10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - slowlog.threshold.warn: -1
 10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - slowlog.threshold.info: -1
 10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - slowlog.threshold.debug: -1
 10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - slowlog.threshold.trace: -1
 10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - path.queue: "C:/DATA/Logstash/logstash-5.2.2/data/queue"
 10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - path.settings: "C:/DATA/Logstash/logstash-5.2.2/config"
 10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - path.logs: "C:/DATA/Logstash/logstash-5.2.2/logs"

#3
 10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - --------------- Logstash Settings -------------------
 10:54:06.824 [LogStash::Runner] DEBUG logstash.agent - Agent: Configuring metric collection
 10:54:06.840 [LogStash::Runner] DEBUG logstash.instrument.periodicpoller.os - PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
 10:54:06.887 [LogStash::Runner] DEBUG logstash.instrument.periodicpoller.jvm - PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
 10:54:06.996 [LogStash::Runner] DEBUG logstash.instrument.periodicpoller.persistentqueue - PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
 10:54:07.059 [LogStash::Runner] DEBUG logstash.agent - Reading config file {:config_file=>"C:/DATA/Logstash/logstash-5.2.2/bin/logstashAPI.config"}
 10:54:07.731 [LogStash::Runner] DEBUG logstash.plugins.registry - On demand adding plugin to the registry {:name=>"http_poller", :type=>"input", :class=>LogStash::Inputs::HTTP_Poller}
 10:54:07.746 [LogStash::Runner] WARN  logstash.inputs.http_poller - You are using a deprecated config setting "interval" set in http_poller. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future.  If you have any
 questions about this, please visit the #logstash channel on freenode irc. {:name=>"interval", :plugin=><LogStash::Inputs::HTTP_Poller urls=>{"uat3"=>"https://MyTeamCityServer/app/rest/server/"}, interval=>60, codec=>"json", truststore=>"C:\\DATA\\Log
 stash\\certfile\\RootFile.jks", id=>"44d33c5fb43af50fe180e0c136aa3d44f5bc0436-1">}
 10:54:07.777 [LogStash::Runner] DEBUG logstash.plugins.registry - On demand adding plugin to the registry {:name=>"json", :type=>"codec", :class=>LogStash::Codecs::JSON}
 10:54:07.793 [LogStash::Runner] DEBUG logstash.codecs.json - config LogStash::Codecs::JSON/@id = "json_bd65a57c-9608-456d-a521-9fcf559ec37e"
 10:54:07.793 [LogStash::Runner] DEBUG logstash.codecs.json - config LogStash::Codecs::JSON/@enable_metric = true
 10:54:07.793 [LogStash::Runner] DEBUG logstash.codecs.json - config LogStash::Codecs::JSON/@charset = "UTF-8"
 10:54:07.793 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@urls = {"uat3"=>"https://MyTeamCityServer/app/rest/server/"}
 10:54:07.809 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@interval = 60
 10:54:07.809 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@codec = <LogStash::Codecs::JSON id=>"json_bd65a57c-9608-456d-a521-9fcf559ec37e", enable_metric=>true, charset=>"UTF-8">
 10:54:07.809 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@truststore = "C:\\DATA\\Logstash\\certfile\\RootFile.jks"
 10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@id = "44d33c5fb43af50fe180e0c136aa3d44f5bc0436-1"
 10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@enable_metric = true
 10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@add_field = {}
 10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@request_timeout = 60
 10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@socket_timeout = 10
 10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@connect_timeout = 10
 10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@follow_redirects = true
 10:54:07.840 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@pool_max = 50
 10:54:07.840 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@pool_max_per_route = 25
 10:54:07.840 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@keepalive = true
 10:54:07.840 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@automatic_retries = 1
 10:54:07.856 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@retry_non_idempotent = false
 10:54:08.684 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@validate_after_inactivity = 200
 10:54:08.684 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@ssl_certificate_validation = true
 10:54:08.715 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@keystore_type = "JKS"
 10:54:08.715 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@truststore_type = "JKS"
 10:54:08.715 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@cookies = true
 10:54:08.715 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@metadata_target = "@metadata"
 10:54:08.731 [LogStash::Runner] DEBUG logstash.plugins.registry - On demand adding plugin to the registry {:name=>"stdout", :type=>"output", :class=>LogStash::Outputs::Stdout}
 10:54:08.762 [LogStash::Runner] DEBUG logstash.plugins.registry - On demand adding plugin to the registry {:name=>"rubydebug", :type=>"codec", :class=>LogStash::Codecs::RubyDebug}
 10:54:08.762 [LogStash::Runner] DEBUG logstash.codecs.rubydebug - config LogStash::Codecs::RubyDebug/@id = "rubydebug_1aab7cc6-a81e-412a-95ba-6bc1c7db1d02"
 10:54:08.777 [LogStash::Runner] DEBUG logstash.codecs.rubydebug - config LogStash::Codecs::RubyDebug/@enable_metric = true
 10:54:08.777 [LogStash::Runner] DEBUG logstash.codecs.rubydebug - config LogStash::Codecs::RubyDebug/@metadata = false
 10:54:08.934 [LogStash::Runner] DEBUG logstash.outputs.stdout - config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::RubyDebug id=>"rubydebug_1aab7cc6-a81e-412a-95ba-6bc1c7db1d02", enable_metric=>true, metadata=>false>
 10:54:08.949 [LogStash::Runner] DEBUG logstash.outputs.stdout - config LogStash::Outputs::Stdout/@id = "44d33c5fb43af50fe180e0c136aa3d44f5bc0436-2"
 10:54:08.949 [LogStash::Runner] DEBUG logstash.outputs.stdout - config LogStash::Outputs::Stdout/@enable_metric = true
 10:54:08.965 [LogStash::Runner] DEBUG logstash.outputs.stdout - config LogStash::Outputs::Stdout/@workers = 1

#4
 10:54:08.981 [Ruby-0-Thread-3: C:/DATA/Logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:22] DEBUG logstash.agent - starting agent
 10:54:08.996 [Ruby-0-Thread-3: C:/DATA/Logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:22] DEBUG logstash.agent - starting pipeline {:id=>"main"}
 10:54:09.074 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
 10:54:09.152 [[main]-pipeline-manager] INFO  logstash.inputs.http_poller - Registering http_poller Input {:type=>nil, :urls=>{"uat3"=>"https://MyTeamCityServer/app/rest/server/"}, :interval=>60, :schedule=>nil, :timeout=>nil}
 10:54:09.152 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
 10:54:09.152 [[main]<http_poller] DEBUG logstash.inputs.http_poller - Fetching URL {:name=>"uat3", :url=>[:get, "https://MyTeamCityServer/app/rest/server/"]}
 10:54:09.184 [[main]<http_poller] ERROR logstash.pipeline - A plugin had an unrecoverable error. Will restart this plugin.
   Plugin: <LogStash::Inputs::HTTP_Poller urls=>{"uat3"=>"https://MyTeamCityServer/app/rest/server/"}, interval=>60, codec=><LogStash::Codecs::JSON id=>"json_bd65a57c-9608-456d-a521-9fcf559ec37e", enable_metric=>true, charset=>"UTF-8">, truststore=>"C
 :\\DATA\\Logstash\\certfile\\RootFile.jks", id=>"44d33c5fb43af50fe180e0c136aa3d44f5bc0436-1", enable_metric=>true, request_timeout=>60, socket_timeout=>10, connect_timeout=>10, follow_redirects=>true, pool_max=>50, pool_max_per_route=>25, keepalive=>true, automa
 tic_retries=>1, retry_non_idempotent=>false, validate_after_inactivity=>200, ssl_certificate_validation=>true, keystore_type=>"JKS", truststore_type=>"JKS", cookies=>true, metadata_target=>"@metadata">
   Error: undefined method `value' for nil:NilClass
   Exception: NoMethodError

#5

Bumping the thread


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.