shirm
October 23, 2018, 10:26am
1
I am trying to create a dashboard for our devops tools, e.g
GIT commits/pull requests
TeamCity Build Status
Sonar metrics
I am trying to fetch these data via Logstash
I am using the input plugin, http_poller
Below is my config file
# contents of logstash\bin\logstash.config
# input
input {
http_poller {
urls => {
'uat3' => "https://<myTeamCityServer>/app/rest/server/"
}
interval => 60
codec => "json"
truststore => "C:\DATA\Logstash\certfile\RootFile.jks"
#ssl_certificate_validation=>false
}
}
output {
stdout {
codec => rubydebug
}
}
And this is the corresponding Console log
shirm
October 23, 2018, 10:26am
2
C:\DATA\Logstash\logstash-5.2.2\bin>logstash -f logstashAPI.config --debug
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Could not find log4j2 configuration at path /DATA/Logstash/logstash-5.2.2/config/log4j2.properties. Using default config which logs to console
10:54:06.668 [LogStash::Runner] DEBUG logstash.runner - -------- Logstash Settings (* means modified) ---------
10:54:06.684 [LogStash::Runner] DEBUG logstash.runner - node.name: "MyNode"
10:54:06.684 [LogStash::Runner] DEBUG logstash.runner - *path.config: "logstashAPI.config"
10:54:06.684 [LogStash::Runner] DEBUG logstash.runner - path.data: "C:/DATA/Logstash/logstash-5.2.2/data"
10:54:06.684 [LogStash::Runner] DEBUG logstash.runner - config.test_and_exit: false
10:54:06.684 [LogStash::Runner] DEBUG logstash.runner - config.reload.automatic: false
10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - config.reload.interval: 3
10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - metric.collect: true
10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - pipeline.id: "main"
10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - pipeline.workers: 4
10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - pipeline.output.workers: 1
10:54:06.699 [LogStash::Runner] DEBUG logstash.runner - pipeline.batch.size: 125
10:54:06.715 [LogStash::Runner] DEBUG logstash.runner - pipeline.batch.delay: 5
10:54:06.715 [LogStash::Runner] DEBUG logstash.runner - pipeline.unsafe_shutdown: false
10:54:06.731 [LogStash::Runner] DEBUG logstash.runner - path.plugins: []
10:54:06.731 [LogStash::Runner] DEBUG logstash.runner - config.debug: false
10:54:06.731 [LogStash::Runner] DEBUG logstash.runner - *log.level: "debug" (default: "info")
10:54:06.731 [LogStash::Runner] DEBUG logstash.runner - version: false
10:54:06.731 [LogStash::Runner] DEBUG logstash.runner - help: false
10:54:06.746 [LogStash::Runner] DEBUG logstash.runner - log.format: "plain"
10:54:06.746 [LogStash::Runner] DEBUG logstash.runner - http.host: "xxxx.0.0.1"
10:54:06.746 [LogStash::Runner] DEBUG logstash.runner - http.port: yyyy..yyyy
10:54:06.746 [LogStash::Runner] DEBUG logstash.runner - http.environment: "production"
10:54:06.746 [LogStash::Runner] DEBUG logstash.runner - queue.type: "memory"
10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.page_capacity: 262144000
10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.max_bytes: 1073741824
10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.max_events: 0
10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.checkpoint.acks: 1024
10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.checkpoint.writes: 1024
10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - queue.checkpoint.interval: 1000
10:54:06.762 [LogStash::Runner] DEBUG logstash.runner - slowlog.threshold.warn: -1
10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - slowlog.threshold.info: -1
10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - slowlog.threshold.debug: -1
10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - slowlog.threshold.trace: -1
10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - path.queue: "C:/DATA/Logstash/logstash-5.2.2/data/queue"
10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - path.settings: "C:/DATA/Logstash/logstash-5.2.2/config"
10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - path.logs: "C:/DATA/Logstash/logstash-5.2.2/logs"
shirm
October 23, 2018, 10:26am
3
10:54:06.777 [LogStash::Runner] DEBUG logstash.runner - --------------- Logstash Settings -------------------
10:54:06.824 [LogStash::Runner] DEBUG logstash.agent - Agent: Configuring metric collection
10:54:06.840 [LogStash::Runner] DEBUG logstash.instrument.periodicpoller.os - PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
10:54:06.887 [LogStash::Runner] DEBUG logstash.instrument.periodicpoller.jvm - PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
10:54:06.996 [LogStash::Runner] DEBUG logstash.instrument.periodicpoller.persistentqueue - PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
10:54:07.059 [LogStash::Runner] DEBUG logstash.agent - Reading config file {:config_file=>"C:/DATA/Logstash/logstash-5.2.2/bin/logstashAPI.config"}
10:54:07.731 [LogStash::Runner] DEBUG logstash.plugins.registry - On demand adding plugin to the registry {:name=>"http_poller", :type=>"input", :class=>LogStash::Inputs::HTTP_Poller}
10:54:07.746 [LogStash::Runner] WARN logstash.inputs.http_poller - You are using a deprecated config setting "interval" set in http_poller. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. If you have any
questions about this, please visit the #logstash channel on freenode irc. {:name=>"interval", :plugin=><LogStash::Inputs::HTTP_Poller urls=>{"uat3"=>"https://MyTeamCityServer/app/rest/server/"}, interval=>60, codec=>"json", truststore=>"C:\\DATA\\Log
stash\\certfile\\RootFile.jks", id=>"44d33c5fb43af50fe180e0c136aa3d44f5bc0436-1">}
10:54:07.777 [LogStash::Runner] DEBUG logstash.plugins.registry - On demand adding plugin to the registry {:name=>"json", :type=>"codec", :class=>LogStash::Codecs::JSON}
10:54:07.793 [LogStash::Runner] DEBUG logstash.codecs.json - config LogStash::Codecs::JSON/@id = "json_bd65a57c-9608-456d-a521-9fcf559ec37e"
10:54:07.793 [LogStash::Runner] DEBUG logstash.codecs.json - config LogStash::Codecs::JSON/@enable_metric = true
10:54:07.793 [LogStash::Runner] DEBUG logstash.codecs.json - config LogStash::Codecs::JSON/@charset = "UTF-8"
10:54:07.793 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@urls = {"uat3"=>"https://MyTeamCityServer/app/rest/server/"}
10:54:07.809 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@interval = 60
10:54:07.809 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@codec = <LogStash::Codecs::JSON id=>"json_bd65a57c-9608-456d-a521-9fcf559ec37e", enable_metric=>true, charset=>"UTF-8">
10:54:07.809 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@truststore = "C:\\DATA\\Logstash\\certfile\\RootFile.jks"
10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@id = "44d33c5fb43af50fe180e0c136aa3d44f5bc0436-1"
10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@enable_metric = true
10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@add_field = {}
10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@request_timeout = 60
10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@socket_timeout = 10
10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@connect_timeout = 10
10:54:07.824 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@follow_redirects = true
10:54:07.840 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@pool_max = 50
10:54:07.840 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@pool_max_per_route = 25
10:54:07.840 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@keepalive = true
10:54:07.840 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@automatic_retries = 1
10:54:07.856 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@retry_non_idempotent = false
10:54:08.684 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@validate_after_inactivity = 200
10:54:08.684 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@ssl_certificate_validation = true
10:54:08.715 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@keystore_type = "JKS"
10:54:08.715 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@truststore_type = "JKS"
10:54:08.715 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@cookies = true
10:54:08.715 [LogStash::Runner] DEBUG logstash.inputs.http_poller - config LogStash::Inputs::HTTP_Poller/@metadata_target = "@metadata"
10:54:08.731 [LogStash::Runner] DEBUG logstash.plugins.registry - On demand adding plugin to the registry {:name=>"stdout", :type=>"output", :class=>LogStash::Outputs::Stdout}
10:54:08.762 [LogStash::Runner] DEBUG logstash.plugins.registry - On demand adding plugin to the registry {:name=>"rubydebug", :type=>"codec", :class=>LogStash::Codecs::RubyDebug}
10:54:08.762 [LogStash::Runner] DEBUG logstash.codecs.rubydebug - config LogStash::Codecs::RubyDebug/@id = "rubydebug_1aab7cc6-a81e-412a-95ba-6bc1c7db1d02"
10:54:08.777 [LogStash::Runner] DEBUG logstash.codecs.rubydebug - config LogStash::Codecs::RubyDebug/@enable_metric = true
10:54:08.777 [LogStash::Runner] DEBUG logstash.codecs.rubydebug - config LogStash::Codecs::RubyDebug/@metadata = false
10:54:08.934 [LogStash::Runner] DEBUG logstash.outputs.stdout - config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::RubyDebug id=>"rubydebug_1aab7cc6-a81e-412a-95ba-6bc1c7db1d02", enable_metric=>true, metadata=>false>
10:54:08.949 [LogStash::Runner] DEBUG logstash.outputs.stdout - config LogStash::Outputs::Stdout/@id = "44d33c5fb43af50fe180e0c136aa3d44f5bc0436-2"
10:54:08.949 [LogStash::Runner] DEBUG logstash.outputs.stdout - config LogStash::Outputs::Stdout/@enable_metric = true
10:54:08.965 [LogStash::Runner] DEBUG logstash.outputs.stdout - config LogStash::Outputs::Stdout/@workers = 1
shirm
October 23, 2018, 10:27am
4
10:54:08.981 [Ruby-0-Thread-3: C:/DATA/Logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:22] DEBUG logstash.agent - starting agent
10:54:08.996 [Ruby-0-Thread-3: C:/DATA/Logstash/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:22] DEBUG logstash.agent - starting pipeline {:id=>"main"}
10:54:09.074 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
10:54:09.152 [[main]-pipeline-manager] INFO logstash.inputs.http_poller - Registering http_poller Input {:type=>nil, :urls=>{"uat3"=>"https://MyTeamCityServer/app/rest/server/"}, :interval=>60, :schedule=>nil, :timeout=>nil}
10:54:09.152 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
10:54:09.152 [[main]<http_poller] DEBUG logstash.inputs.http_poller - Fetching URL {:name=>"uat3", :url=>[:get, "https://MyTeamCityServer/app/rest/server/"]}
10:54:09.184 [[main]<http_poller] ERROR logstash.pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::HTTP_Poller urls=>{"uat3"=>"https://MyTeamCityServer/app/rest/server/"}, interval=>60, codec=><LogStash::Codecs::JSON id=>"json_bd65a57c-9608-456d-a521-9fcf559ec37e", enable_metric=>true, charset=>"UTF-8">, truststore=>"C
:\\DATA\\Logstash\\certfile\\RootFile.jks", id=>"44d33c5fb43af50fe180e0c136aa3d44f5bc0436-1", enable_metric=>true, request_timeout=>60, socket_timeout=>10, connect_timeout=>10, follow_redirects=>true, pool_max=>50, pool_max_per_route=>25, keepalive=>true, automa
tic_retries=>1, retry_non_idempotent=>false, validate_after_inactivity=>200, ssl_certificate_validation=>true, keystore_type=>"JKS", truststore_type=>"JKS", cookies=>true, metadata_target=>"@metadata">
Error: undefined method `value' for nil:NilClass
Exception: NoMethodError
system
November 23, 2018, 8:12am
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.