Appdynamics data into elasticsearch

Gauti · June 21, 2018, 5:26pm

Hey Guys,

I have been trying to get appdynamics (APM tool) data into elasticsearch through logstash.

We are using appdynamics rest api for consuming the data through GET method.

Everything looks ok, but when i run the config filei'm getting error stating invalid URL.

Here is my config file:

input {
http_poller {
urls => {
url => "http://10.1.4.2:8090/controller/rest/applications/konakart/metric-data?metric-path=Business%20Transaction%20Performance|Business%20Transactions|web_tier||&Time%20%28ms%29&time-range-type=BEFORE_NOW&duration-in-mins=15"
}
request_timeout => 60
user => "*****"
password => "******"
schedule => { cron => " * * * *"}
codec => "json"
metadata_target => "http_poller_metadata"
}
}
filter
{
split
{
field => "message"
}
}
output {
elasticsearch {
hosts => ["10.1.1.1:9200"]
index => "appdynamics"
}
stdout {
codec => json
}
}

Message in log file:

[2018-06-21T16:43:22,145][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Invalid URL https://10.1.4.2:8090/controller/rest/applications/konakart/metric-data?metric-path=Business%20Transaction%20Performance|Business%20Transactions|web_tier||&Time%20%28ms%29&time-range-type=BEFORE_NOW&duration-in-mins=15>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:105:in validate_request!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:97:innormalize_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:57:in block in setup_requests!'", "org/jruby/RubyHash.java:1343:ineach'", "org/jruby/RubyEnumerable.java:830:in map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:57:insetup_requests!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:47:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:353:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:353:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:500:instart_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:394:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:250:in `block in start'"], :thread=>"#<Thread:0x2e688f29@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}

Also is there a way to monitor appdynamics through elasticsearch APM?

Thanks
Gautham

Badger · June 21, 2018, 6:00pm

You need to URL encode | as %7C. Also, your cron schedule is not well formed.

Gauti · June 22, 2018, 5:40am

Yeah that did the magic. Thank you very much.

Also @Badger i'm getting the required data in message field, if i try to filter using split, the data is getting splitted line by line, any suggestions.

Here is the data without split:
message < metric-datas>< metric-data>
< metricId>120315< /metricId>
< metricPath>Business Transaction Performance|Business Transactions|web_tier|/konakart/Computer-Peripherals/Graphics-Cards/Matrox|Calls per Minute< /metricPath>
< metricName>BTM|BTs|BT:43|Component:8|Calls per Minute< /metricName>
< frequency>ONE_MIN< /frequency>
< metricValues>
< metric-value>
< startTimeInMillis>1529637720000< /startTimeInMillis>
< value>0< /value>
< min>2147483647< /min>
< max>-2147483648< /max>
< current>0</ current>
< sum>0< /sum>
< count>0< /count>
< standardDeviation>0.0< /standardDeviation>
< occurrences>0< /occurrences>
< useRange>false< /useRange>
< /metric-value>
< /metricValues>
< /metric-data>
< metric-data>
< metricId>60839< /metricId>
< metricPath>Business Transaction Performance|Business Transactions|web_tier|test./konakart/Computer-Peripherals/Graphics-Cards/Matrox/Matrox-G200-MMS|Errors per Minute< /metricPath>
< metricName>BTM|BTs|BT:40|Component:8|Errors per Minute< /metricName>
< frequency>ONE_MIN< /frequency>
< metricValues>
< metric-value>
< startTimeInMillis>1529637720000< /startTimeInMillis>
< value>0< /value>
< min>2147483647< /min>
< max>-2147483648< /max>
< current>0< /current>
< sum>0< /sum>
< count>0< /count>
< standardDeviation>0.0< /standardDeviation>
< occurrences>0< /occurrences>
< useRange>false< /useRange>
< /metric-value>
< /metricValues>
< /metric-data>
< metric-data>

When i give split by using filter option the above data is getting splitted line by line, is it possible to separate the data by metric ID which i have highlighted in bold.

Thanks
Gautham

Badger · June 22, 2018, 4:24pm

Well, that's not valid XML, but assuming it really ends with </metric-datas> then it can be turned into XML and split using

    mutate { gsub => [ "message", "< ", "<" ] }
    mutate { gsub => [ "message", "/ ", "/" ] }
    xml { source => "message" store_xml => true target => "theXML" force_array => false }
    split { field => "[theXML][metric-data]" }

If you select part of your post in the edit window and click on the </> button in the toolbar it may help with your formatting.

Gauti · June 24, 2018, 9:09am

Hey @Badger i actual i'm pulling the data through a webservices, which in turn provides me data in xml format, have copied only two sets in my previous post as the message field is pretty big, and yeah .

I had tried the options which you gave looks like there is an issue with config file after adding the fileds.

Error MEssage:
[2018-06-24T12:45:46,421][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 21, column 1 (byte 742) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

**Filter part of My config file now:

filter
{
mutate { gsub => [ "message", "< ", "<" ] }
mutate { gsub => [ "message", "/ ", "/" ] }
xml {
source => "message"
store_xml => true
target => "theXML"
force_array => false }
split { field => "[theXML][metric-data]" }
}
}

Am i missing something

Thanks
Gautham

Badger · June 24, 2018, 8:09pm

Without knowing which line is line 21 there is no way to say.

Gauti · June 25, 2018, 4:50am

Got it @Badger i had missed an open and close braces in the split field section, now the data is proper. Thank you very much,

For future reference, here is the working config file.

input {
http_poller {
urls => {
url => "http://1.1.1.2:8090/controller/rest/applications/konakart/metric-data?metric-path=Business%20Transaction%20Performance|Business%20Transactions|web_tier|*|*&Time%20(ms)&time-range-type=BEFORE_NOW&duration-in-mins=15"
}
request_timeout => 60
user => "*****"
password => "***"
schedule => { cron => " * * * *"}
codec => "json"
metadata_target => "http_poller_metadata"
}
}
filter
{
mutate { gsub => [ "message", "< ", "<" ] }
mutate { gsub => [ "message", "/ ", "/" ] }
xml {
source => "message"
store_xml => true
target => "theXML"
force_array => false
}
split
{
field => "[theXML][metric-data]"
}
}
output {
elasticsearch {
hosts => ["1.1.1.1:9200"]
index => "appdynamics"
}

Thanks
Gauti

system · July 23, 2018, 4:50am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Getting Appdynamics Data into ELK Logstash	1	371	September 18, 2020
Ingesting AppDynamics metric information into Elasticsearch Elasticsearch	1	518	February 13, 2020
Problem using http_poller to get access token from AppDynamics OAUTH API Logstash	2	937	November 17, 2021
Http_poller Logstash	2	308	October 8, 2019
Logstash http_poller input / http filter design question Logstash	2	236	December 15, 2021

Appdynamics data into elasticsearch

Related topics