# Appdynamics data into elasticsearch

**URL:** <https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922>\
**Category:** Logstash\
**Created:** [June 21, 2018, 5:26pm UTC](https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922 "2018-06-21T17:26:19Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [June 21, 2018, 5:26pm UTC](https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922/1 "2018-06-21T17:26:19Z")

</div>

Hey Guys,

I have been trying to get appdynamics (APM tool) data into elasticsearch through logstash.

We are using appdynamics rest api for consuming the data through GET method.

Everything looks ok, but when i run the config filei'm getting error stating invalid URL.

**Here is my config file:**

input {  
http\_poller {  
urls =\> {  
url =\> "[http://10.1.4.2:8090/controller/rest/applications/konakart/metric-data?metric-path=Business%20Transaction%20Performance|Business%20Transactions|web\_tier|](http://10.1.4.2:8090/controller/rest/applications/konakart/metric-data?metric-path=Business%20Transaction%20Performance%7CBusiness%20Transactions%7Cweb_tier%7C)_|_&Time%20%28ms%29&time-range-type=BEFORE\_NOW&duration-in-mins=15"  
}  
request\_timeout =\> 60  
user =\> "\*\*\*\*\*"  
password =\> "\*\*\*\*\*\*_"  
schedule =\> { cron =\> "_ \* \* \* \*"}  
codec =\> "json"  
metadata\_target =\> "http\_poller\_metadata"  
}  
}  
filter  
{  
split  
{  
field =\> "message"  
}  
}  
output {  
elasticsearch {  
hosts =\> ["10.1.1.1:9200"]  
index =\> "appdynamics"  
}  
stdout {  
codec =\> json  
}  
}

**Message in log file:**

[2018-06-21T16:43:22,145][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<LogStash::ConfigurationError: Invalid URL [https://10.1.4.2:8090/controller/rest/applications/konakart/metric-data?metric-path=Business%20Transaction%20Performance|Business%20Transactions|web\_tier|](https://10.1.4.2:8090/controller/rest/applications/konakart/metric-data?metric-path=Business%20Transaction%20Performance%7CBusiness%20Transactions%7Cweb_tier%7C)_|_&Time%20%28ms%29&time-range-type=BEFORE\_NOW&duration-in-mins=15\>, :backtrace=\>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http\_poller-4.0.5/lib/logstash/inputs/http\_poller.rb:105:in `validate_request!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:97:in`normalize\_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http\_poller-4.0.5/lib/logstash/inputs/http\_poller.rb:57:in `block in setup_requests!'", "org/jruby/RubyHash.java:1343:in`each'", "org/jruby/RubyEnumerable.java:830:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:57:in`setup\_requests!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http\_poller-4.0.5/lib/logstash/inputs/http\_poller.rb:47:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:in`register\_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:353:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in`each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:353:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:500:in`start\_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:394:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:in`run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:250:in `block in start'"], :thread=\>"#\<Thread:0x2e688f29@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run\>"}

Also is there a way to monitor appdynamics through elasticsearch APM?

Thanks  
Gautham

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 21, 2018, 6:00pm UTC](https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922/2 "2018-06-21T18:00:35Z")

</div>

You need to URL encode | as %7C. Also, your cron schedule is not well formed.

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [June 22, 2018, 5:40am UTC](https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922/3 "2018-06-22T05:40:46Z")

</div>

Yeah that did the magic. Thank you very much.

Also @Badger i'm getting the required data in message field, if i try to filter using split, the data is getting splitted line by line, any suggestions.

**Here is the data without split:**  
message \< metric-datas\>\< metric-data\>  
**\< metricId\>120315\< /metricId\>**  
\< metricPath\>Business Transaction Performance|Business Transactions|web\_tier|/konakart/Computer-Peripherals/Graphics-Cards/Matrox|Calls per Minute\< /metricPath\>  
\< metricName\>BTM|BTs|BT:43|Component:8|Calls per Minute\< /metricName\>  
\< frequency\>ONE\_MIN\< /frequency\>  
\< metricValues\>  
\< metric-value\>  
\< startTimeInMillis\>1529637720000\< /startTimeInMillis\>  
\< value\>0\< /value\>  
\< min\>2147483647\< /min\>  
\< max\>-2147483648\< /max\>  
\< current\>0\</ current\>  
\< sum\>0\< /sum\>  
\< count\>0\< /count\>  
\< standardDeviation\>0.0\< /standardDeviation\>  
\< occurrences\>0\< /occurrences\>  
\< useRange\>false\< /useRange\>  
\< /metric-value\>  
\< /metricValues\>  
\< /metric-data\>  
\< metric-data\>  
**\< metricId\>60839\< /metricId\>**  
\< metricPath\>Business Transaction Performance|Business Transactions|web\_tier|test./konakart/Computer-Peripherals/Graphics-Cards/Matrox/Matrox-G200-MMS|Errors per Minute\< /metricPath\>  
\< metricName\>BTM|BTs|BT:40|Component:8|Errors per Minute\< /metricName\>  
\< frequency\>ONE\_MIN\< /frequency\>  
\< metricValues\>  
\< metric-value\>  
\< startTimeInMillis\>1529637720000\< /startTimeInMillis\>  
\< value\>0\< /value\>  
\< min\>2147483647\< /min\>  
\< max\>-2147483648\< /max\>  
\< current\>0\< /current\>  
\< sum\>0\< /sum\>  
\< count\>0\< /count\>  
\< standardDeviation\>0.0\< /standardDeviation\>  
\< occurrences\>0\< /occurrences\>  
\< useRange\>false\< /useRange\>  
\< /metric-value\>  
\< /metricValues\>  
\< /metric-data\>  
\< metric-data\>

When i give split by using filter option the above data is getting splitted line by line, is it possible to separate the data by metric ID which i have highlighted in bold.

Thanks  
Gautham

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 22, 2018, 4:24pm UTC](https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922/4 "2018-06-22T16:24:21Z")

</div>

Well, that's not valid XML, but assuming it really ends with \</metric-datas\> then it can be turned into XML and split using

```
    mutate { gsub => ["message", "< ", "<"] }
    mutate { gsub => ["message", "/ ", "/"] }
    xml { source => "message" store_xml => true target => "theXML" force_array => false }
    split { field => "[theXML][metric-data]" }

```

If you select part of your post in the edit window and click on the \</\> button in the toolbar it may help with your formatting.

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [June 24, 2018, 9:09am UTC](https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922/5 "2018-06-24T09:09:48Z")

</div>

Hey @Badger i actual i'm pulling the data through a webservices, which in turn provides me data in xml format, have copied only two sets in my previous post as the message field is pretty big, and yeah .

I had tried the options which you gave looks like there is an issue with config file after adding the fileds.

**Error MEssage:**  
[2018-06-24T12:45:46,421][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, input, filter, output at line 21, column 1 (byte 742) after ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in`compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in`map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in`execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in`with\_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in`each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in`block in converge\_state\_and\_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in`converge\_state\_and\_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in`block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

\*\ ***Filter part of My config file now:**

filter  
{  
mutate { gsub =\> ["message", "\< ", "\<"] }  
mutate { gsub =\> ["message", "/ ", "/"] }  
xml {  
source =\> "message"  
store\_xml =\> true  
target =\> "theXML"  
force\_array =\> false }  
split { field =\> "[theXML][metric-data]" }  
}  
}

Am i missing something

Thanks  
Gautham

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 24, 2018, 8:09pm UTC](https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922/6 "2018-06-24T20:09:23Z")

</div>

Without knowing which line is line 21 there is no way to say.

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [June 25, 2018, 4:50am UTC](https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922/7 "2018-06-25T04:50:29Z")

</div>

Got it @Badger i had missed an open and close braces in the split field section, now the data is proper. Thank you very much,

For future reference, here is the working config file.

input {  
http\_poller {  
urls =\> {  
url =\> "[http://1.1.1.2:8090/controller/rest/applications/konakart/metric-data?metric-path=Business%20Transaction%20Performance|Business%20Transactions|web\_tier|\*|\*&Time%20(ms)&time-range-type=BEFORE\_NOW&duration-in-mins=15](http://1.1.1.2:8090/controller/rest/applications/konakart/metric-data?metric-path=Business%20Transaction%20Performance%7CBusiness%20Transactions%7Cweb_tier%7C*%7C*&Time%20%28ms%29&time-range-type=BEFORE_NOW&duration-in-mins=15)"  
}  
request\_timeout =\> 60  
user =\> "\*\*\*\*\*"  
password =\> "\*\*\*_"  
schedule =\> { cron =\> "_ \* \* \* \*"}  
codec =\> "json"  
metadata\_target =\> "http\_poller\_metadata"  
}  
}  
filter  
{  
mutate { gsub =\> ["message", "\< ", "\<"] }  
mutate { gsub =\> ["message", "/ ", "/"] }  
xml {  
source =\> "message"  
store\_xml =\> true  
target =\> "theXML"  
force\_array =\> false  
}  
split  
{  
field =\> "[theXML][metric-data]"  
}  
}  
output {  
elasticsearch {  
hosts =\> ["1.1.1.1:9200"]  
index =\> "appdynamics"  
}

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 23, 2018, 4:50am UTC](https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922/8 "2018-07-23T04:50:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.