# Unable to instrument WebLogic Server with APM Java Agent **URL:** \ **Category:** APM\ **Created:** [August 7, 2018, 11:27am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312 "2018-08-07T11:27:24Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Madan](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Madan](https://discuss.elastic.co/u/Madan)\ **Post date:** [August 7, 2018, 11:27am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/1 "2018-08-07T11:27:24Z")
I'm trying to instrument Weblogic server using the Elastic APM Java Agent. I don't see any errors in the logs after restarting the server post applying the required JVM Arguments. When we tried to access the application running on the Weblogic, we cannot see any data that is being sent to elastic/kibana for reporting. Any suggestions or configuration i am missing? ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [August 7, 2018, 11:53am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/2 "2018-08-07T11:53:34Z")
WebLogic is currently not one of the [supported application servers](https://www.elastic.co/guide/en/apm/agent/java/current/intro.html#supported-app-servers). In order to be able to analyze your problem better, could you attach the [debug logs](https://www.elastic.co/guide/en/apm/agent/java/current/trouble-shooting.html#trouble-shooting-logging) of the agent? ---
**Author:** ![Madan](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Madan](https://discuss.elastic.co/u/Madan)\ **Post date:** [August 7, 2018, 2:16pm UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/3 "2018-08-07T14:16:18Z")
Can you let me know how to upload the file? I don't see an option to upload the files. ---
**Author:** ![Madan](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Madan](https://discuss.elastic.co/u/Madan)\ **Post date:** [August 8, 2018, 6:45am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/4 "2018-08-08T06:45:57Z")
I see below errors in the logs. 2018-08-07 11:21:55.837 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 1 elements to APM server [http://server1:8200](http://server1:8200) 2018-08-07 11:21:56.043 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - APM server responded with status code 400 2018-08-07 11:21:56.044 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - data decoding error: data read error: invalid character ',' looking for beginning of value 2018-08-07 11:22:15.829 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG co.elastic.apm.impl.ElasticApmTracer - startTransaction '' 8932cf6bd64b57a02193ba9ed71cd206 { 2018-08-07 11:22:15.838 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG co.elastic.apm.impl.ElasticApmTracer - } endTransaction 'GET' 8932cf6bd64b57a02193ba9ed71cd206 2018-08-07 11:22:15.991 [[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG co.elastic.apm.impl.ElasticApmTracer - startTransaction '' 75c50d820f311475715fc9f7fa073e7f { 2018-08-07 11:22:16.838 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 1 elements to APM server [http://server1:8200](http://server1:8200) 2018-08-07 11:22:17.018 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - APM server responded with status code 400 2018-08-07 11:22:17.019 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - data decoding error: data read error: invalid character ',' looking for beginning of value 2018-08-07 11:22:19.593 [[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG co.elastic.apm.impl.ElasticApmTracer - } endTransaction 'GET' 75c50d820f311475715fc9f7fa073e7f 2018-08-07 11:22:19.746 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG co.elastic.apm.impl.ElasticApmTracer - startTransaction '' 08e13b14d0531c249161a05d194ad4fe { 2018-08-07 11:22:19.837 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 1 elements to APM server [http://server1:8200](http://server1:8200) 2018-08-07 11:22:20.017 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - APM server responded with status code 400 ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [August 8, 2018, 8:23am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/5 "2018-08-08T08:23:25Z")
That looks like a nasty JSON encoding bug: ``` APM server responded with status code 400 data decoding error: data read error: invalid character ',' looking for beginning of value ``` Could you set the `log_level` to `TRACE`? Then we can also see the JSON which gets sent to the APM server. Which version of the agent are you using? ---
**Author:** ![Madan](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Madan](https://discuss.elastic.co/u/Madan)\ **Post date:** [August 8, 2018, 10:21am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/6 "2018-08-08T10:21:51Z")
We are using elastic-apm-agent-0.6.0.jar I'll post the TRACE loggging when ready. ---
**Author:** ![Madan](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Madan](https://discuss.elastic.co/u/Madan)\ **Post date:** [August 9, 2018, 6:15am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/7 "2018-08-09T06:15:55Z")
I see below exception in logs. 2018-08-08 13:34:50.026 [[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'] TRACE co.elastic.apm.impl.ElasticApmTracer - ending transaction at java.lang.RuntimeException: this exception is just used to record where the transaction has been ended from at co.elastic.apm.impl.ElasticApmTracer.endTransaction(ElasticApmTracer.java:257) at co.elastic.apm.impl.transaction.Transaction.end(Transaction.java:219) at co.elastic.apm.impl.transaction.Transaction.end(Transaction.java:210) at co.elastic.apm.servlet.ServletTransactionHelper.onAfter(ServletTransactionHelper.java:138) at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:247) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3679) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3649) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:326) at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197) at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203) at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71) at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2433) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2281) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2259) at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1691) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1651) at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:270) at weblogic.invocation.ComponentInvocationContextManager.\_runAs(ComponentInvocationContextManager.java:348) at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333) at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54) at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41) at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:640) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:406) at weblogic.work.ExecuteThread.run(ExecuteThread.java:346) 2018-08-08 13:34:50.027 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Receiving TRANSACTION event (sequence 468) 2018-08-08 13:34:50.027 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Finished processing TRANSACTION event (sequence 468) 2018-08-08 13:34:50.758 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Receiving FLUSH event (sequence 469) 2018-08-08 13:34:50.758 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 2 elements to APM server [http://server1:8200](http://server1:8200) 2018-08-08 13:34:50.027 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Receiving TRANSACTION event (sequence 468) 2018-08-08 13:34:50.027 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Finished processing TRANSACTION event (sequence 468) 2018-08-08 13:34:50.758 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Receiving FLUSH event (sequence 469) 2018-08-08 13:34:50.758 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 2 elements to APM server [http://server1:8200](http://server1:8200) 2018-08-08 13:34:50.764 [apm-reporter] TRACE co.elastic.apm.report.serialize.DslJsonSerializer - **_XYZ_** 2018-08-08 13:34:50.948 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - APM server responded with status code 400 2018-08-08 13:34:50.949 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - data decoding error: unexpected EOF XYZ - I removed the actual json data as it is huge. ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [August 9, 2018, 6:30am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/8 "2018-08-09T06:30:36Z")
It’s the actual JSON data I’m interested in. I need it to be able to see where the JSON is invalid. Alternatively, you can validate it yourself and tell me where it is invalid. ---
**Author:** ![Madan](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Madan](https://discuss.elastic.co/u/Madan)\ **Post date:** [August 9, 2018, 6:34am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/9 "2018-08-09T06:34:55Z")
When i format it, below is the error i received. Parse error on line 1: ..."headers":{"Date: ":,"Content-Length: ": -----------------------^ Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got ',' Extract of actual Data is: "response":{"headers":{"Date: ":,"Content-Length: ":,"Content-Type: ":,"Content-Transfer-Encoding":"binary","ASCFServer-Elapsed-Nano":"552122006"} ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [August 9, 2018, 6:56am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/10 "2018-08-09T06:56:40Z")
This is very helpful! Thanks a lot! I'll investigate. ---
**Author:** ![Madan](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Madan](https://discuss.elastic.co/u/Madan)\ **Post date:** [August 9, 2018, 7:35am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/11 "2018-08-09T07:35:38Z")
When i change it the json string as below it's able to render correctly. "response":{"headers":{"Date: ": **""** ,"Content-Length: ": **""** ,"Content-Type: ":,"Content-Transfer-Encoding":"binary","ASCFServer-Elapsed-Nano":"552122006"} Like this i observed it at multiple locations, post correcting those it rendered correctly and no JSON parsing error. Hope it helps to narrow down the issue. Any comments about the Exception i mentioned above? Is this related to same json parsing? ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [August 9, 2018, 7:42am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/12 "2018-08-09T07:42:45Z")
This PR should fix it: [https://github.com/elastic/apm-agent-java/pull/158](https://github.com/elastic/apm-agent-java/pull/158) What's really strange is this header key: `"Date: "`. Why is there a colon and a space after the header name? It seems to indicate that WebLogic returns `"Date: "` instead of `"Date"` when iterating over the header name `Enumeration` of `javax.servlet.http.HttpServletRequest#getHeaderNames`. This seems true for all headers where there is another method in the servlet API to get the header values: `HttpServletRequest#getDateHeader`, `ServletRequest#getContentType`, `ServletRequest#getContentLength`. Which WebLogic version are you using? This seems to be a WebLogic bug. Nevertheless, the PR fixes the validation error. However, you won't see the value of the three aforementioned headers due to the WebLogic bug. ---
**Author:** ![Madan](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Madan](https://discuss.elastic.co/u/Madan)\ **Post date:** [August 9, 2018, 8:17am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/13 "2018-08-09T08:17:32Z")
WebLogic Server 12.2.1.2.0 ---
**Author:** ![Madan](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Madan](https://discuss.elastic.co/u/Madan)\ **Post date:** [August 9, 2018, 11:24am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/14 "2018-08-09T11:24:14Z")
> [@felixbarny](#): > > This PR should fix it: [https://github.com/elastic/apm-agent-java/pull/158](https://github.com/elastic/apm-agent-java/pull/158) Will this part of the next release? ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [August 9, 2018, 11:58am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/15 "2018-08-09T11:58:57Z")
> [@Madan](#): > > Will this part of the next release? Yes, expect a bugfix release soon. I'll update on this thread as soon as we did the release. ---
**Author:** ![Madan](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Madan](https://discuss.elastic.co/u/Madan)\ **Post date:** [August 9, 2018, 12:07pm UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/16 "2018-08-09T12:07:18Z")
Thank you! Awaiting for the bugfix release to validate it. ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [August 10, 2018, 7:19am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/17 "2018-08-10T07:19:38Z")
In the meantime, you can try out the latest snapshot: [https://github.com/elastic/apm-agent-java#snapshots](https://github.com/elastic/apm-agent-java#snapshots) ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [August 13, 2018, 5:26pm UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/18 "2018-08-13T17:26:05Z")
I have just released 0.6.2. It might take a while to be available on maven central. ---
**Author:** ![Madan](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Madan](https://discuss.elastic.co/u/Madan)\ **Post date:** [August 22, 2018, 5:40am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/19 "2018-08-22T05:40:15Z")
Can you share the download link please? i am not able to find it below where i downloaded last time. [http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22co.elastic.apm%22%20AND%20a%3A%22elastic-apm-agent%22](http://search.maven.org/%23search%7cgav%7c1%7cg:%22co.elastic.apm%22%20AND%20a:%22elastic-apm-agent%22) good to have the release binary also in github itself for users to download and try. ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [August 22, 2018, 6:37am UTC](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312/20 "2018-08-22T06:37:48Z")
The docs contain a link to the download page: [https://www.elastic.co/guide/en/apm/agent/java/current/intro.html#get-started](https://www.elastic.co/guide/en/apm/agent/java/current/intro.html#get-started) [Next page](https://discuss.elastic.co/t/unable-to-instrument-weblogic-server-with-apm-java-agent/143312.md?page=2)