I'm trying to instrument Weblogic server using the Elastic APM Java Agent. I don't see any errors in the logs after restarting the server post applying the required JVM Arguments.
When we tried to access the application running on the Weblogic, we cannot see any data that is being sent to elastic/kibana for reporting.
Any suggestions or configuration i am missing?
WebLogic is currently not one of the supported application servers.
In order to be able to analyze your problem better, could you attach the debug logs of the agent?
Can you let me know how to upload the file? I don't see an option to upload the files.
I see below errors in the logs.
2018-08-07 11:21:55.837 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 1 elements to APM server http://server1:8200
2018-08-07 11:21:56.043 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - APM server responded with status code 400
2018-08-07 11:21:56.044 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - data decoding error: data read error: invalid character ',' looking for beginning of value
2018-08-07 11:22:15.829 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG co.elastic.apm.impl.ElasticApmTracer - startTransaction '' 8932cf6bd64b57a02193ba9ed71cd206 {
2018-08-07 11:22:15.838 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG co.elastic.apm.impl.ElasticApmTracer - } endTransaction 'GET' 8932cf6bd64b57a02193ba9ed71cd206
2018-08-07 11:22:15.991 [[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG co.elastic.apm.impl.ElasticApmTracer - startTransaction '' 75c50d820f311475715fc9f7fa073e7f {
2018-08-07 11:22:16.838 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 1 elements to APM server http://server1:8200
2018-08-07 11:22:17.018 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - APM server responded with status code 400
2018-08-07 11:22:17.019 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - data decoding error: data read error: invalid character ',' looking for beginning of value
2018-08-07 11:22:19.593 [[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG co.elastic.apm.impl.ElasticApmTracer - } endTransaction 'GET' 75c50d820f311475715fc9f7fa073e7f
2018-08-07 11:22:19.746 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG co.elastic.apm.impl.ElasticApmTracer - startTransaction '' 08e13b14d0531c249161a05d194ad4fe {
2018-08-07 11:22:19.837 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 1 elements to APM server http://server1:8200
2018-08-07 11:22:20.017 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - APM server responded with status code 400
That looks like a nasty JSON encoding bug:
APM server responded with status code 400
data decoding error: data read error: invalid character ',' looking for beginning of value
Could you set the log_level to TRACE? Then we can also see the JSON which gets sent to the APM server.
Which version of the agent are you using?
We are using elastic-apm-agent-0.6.0.jar
I'll post the TRACE loggging when ready.
I see below exception in logs.
2018-08-08 13:34:50.026 [[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'] TRACE co.elastic.apm.impl.ElasticApmTracer - ending transaction at
java.lang.RuntimeException: this exception is just used to record where the transaction has been ended from
at co.elastic.apm.impl.ElasticApmTracer.endTransaction(ElasticApmTracer.java:257)
at co.elastic.apm.impl.transaction.Transaction.end(Transaction.java:219)
at co.elastic.apm.impl.transaction.Transaction.end(Transaction.java:210)
at co.elastic.apm.servlet.ServletTransactionHelper.onAfter(ServletTransactionHelper.java:138)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:247)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3679)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3649)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:326)
at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2433)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2281)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2259)
at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1691)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1651)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:270)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:640)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:406)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:346)
2018-08-08 13:34:50.027 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Receiving TRANSACTION event (sequence 468)
2018-08-08 13:34:50.027 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Finished processing TRANSACTION event (sequence 468)
2018-08-08 13:34:50.758 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Receiving FLUSH event (sequence 469)
2018-08-08 13:34:50.758 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 2 elements to APM server http://server1:8200
2018-08-08 13:34:50.027 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Receiving TRANSACTION event (sequence 468)
2018-08-08 13:34:50.027 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Finished processing TRANSACTION event (sequence 468)
2018-08-08 13:34:50.758 [apm-reporter] TRACE co.elastic.apm.report.ReportingEventHandler - Receiving FLUSH event (sequence 469)
2018-08-08 13:34:50.758 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - Sending payload with 2 elements to APM server http://server1:8200
2018-08-08 13:34:50.764 [apm-reporter] TRACE co.elastic.apm.report.serialize.DslJsonSerializer - XYZ
2018-08-08 13:34:50.948 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - APM server responded with status code 400
2018-08-08 13:34:50.949 [apm-reporter] DEBUG co.elastic.apm.report.ApmServerHttpPayloadSender - data decoding error: unexpected EOF
XYZ - I removed the actual json data as it is huge.
It’s the actual JSON data I’m interested in. I need it to be able to see where the JSON is invalid. Alternatively, you can validate it yourself and tell me where it is invalid.
When i format it, below is the error i received.
Parse error on line 1:
..."headers":{"Date: ":,"Content-Length: ":
-----------------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got ','
Extract of actual Data is:
"response":{"headers":{"Date: ":,"Content-Length: ":,"Content-Type: ":,"Content-Transfer-Encoding":"binary","ASCFServer-Elapsed-Nano":"552122006"}
This is very helpful! Thanks a lot! I'll investigate.
When i change it the json string as below it's able to render correctly.
"response":{"headers":{"Date: ": "","Content-Length: ": "" ,"Content-Type: ":,"Content-Transfer-Encoding":"binary","ASCFServer-Elapsed-Nano":"552122006"}
Like this i observed it at multiple locations, post correcting those it rendered correctly and no JSON parsing error.
Hope it helps to narrow down the issue.
Any comments about the Exception i mentioned above? Is this related to same json parsing?
This PR should fix it: https://github.com/elastic/apm-agent-java/pull/158
What's really strange is this header key: "Date: ". Why is there a colon and a space after the header name? It seems to indicate that WebLogic returns "Date: " instead of "Date" when iterating over the header name Enumeration of javax.servlet.http.HttpServletRequest#getHeaderNames. This seems true for all headers where there is another method in the servlet API to get the header values: HttpServletRequest#getDateHeader, ServletRequest#getContentType, ServletRequest#getContentLength. Which WebLogic version are you using? This seems to be a WebLogic bug.
Nevertheless, the PR fixes the validation error. However, you won't see the value of the three aforementioned headers due to the WebLogic bug.
WebLogic Server 12.2.1.2.0
Will this part of the next release?
Yes, expect a bugfix release soon. I'll update on this thread as soon as we did the release.
Thank you!
Awaiting for the bugfix release to validate it.
In the meantime, you can try out the latest snapshot: https://github.com/elastic/apm-agent-java#snapshots
I have just released 0.6.2. It might take a while to be available on maven central.
Can you share the download link please?
i am not able to find it below where i downloaded last time. http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22co.elastic.apm%22%20AND%20a%3A%22elastic-apm-agent%22
good to have the release binary also in github itself for users to download and try.
The docs contain a link to the download page: https://www.elastic.co/guide/en/apm/agent/java/current/intro.html#get-started
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.