No logs in elasticsearch while using log_sending and stdout log level = OFF

Elastic Observability APM
1

Kibana version: 8.8.1

Elasticsearch version: 8.8.1

APM Server version: 8.8.1

APM Agent language and version: java 1.39.0

Browser version: chrome 114.0.5735.134

Original install method (e.g. download page, yum, deb, from source, etc.) and version: maven

Fresh install or upgraded from other version? fresh

Is there anything special in your setup?

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

We have activated log_sending and experimental features is set to true

We only see limited logs in kibana from the application, specifically the apm agent logs only.

The application writes a lot of logs to files and the sdout (console) level is set to OFF:

<deployable>
	<name>logging.stdout</name>
	<type>wildfly.LoggingCategorySpec</type>
	<category>stdout</category>
	<level>OFF</level>
	<tags>
		<tag>${tag.default}</tag>
	</tags>
</deployable>

We receive logs only from the apm-agent itself and none of the logs written to the filesystem.

	Jun 20, 2023 @ 17:42:51.768	hp-backoffice	ERROR	Remote configuration is not available. Check the connection between APM Server and Kibana.
	Jun 20, 2023 @ 15:38:56.511	hp-backoffice	WARN	com.ibm.as400.access.AS400JDBCConnection uses an unsupported class file version (pre Java 4)) and can't be instrumented. You may try setting the 'instrument_ancient_bytecode' config option to 'true', but notice that it may cause VerificationErrors or other issues.
	Jun 20, 2023 @ 15:38:56.491	hp-backoffice	WARN	com.ibm.as400.access.AS400JDBCStatement uses an unsupported class file version (pre Java 4)) and can't be instrumented. You may try setting the 'instrument_ancient_bytecode' config option to 'true', but notice that it may cause VerificationErrors or other issues.
	Jun 20, 2023 @ 15:38:56.465	hp-backoffice	WARN	com.ibm.as400.access.AS400JDBCPreparedStatement uses an unsupported class file version (pre Java 4)) and can't be instrumented. You may try setting the 'instrument_ancient_bytecode' config option to 'true', but notice that it may cause VerificationErrors or other issues.
	Jun 20, 2023 @ 15:38:56.439	hp-backoffice	WARN	com.ibm.as400.access.AS400JDBCCallableStatement uses an unsupported class file version (pre Java 4)) and can't be instrumented. You may try setting the 'instrument_ancient_bytecode' config option to 'true', but notice that it may cause VerificationErrors or other issues.
	Jun 20, 2023 @ 15:38:39.401	hp-backoffice	INFO	Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: URI scheme is not "file"
	Jun 20, 2023 @ 15:38:39.344	hp-backoffice	INFO	Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: URI scheme is not "file"
	Jun 20, 2023 @ 15:38:39.334	hp-backoffice	WARN	org.apache.commons.httpclient.HttpMethodDirector uses an unsupported class file version (pre Java 4)) and can't be instrumented. You may try setting the 'instrument_ancient_bytecode' config option to 'true', but notice that it may cause VerificationErrors or other issues.
	Jun 20, 2023 @ 15:38:39.097	hp-backoffice	INFO	Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: URI scheme is not "file"
	Jun 20, 2023 @ 15:38:38.693	hp-backoffice	INFO	Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: URI scheme is not "file"
	Jun 20, 2023 @ 15:38:38.411	hp-backoffice	INFO	Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: URI scheme is not "file"
	Jun 20, 2023 @ 15:38:38.159	hp-backoffice	INFO	Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: URI scheme is not "file"
	Jun 20, 2023 @ 15:38:35.583	hp-backoffice	INFO	Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: URI scheme is not "file"
	Jun 20, 2023 @ 15:38:35.583	hp-backoffice	INFO	Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: URI scheme is not "file"
	Jun 20, 2023 @ 15:38:35.401	hp-backoffice	INFO	Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: URI scheme is not "file"
...
...
...

Does the log_sending feature only intercept logs going to system out?
Is there a way to target specific loggers to be intercepted in our Java code/settings?

Maybe this commit can be helpfull:

We understand that log_sending is an experimental feature but comes really handy in our PoC fese of implementing Observability.

Thanks.

2

Hi, what logging mechanism/framework is being used here? Can you be clearer about the components and versions, eg wildfly version, jdk version, logging framework and version. Thanks

3

Hello,

It seems that our JBOSS server uses the v1 logging library.
We will try upgrading to v3.0.0 first.

Thanks!

4

We also use wildfly 24 and this version is not currently supported:

Only wildfly versions 8-16 are tested.

We run on Java 11 so we cannot use apm-agent version 1.39.0 it seems..

5

Although these are tested, we expect the agent to work with later versions, and we are intending to start testing more recent versions

All our agent versions should work on java 11

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.