Java-ecs-logging

[Rem]

Hi,
Hope there is someone that can help me.

Config.xml for Log4j2 :

<Console name="LogToConsole" target="SYSTEM_OUT">
	<Appenders>
		<Console name="LogToConsole" target="SYSTEM_OUT">
			<EcsLayout>
				<KeyValuePair key="additionalField1" value="constant value"/>
				<KeyValuePair key="typeFromStructMsg" value="${sd:msg}"/>
				<KeyValuePair key="nameFromMapMsg" value="${map:name}"/>
                                <KeyValuePair key="nameFromContext" value="${ctx:test}"/>
				<KeyValuePair key="mySysProperty" value="${sys:mySysProperty}"/>
			</EcsLayout>
		</Console>
	</Appenders>
	<Loggers>
		<Root>
			<AppenderRef ref="LogToConsole" />
		</Root>
	</Loggers>
</Configuration>

When I add value in a Map or in the ThreadContext, I always get "labels.name" or "labels.test" but I need the fields to be at the base of my Elastic Common Schema (ECS).

How can I do it and is it thread safe?

Thank you!

Example of code :

StringMapMessage mapMsg = new StringMapMessage();
        mapMsg.put("name", "arun");
		logger.warn(mapMsg);

or

ThreadContext.put("test", "test");
logger.info("any message");

using this open source project : https://github.com/elastic/java-ecs-logging

[Kerry (Kerry Gallagher)]

Hi @Rem, I've reached out to someone who knows about the java-ecs-logging project, as this is unfortunately not something I can help you with directly myself.

[felixbarny (Felix Barnsteiner)]

Thanks for checking out the EcsLayout!

As you have noticed, by default the MDC values are nested under labels as this is the most natural place from the ECS perspective. However, there is a configuration setting for the log4j2 EcsLayout where you can define specific fields which should be serialized at the top level: https://github.com/elastic/java-ecs-logging/blob/master/log4j2-ecs-layout/README.md#layout-parameters.

Also make sure to have a look at the tips and gotchas listed here: https://github.com/elastic/java-ecs-logging/blob/master/log4j2-ecs-layout/README.md#structured-logging.

Cheers,
Felix

[Rem]

Hi,
Here is my Config.xml file for Log4j2. Notice that event.action is an official field that I took from the Elastic Common Scheman (ECS).
Therefore, I expect the fields to be at the base of the the json output (no labels prefix).

<Console name="LogToConsole" target="SYSTEM_OUT">
	<Appenders>
		<Console name="LogToConsole" target="SYSTEM_OUT">
			<EcsLayout>
				<KeyValuePair key="event.action" value="$${ctx:event.action}" />
			</EcsLayout>
		</Console>
	</Appenders>
	<Loggers>
		<Root>
			<AppenderRef ref="LogToConsole" />
		</Root>
	</Loggers>
</Configuration>

Here's my code from my JUnit test :

package com.myapplication.logs

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.ThreadContext;

import junit.framework.TestCase;

public class ECSLayoutJUnit extends TestCase{
	private Logger logger = LogManager.getLogger(ECSLayoutJUnit.class);
    
	public void testLogConsole() throws Exception {
		ThreadContext.put("event.action", "Processing");
		logger.info("I am a simple test.");
		ThreadContext.clearMap();
	}
}

And here is the output :

{
    "@timestamp": "2019-09-27T15:33:50.745Z",
    "log.level": "INFO",
    "message": "I am a simple test.",
    "process.thread.name": "main",
    "log.logger": "com.myapplication.logs.ECSLayoutJUnit",
    "event.action": "Processing",
    "labels.event.action": "Processing"
}

Expected behavior :

{
    "@timestamp": "2019-09-27T15:33:50.745Z",
    "log.level": "INFO",
    "message": "I am a simple test.",
    "process.thread.name": "main",
    "log.logger": "com.myapplication.logs.ECSLayoutJUnit",
    "event.action": "Processing"
}

Why am I having the field "labels.event.action" if this is an official ECS field?
What am I doing wrong?
Note that the field "event.action" is duplicated by "labels.event.action".
Note that I need to fill the ECS field "event.action" at the runtime and I need some java code to do so.

Thank you!

[felixbarny (Felix Barnsteiner)]

Try with this configuration:

<Console name="LogToConsole" target="SYSTEM_OUT">
	<Appenders>
		<Console name="LogToConsole" target="SYSTEM_OUT">
			<EcsLayout topLevelLabels="event.action"/>
		</Console>
	</Appenders>
	<Loggers>
		<Root>
			<AppenderRef ref="LogToConsole" />
		</Root>
	</Loggers>
</Configuration>

Then you can do this:

ThreadContext.put("event.action", "Processing");
logger.info("I am a simple test.");
ThreadContext.clearMap();

However, I would recommend using StringMapMessage. No special configuration is required then:

logger.info(new StringMapMessage()
    .with("message", "I am a simple test.")
    .with("event.action", "Processing"));

[Rem]

The configurations work from version 0.1.1 of the project.

Thank you so much!

Thought, we do not need anymore to add the topLevelLabels="event.action" with this version.

So, I guess I will have add it manually if necessary.

The configuration topLevelLabels is now necessary only if I use the ThreadContext at runtime to add a JSON parameter to the logs.

If the field is add at runtime via StringMapMessage, it is at the top level.
If the field is add at runtime with ThreadContext. Then, the prefix label is add.

[felixbarny (Felix Barnsteiner)]

I've just released 0.1.1