Possible collision with user-defined field 'parameter' and built-in fields

[macs (Marc Cser)]

Hey,

I am working on an app where we tried to append a Logstash Marker with the custom key "parameter", containing a string with some information about where in the application the log event occured. Unfortunately, since about May 2021, when we tried to append this marker, and the content of "parameter" was not null, the log line would not show up in Kibana/Elasticsearch at all (completely vanished).

As soon as I give the parameter a different name, for example "parameters" or "foobar", the log line will show up in elasticsearch again. This has led me to believe that "parameter" may be a reserved field name that conflicts with an existing field that elastic uses internally, and when I try to overwrite it this runs into some sort of error.

If this is the case, it may be useful to post a list of reserved variables in elastic that may lead to such unforeseen problems, so that we, and other applications can take care not to use these keywords in custom logstash markers.

Thanks for looking into this!

Below is a minimum working example written in Java 11:

import net.logstash.logback.marker.Markers;

public class AnalyticsController {

    private static final Logger log = org.slf4j.LoggerFactory.getLogger(AnalyticsController.class);

    public void log() {
       log.info(Markers.append("parameter", "This parameter is not null"), "This will not be logged into Kibana.");
    }
}

With a Logback config similar to this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">

    <appender name="ELK" class="net.logstash.logback.appender.LogstashTcpSocketAppender">

        <destination>http://my-logback-url.com/</destination>
        <writeBufferSize>16384</writeBufferSize>
        <reconnectionDelay>1 second</reconnectionDelay>
        <keepAliveDuration>5 minutes</keepAliveDuration>


        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <includeCallerData>true</includeCallerData>
        </encoder>
    </appender>

    <root level="WARN">
        <appender-ref ref="ELK"/>
    </root>

</configuration>