How to enable deprecation logging in Elasticsearch 6.8?

Elastic Stack Elasticsearch
1

Ultimately, I want to upgrade a bunch of Python code that uses Elasticsearch 6.8 to use Elasticsearch 7.3.

Toward that goal, I want to enable Elasticsearch 6.8 "deprecation logging". It seems likely to be the best place to start.

I've been attempting to set this up mostly based on: https://www.elastic.co/guide/en/elasticsearch/reference/6.7/logging.html#deprecation-logging ...but it seems a bit inspecific, and I'm not finding a lot of other good hits in Google (many of which seem to be about much older versions of Elasticsearch).

Elasticsearch is running inside a docker container. Also, I do not need to preserve the data.

I have tried both /usr/share/elasticsearch/config/logging.yml (which seems to be ignored) and /usr/share/elasticsearch/config/log4j2.properties:(which gives "Exception in thread "main" org.apache.logging.log4j.core.config.ConfigurationException: No name attribute provided for Logger deprecation" and the container quickly exits). In both cases, I'm adding the line:
logger.deprecation.level = warn

After restarting the ES container, in /usr/share/elasticsearch/logs, I (still) have only one file: gc.log, which contains no occurrences of the string 'dep' (case insensitive).

My questions are:

  1. What file should the deprecation logs appear in?
  2. What do I need to do to get deprecation logs written with Elasticsearch 6.8?

Thanks!

2

Hey Dan

See Logging configuration | Elasticsearch Guide [6.7] | Elastic

By default, deprecation logging is enabled at the WARN level, the level at which all deprecation log messages will be emitted.

you should not need to configure anything in your logging configuration. If you remove any configuration, start elasticsearch and do something that is deprecated, there is no logfile being created? Have you tested that? If so, how exactly did you test that?

3

Hi Alexander.

I first destroy my docker volumes, then recreate them, then fire up the relevant docker containers, then run my automated test suite, and then:

$ docker exec -it elasticsearch /bin/bash
below cmd output started 2019 Tue Sep 03 02:42:59 PM PDT
[root@c489bb8f30ea elasticsearch]# pwd
/usr/share/elasticsearch
[root@c489bb8f30ea elasticsearch]# ls
LICENSE.txt NOTICE.txt README.textile bin config data lib logs modules plugins
[root@c489bb8f30ea elasticsearch]# ls logs/
gc.log
[root@c489bb8f30ea elasticsearch]# ls -l config/
total 24
-rw-rw---- 1 elasticsearch root 199 Sep 3 21:17 elasticsearch.keystore
-rw-r--r-- 1 root root 110 Aug 29 17:49 elasticsearch.yml
-rw-rw---- 1 elasticsearch root 3613 Jul 24 15:21 jvm.options
-rw-r--r-- 1 root root 330 Aug 29 23:18 log4j2.properties
-rw-rw---- 1 elasticsearch root 473 Jul 24 15:26 role_mapping.yml
-rw-rw---- 1 elasticsearch root 197 Jul 24 15:26 roles.yml
-rw-rw---- 1 elasticsearch root 0 Jul 24 15:26 users
-rw-rw---- 1 elasticsearch root 0 Jul 24 15:26 users_roles
[root@c489bb8f30ea elasticsearch]# cat config/log4j2.properties
status = error

appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n

rootLogger.level = debug
rootLogger.appenderRef.console.ref = console

# logger.deprecation.level = warn

# Foo!

[root@c489bb8f30ea elasticsearch]#

Is there anything else I should check? I'm told this is going to be an interesting upgrade, because we're using "types". Is it possible deprecation logging is enabled, but our use of types is going undetected anyway?

Thanks!

4

have you run docker logs? Does this contain only the regular log messages?

5

I hadn't checked docker logs.

It gives lots of output, and a little about deprecation:
$ docker logs elasticsearch | egrep -i deprecat
below cmd output started 2019 Wed Sep 04 08:57:51 AM PDT
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-09-04T15:15:15,825][DEBUG][o.e.b.JarHell ] [wC8pkqS] examining jar: /usr/share/elasticsearch/modules/x-pack-deprecation/x-pack-deprecation-6.8.2.jar
[2019-09-04T15:15:15,863][DEBUG][o.e.b.JarHell ] [wC8pkqS] examining jar: /usr/share/elasticsearch/modules/x-pack-deprecation/x-pack-deprecation-6.8.2.jar
[2019-09-04T15:15:18,034][INFO ][o.e.p.PluginsService ] [wC8pkqS] loaded module [x-pack-deprecation]
[2019-09-04T15:27:36,091][WARN ][o.e.d.i.m.MapperService ] [wC8pkqS] Setting index.mapper.dynamic is deprecated since indices may not have more than one type anymore.
[2019-09-04T15:27:36,165][WARN ][o.e.d.i.m.MapperService ] [wC8pkqS] Setting index.mapper.dynamic is deprecated since indices may not have more than one type anymore.
[2019-09-04T15:40:13,137][WARN ][o.e.d.i.m.MapperService ] [wC8pkqS] Setting index.mapper.dynamic is deprecated since indices may not have more than one type anymore.
[2019-09-04T15:41:55,475][WARN ][o.e.d.i.m.MapperService ] [wC8pkqS] Setting index.mapper.dynamic is deprecated since indices may not have more than one type anymore.
[2019-09-04T15:45:54,357][WARN ][o.e.d.i.m.MapperService ] [wC8pkqS] Setting index.mapper.dynamic is deprecated since indices may not have more than one type anymore.

Is that what is expected from "deprecation logs"?

6

I ran into this post wheen googling about the same issue as Dan seems to be having regarding deprecation logging: When setting logger.deprecation.level = warn in log4j2.properties we're getting the error "ConfigurationException: No name attribute provided for Logger deprecation" (we're also planning to upgrade from 6.8 to 7.x).

We have always been seeing the warning messages about deprectated functionality when running docker log. However, from what I can understand based on the documentation, I'm expecting to see a separate file for the deprecation log messages (my emphasis):

This will create a daily rolling deprecation log file in your log directory. Check this file regularly, especially when you intend to upgrade to a new major version.

But I see no such file. We can of course go through the "regular" log file and search for deprecation messages, but it kind of defeats the purpose of having a dedicated setting for deprecation logging. Especially when the documentation encourages you to check this file regularly...

Could @spinscale or somebody else from the Elastic team verify if this is working as intended (and the documentation ought to be updated/clarified), or if I'm missing something?

7

The logging is different based on the distribution you are using. See the different log4j files for

Does that make sense to you? If it is not obvious, please open a github issue so the docs get fixed.

Thank you!

1 Like
8

Thank you for pointing that out to me Alexander. The differences in configuration is obvious when you see it, but given how "simple" it appeared to be according to the documentation, it wasn't all the clear... I'll open a GitHub issue suggesting an improvement.

And in case anybody else is looking for a solution to this exact issue, I solved it by copying these lines [37-52] into the log4j2.properties that I've mounted for my Docker image for Elasticsearch.

Edit: Link to GitHub issue, for future references.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.