Logging configuration issues in Kibana 7.16

Elastic Stack Kibana
1

Kibana 7.16

END GOAL:

  • All Kibana logging options can be configured from kibana.yml
  • All Kibana logs should be visible with journalctl. (they stream to journald)

BACKGROUND:

  • The below issues were discovered after I realized that Kibana was creating a massive log file in /var/log/kibana/kibana.log

  • Kibana was installed through the official apt repo:
    https://artifacts.elastic.co/packages/7.x/apt stable main

  • Kibana installed using: apt install kibana

  • I am using the official documentation for 7.16 here: Configure Kibana | Kibana Guide [7.16] | Elastic

FIRST ISSUE:

Provided systemd unit file has logging config hard-coded in the service file.

Snippet from /etc/systemd/system/kibana.service that was installed with the apt package:

ExecStart=/usr/share/kibana/bin/kibana --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kibana.pid"

This creates an issue when you want to set logging params with in kibana.yml since this is essentially overriding at the command line.

I manually removed logging.dest= from the unit file and my logs started streaming to journald as expected.

Questions for the Forum about the first issue:

  • Why is any logging config hard-coded in this service file? Especially sending to a flat file for any system running systemd one would want the journal to handle logging.
  • Can I override this without having to modify the supplied unit file? Modifying or replacing it brings up compatibility concerns or overwrites if kibana is upgraded through apt. My site has several ELK servers globally and ongoing maintenance is a concern

SECOND ISSUE:

Logging options are not working as documented, but legacy options are

Using the official documentation Configure Kibana | Kibana Guide [7.16] | Elastic the config item for setting logging level should be defined as logging.root.level

Contents of kibana.yml

server.publicBaseUrl: "https://kibana.mysite.example.com"
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
 
logging:
  root:
    level: "warn"

I've also tried in-line: logging.root.level: "warn" and have tried quoted and unquoted "warn" vs warn

This does not work. Every web request is being logged and tons of noise. One refresh of the Kibana dashboard creates many lines in the log file.

However, I found in an "old" config document for version 6.8 Configuring Kibana | Kibana Guide [6.8] | Elastic that there was a config param logging.quiet

server.publicBaseUrl: "https://kibana.mysite.example.com"
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
 
logging.quiet: true

So I experimented and used this 6.8 config param on my 7.16 Kibana instance and IT WORKED -- Logs were no longer noisy.

I have confirmed that I'm indeed using 7.16, and not 6.x:

# /usr/share/kibana/bin/kibana --allow-root --version
7.16.3

Questions for the Forum about the second issue:

  • What is wrong with my formatting of logging.root.level that's causing it to not be honored?
  • Is there a reason why logging.quiet is honored in Kibana 7.16 even though it's not documented?

Thanks!!

2 Likes
2

@jbudz / @LeeDr can you please grab this question.

Thanks,
Bhavya

3

I think the first issue was a bug which is fixed in 8.0.0 release in this PR https://github.com/elastic/kibana/pull/98213

We may need someone else to comment on the second issue.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.