Kibana 7.16
END GOAL:
- All Kibana logging options can be configured from
kibana.yml - All Kibana logs should be visible with journalctl. (they stream to journald)
BACKGROUND:
-
The below issues were discovered after I realized that Kibana was creating a massive log file in
/var/log/kibana/kibana.log -
Kibana was installed through the official apt repo:
https://artifacts.elastic.co/packages/7.x/apt stable main -
Kibana installed using:
apt install kibana -
I am using the official documentation for 7.16 here: Configure Kibana | Kibana Guide [7.16] | Elastic
FIRST ISSUE:
Provided systemd unit file has logging config hard-coded in the service file.
Snippet from /etc/systemd/system/kibana.service that was installed with the apt package:
ExecStart=/usr/share/kibana/bin/kibana --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kibana.pid"
This creates an issue when you want to set logging params with in kibana.yml since this is essentially overriding at the command line.
I manually removed logging.dest= from the unit file and my logs started streaming to journald as expected.
Questions for the Forum about the first issue:
- Why is any logging config hard-coded in this service file? Especially sending to a flat file for any system running systemd one would want the journal to handle logging.
- Can I override this without having to modify the supplied unit file? Modifying or replacing it brings up compatibility concerns or overwrites if kibana is upgraded through apt. My site has several ELK servers globally and ongoing maintenance is a concern
SECOND ISSUE:
Logging options are not working as documented, but legacy options are
Using the official documentation Configure Kibana | Kibana Guide [7.16] | Elastic the config item for setting logging level should be defined as logging.root.level
Contents of kibana.yml
server.publicBaseUrl: "https://kibana.mysite.example.com"
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
logging:
root:
level: "warn"
I've also tried in-line: logging.root.level: "warn" and have tried quoted and unquoted "warn" vs warn
This does not work. Every web request is being logged and tons of noise. One refresh of the Kibana dashboard creates many lines in the log file.
However, I found in an "old" config document for version 6.8 Configuring Kibana | Kibana Guide [6.8] | Elastic that there was a config param logging.quiet
server.publicBaseUrl: "https://kibana.mysite.example.com"
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
logging.quiet: true
So I experimented and used this 6.8 config param on my 7.16 Kibana instance and IT WORKED -- Logs were no longer noisy.
I have confirmed that I'm indeed using 7.16, and not 6.x:
# /usr/share/kibana/bin/kibana --allow-root --version
7.16.3
Questions for the Forum about the second issue:
- What is wrong with my formatting of
logging.root.levelthat's causing it to not be honored? - Is there a reason why
logging.quietis honored in Kibana 7.16 even though it's not documented?
Thanks!!