I have an existing ELK stack with logs, metrics and APM all reporting in successfully for a couple of years now.
We recently added a Cassandra cluster to our application, and I'd like to start getting logs and metrics to it. I could set up Filebeat/Metricbeat the way we have for other services, but I see there's an official Cassandra / Jolokia integration available - but I'm having trouble getting it to work.
In cassandra-env.sh
I've added
JVM_OPTS="$JVM_OPTS -javaagent:/usr/share/cassandra/lib/jolokia-jvm-1.7.2.jar=port=9000,host=0.0.0.0"
I can curl that from the local host, and I get a message back, e.g.
curl 'localhost:9000/jolokia/?ignoreErrors=true&canonicalNaming=false'
{
"request": {
"type": "version"
},
"value": {
"agent": "1.7.1",
"protocol": "7.2",
"config": {
"listenForHttpService": "true",
"maxCollectionSize": "0",
"authIgnoreCerts": "false",
...
I've also set up the APM agent, yaml:
id: 2016d7cc-135e-5583-9758-3ba01f5a06e5
revision: 3
outputs:
default:
type: elasticsearch
hosts:
- 'https://...:9200'
username: 'elastic'
password: '...'
output_permissions:
default:
_elastic_agent_monitoring:
indices:
- names:
- logs-elastic_agent.apm_server-default
...
Everything seems like its working. There are log files getting generated on the Cassandra instance in /opt/Elastic/Agent/data/elastic-agent-bc2cbf/logs
for filebeat and metricbeat.
Switching back over to Kibana, I added the Cassandra integration. Some of the input options are non-obvious to me, but essentially:
Integration name: cassandra-1
Collect System Logs from Cassandra (switch on)
Cassandra logs (switch on)
Paths /var/log/cassandra/system.log
Collect Cassandra metrics (switch on)
Settings
hosts: ....
Cassandra metrics (switched on)
Period 10s
Everything seems fine - ELK stack gets some log and metric indexes added (logs-* and metrics-*), but they are completely empty. Nothing populates.
How can I try and debug this?