I have "successfully" enrolled elastic agent.
Kibana shows that the agent is Healthy for some time, until Endpoint Security starts communication with Elasticsearch.
In logs I get following error.
{"@timestamp":"2023-02-14T19:13:26.867416801Z","agent":{"id":"4f9ee333-a112-4417-a95b-c4d04931683b","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":327,"name":"Http.cpp"}}},"message":"Http.cpp:327 CURL error 60: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: unable to get local issuer certificate]","process":{"pid":3875,"thread":{"id":3881}}}
{"@timestamp":"2023-02-14T19:13:26.867492161Z","agent":{"id":"4f9ee333-a112-4417-a95b-c4d04931683b","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":93,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:93 Elasticsearch connection is down","process":{"pid":3875,"thread":{"id":3881}}}
I started elastic-agent with following command:
sudo ./elastic-agent install --url=https://10.0.0.9:8220 --enrollment-token=eXF4MVQ0WUJIT2hxWjN2VWVuTDU6bGstSHZmczlUVkNaem9LYjhoY1Awdw== --certificate-authorities=/home/user/Desktop/ca.crt --insecure
Everything except elastic-agent, running on external ubuntu machine, runs in docker.
My docker-compose.yml file
version: "3.8"
services:
setup:
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- certs:/usr/share/elasticsearch/config/certs
user: "0"
command: >
bash -c '
if [ ! -f config/certs/ca.zip ]; then
echo "Creating CA";
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f config/certs/certs.zip ]; then
echo "Creating certs";
echo -ne \
"instances:\n"\
" - name: esnode1\n"\
" dns:\n"\
" - esnode1\n"\
" - localhost\n"\
" ip:\n"\
" - 10.0.0.9\n"\
> config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
echo "Setting file permissions"
chown -R root:root config/certs;
find . -type d -exec chmod 750 \{\} \;;
find . -type f -exec chmod 640 \{\} \;;
echo "Waiting for Elasticsearch availability";
until curl -s --cacert config/certs/ca/ca.crt https://10.0.0.9:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
echo "Setting kibana_system password";
until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://10.0.0.9:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
echo "Good to go!";
'
esnode1:
networks:
- esnet
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
depends_on:
- setup
volumes:
- certs:/usr/share/elasticsearch/config/certs
- esnode1-data:/usr/share/elasticsearch/data
ports:
- 9200:9200
environment:
- node.name=esnode1
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- bootstrap.memory_lock=true
- discovery.type=single-node
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/esnode1/esnode1.key
- xpack.security.http.ssl.certificate=certs/esnode1/esnode1.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.http.ssl.verification_mode=none
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/esnode1/esnode1.key
- xpack.security.transport.ssl.certificate=certs/esnode1/esnode1.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=none
ulimits:
memlock:
soft: -1
hard: -1
kibana:
networks:
- esnet
depends_on:
- esnode1
image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
volumes:
- certs:/usr/share/kibana/config/certs
- kibana-data:/usr/share/kibana/data
ports:
- 5601:5601
environment:
- SERVERNAME=kibana
- ELASTICSEARCH_HOSTS=https://10.0.0.9:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
- XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${KIBANA_ENCRYPTIONKEY}
- SSL_VERIFICATIONMODE=none
fleet:
networks:
- esnet
depends_on:
- esnode1
image: docker.elastic.co/beats/elastic-agent:${STACK_VERSION}
volumes:
- certs:/usr/share/elastic-agent/config/certs
- fleet-data:/usr/share/elastic-agent/data
ports:
- 8220:8220
- 1234:1234
container_name: fleet
tty: true
user: root # note, synthetic browser monitors require this set to `elastic-agent`
environment:
- SSL_VERIFICATIONMODE=none
- FLEET_SERVER_ENABLE=true
- FLEET_SERVER_ELASTICSEARCH_HOST=https://10.0.0.9:9200
- FLEET_SERVER_SERVICE_TOKEN=${FLEET_SERVICE_TOKEN}
- FLEET_SERVER_ELASTICSEARCH_CA=/usr/share/elastic-agent/config/certs/ca/ca.crt
- KIBANA_FLEET_CA=/usr/share/elastic-agent/config/certs/ca/ca.crt
- FLEET_SERVER_POLICY_ID=fleet-server-policy
- FLEET_URL=https://10.0.0.9:8220
networks:
esnet:
driver: bridge
volumes:
certs:
driver: local
esnode1-data:
driver: local
kibana-data:
driver: local
fleet-data:
driver: local
running
curl https://10.0.0.9:9200 -u elastic:tmEuLZM9Ho1L6jPD2t7G --cacert ca.crt
works fine and returns
{
"name" : "esnode1",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "52xv7br-QM24_RMnvvCn1A",
"version" : {
"number" : "8.6.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "180c9830da956993e59e2cd70eb32b5e383ea42c",
"build_date" : "2023-01-24T21:35:11.506992272Z",
"build_snapshot" : false,
"lucene_version" : "9.4.2",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
Note that ca.crt is the same certificate as the one in the ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES variable.
These are my fleet settings in kibana.
Any ideas what's wrong?