Hi,
I'm attempting to run the Elasticsearch/Kibana stack along with elastic-agent as a Fleet Server and APM Server via Docker Compose in order that I may have a complete local development setup that I can spin up and down.
It seems that with newer versions security is required and thus once enabling that, it makes the setup/configuration without using the UI much more difficult and documentation must be pieced together using the docs (spread out across all the different components) combined with forum posts.
But I have yet to find a single posting that synthesizes all the missing pieces in one place, with an out-of-the-box approach.
This is what I currently have:
version: "3.7"
services:
filebeat:
container_name: filebeat-container
depends_on:
- elasticsearch
deploy:
mode: global
user: root
image: filebeat-oss:7.17.15
environment:
strict.perms: false
ENVIRONMENT: local-dev
HOST: ${HOSTNAME}
ES_HOST: es-container:9200
networks:
- fek-network
volumes:
- /tmp/vcr-cloud/json-log:/mnt/vcr-cloud/json-log:ro
- ${PWD}/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml
restart: always
logging:
driver: "json-file"
options:
max-file: "5"
max-size: "2m"
elasticsearch:
container_name: es-container
image: elasticsearch/elasticsearch:7.17.15
environment:
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
- bootstrap.memory_lock=true
- "discovery.type=single-node"
- xpack.security.enabled=true
- xpack.security.authc.api_key.enabled=true
- "xpack.security.authc.realms.file.file1.order=0"
- "xpack.security.authc.realms.native.native1.order=1"
restart: always
networks:
- fek-network
ports:
- 9200:9200
- 9300:9300
volumes:
- local-elasticsearch:/usr/share/elasticsearch/data
- "${PWD}/configuration/roles.yml:/usr/share/elasticsearch/config/roles.yml"
- "${PWD}/configuration/users:/usr/share/elasticsearch/config/users"
- "${PWD}/configuration/users_roles:/usr/share/elasticsearch/config/users_roles"
kibana:
container_name: kb-container
image: kibana/kibana:7.17.15
environment:
- ELASTICSEARCH_HOSTS=http://es-container:9200
- ELASTICSEARCH_USERNAME=$ADMIN_USER
- "ELASTICSEARCH_PASSWORD=$ADMIN_PWD"
- XPACK_FLEET_AGENTS_FLEET_SERVER_HOSTS=["http://fleet-server:8220"]
- XPACK_FLEET_AGENTS_ELASTICSEARCH_HOSTS=["http://es-container:9200"]
networks:
- fek-network
depends_on:
- elasticsearch
ports:
- 5601:5601
- 8449:443
volumes:
- ${PWD}/kibana.yml:/usr/share/kibana/config/kibana.yml
fleet-server:
container_name: fleet-server
image: beats/elastic-agent:7.17.15
environment:
- FLEET_SERVER_ENABLE=true
- FLEET_SERVER_ELASTICSEARCH_HOST=http://es-container:9200
- "FLEET_SERVER_POLICY_ID=fleet-server-apm"
- FLEET_SERVER_ELASTICSEARCH_USERNAME=$ADMIN_USER
- "FLEET_SERVER_ELASTICSEARCH_PASSWORD=$ADMIN_PWD"
- FLEET_URL=http://fleet-server:8220
- KIBANA_FLEET_SETUP=true
- KIBANA_FLEET_HOST=http://kb-container:5601
- KIBANA_FLEET_USERNAME=$ADMIN_USER
- "KIBANA_FLEET_PASSWORD=$ADMIN_PWD"
networks:
- fek-network
depends_on:
- elasticsearch
- kibana
ports:
- 8220:8220
- 8200:8200
networks:
fek-network:
volumes:
local-elasticsearch:
external: true
I saw elsewhere that if I have the elastic-agent configured properly (according to what I have above) AND I have the policy correct (pasted below), then the elastic-agent will be both a fleet server and an apm server.
xpack.fleet.packages:
- name: fleet_server
version: latest
- name: apm
version: latest
xpack.fleet.agentPolicies:
- name: Fleet Server (APM)
id: fleet-server-apm
is_default_fleet_server: true
is_managed: false
namespace: default
package_policies:
- name: Fleet Server policy
id: default-fleet-server
package:
name: fleet_server
- name: APM policy
package:
name: apm
inputs:
- type: apm
enabled: true
vars:
- name: host
value: "localhost:8200"
- name: url
value: "http://localhost:8200"
However, my apm client agent (Java) is not able to connect to localhost:8200
Although, when I go inside Kibana to Add the APM Integration, there is a UI test that shows I do have an APM server running.
Do I need to add an apm-server service/container as well? Where are the docs that explain all this???