Error in fleet-server, kibana, elastic search

soularius · September 10, 2024, 11:38pm

Issue when trying to connect Fleet Server with Elasticsearch in Docker

I am setting up an Elastic Stack environment in Docker, including containers for Elasticsearch, Kibana, and Elastic Agent with Fleet Server enabled. However, I have encountered an issue where the Fleet Server is not connecting properly to Elasticsearch, and the logs show several errors.
Container Configuration

Here is the relevant configuration for my containers (credentials and tokens replaced for security reasons):

yaml

elasticsearch:
  container_name: "elasticsearch"
  image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
  environment:
    - discovery.type=single-node
    - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    - ELASTIC_PASSWORD=<replace>
  ports:
    - "9200:9200"
  volumes:
    - elasticsearch-service-data:/usr/share/elasticsearch/data
    - elasticsearch-service-certs:/usr/share/elasticsearch/config/certs
    - ./elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
  networks:
    - bridge-net

kibana:
  image: docker.elastic.co/kibana/kibana:8.15.0
  container_name: kibana
  environment:
    - SERVERNAME=kibana
    - ELASTICSEARCH_HOSTS=["https://elasticsearch:9200"]
    - ELASTIC_USERNAME=kibana_user
    - ELASTICSEARCH_PASSWORD=<replace>
  volumes:
    - kibana-data:/usr/share/kibana/data
    - kibana-certs:/usr/share/kibana/config/certs
    - ./kibana/kibana.yml:/usr/share/kibana/config/kibana.yml
  ports:
    - "5601:5601"
  networks:
    - bridge-net

elastic-agent:
  image: docker.elastic.co/beats/elastic-agent-complete:8.15.1
  container_name: elastic-agent
  environment:
    - URL=https://kibana:8220
    - FLEET_ENROLLMENT_TOKEN=<replace>
    - CERTIFICATE_AUTHORITIES=/usr/share/elastic-agent/config/certs/chain.pem
    - FLEET_CA=/usr/share/elastic-agent/config/certs/fullchain.pem
    - FLEET_SERVER_ENABLE=true
    - FLEET_SERVER_PORT=8220
    - FLEET_SERVER_ES=https://dominio.com:9200
    - FLEET_SERVER_ES_CA=/usr/share/elastic-agent/config/certs/chain.pem
    - FLEET_SERVER_CERT=/usr/share/elastic-agent/config/certs/fullchain.pem
    - FLEET_SERVER_CERT_KEY=/usr/share/elastic-agent/config/certs/privkey.pem

   - FLEET_SERVER_ES_CERT=/usr/share/elastic-agent/config/certs/fullchain.pem
   - FLEET_SERVER_ES_CERT_KEY=/usr/share/elastic-agent/config/certs/privkey.pem
   - FLEET_SERVER_CLIENT_AUTH=required
      
   - KIBANA_FLEET_SETUP=1
   - KIBANA_FLEET_HOST=https://dominio.com:5601
   - KIBANA_FLEET_USERNAME="elastic"
   - KIBANA_FLEET_PASSWORD="PASS"
   - KIBANA_FLEET_CA=/usr/share/elastic-agent/config/certs/chain.pem

   - KIBANA_HOST=https://dominio.com:5601
   - KIBANA_USERNAME="elastic"
   - KIBANA_PASSWORD="PASS"
   - KIBANA_CA=/usr/share/elastic-agent/config/certs/chain.pem

   - ELASTICSEARCH_HOST=https://dominio:9200
   - ELASTICSEARCH_USERNAME="kibana_user"
   - ELASTICSEARCH_PASSWORD="PASS"
   - ELASTICSEARCH_CA=/usr/share/elastic-agent/config/certs/chain.pem
  volumes:
    - ./fleet/config/certs/:/usr/share/elastic-agent/config/certs/
  ports:
    - "8220:8220"
  networks:
    - bridge-net

Errors Encountered

Logs in Kibana:

[2024-09-10T22:46:31.887+00:00][INFO ][plugins.fleet] Secrets storage is disabled as minimum fleet server version has not been met

Logs in Elastic Agent (Fleet Server):

{"log.level":"info","@timestamp":"2024-09-10T23:34:57.648Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":90},"message":"Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: use of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
Error: context canceled
{"log.level":"error","@timestamp":"2024-09-10T23:23:28.840Z","message":"Fleet Server failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"failed version compatibility check with elasticsearch: dial tcp 209.126.6.195:9200: i/o timeout","@timestamp":"2024-09-10T23:23:28.84Z","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-10T23:23:28.841Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: dial tcp 209.126.6.195:9200: i/o timeout","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-10T23:23:28.841Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: dial tcp 209.126.6.195:9200: i/o timeout","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}

kibana.yml

elasticsearch.hosts: "https://temp-elasticsearch-url:9200"
elasticsearch.username: "elastic"
elasticsearch.password: "TEMPORARY_PASSWORD"

monitoring.ui.container.elasticsearch.enabled: true

server.publicBaseUrl: "https://temp-kibana-url"
server.name: temp-kibana

elasticsearch.ssl:
  verificationMode: certificate
  key: /usr/share/kibana/config/certs/privkey.pem # issued by Let's Encrypt
  certificate: /usr/share/kibana/config/certs/fullchain.pem # issued by Let's Encrypt
  certificateAuthorities: /usr/share/kibana/config/certs/chain.pem

server.ssl:
  enabled: true
  key: /usr/share/kibana/config/certs/privkey.pem # issued by Let's Encrypt
  certificate: /usr/share/kibana/config/certs/fullchain.pem # issued by Let's Encrypt
  certificateAuthorities: /usr/share/kibana/config/certs/chain.pem

xpack.reporting.encryptionKey: "TEMPORARY_ENCRYPTION_KEY"
xpack.security.encryptionKey: "TEMPORARY_ENCRYPTION_KEY"
xpack.encryptedSavedObjects.encryptionKey: "TEMPORARY_ENCRYPTION_KEY"
xpack.screenshotting.browser.chromium.disableSandbox: false
xpack.reporting.roles.enabled: false
xpack.fleet.enabled: true

telemetry.enabled: false

server.port: 5601
server.host: "0.0.0.0"

elasticsearch.yml

http.port: 9200

cluster.name: name
node.name: name-1
network.host: "0.0.0.0"
discovery.type: single-node

xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.security.transport.ssl.enabled: true

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  key: /usr/share/elasticsearch/config/certs/privkey.pem
  certificate: /usr/share/elasticsearch/config/certs/fullchain.pem
  certificate_authorities: /usr/share/elasticsearch/config/certs/chain.pem

xpack.security.http.ssl:
  enabled: true
  verification_mode: certificate
  key: /usr/share/elasticsearch/config/certs/privkey.pem
  certificate: /usr/share/elasticsearch/config/certs/fullchain.pem
  certificate_authorities: /usr/share/elasticsearch/config/certs/chain.pem

ingest.geoip.downloader.enabled: false

I have also checked the status of elastic-agent

elastic-agent@7e798926c7ce:~$ ./elastic-agent status
┌─ fleet
│  └─ status: (STARTING) 
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   └─ fleet-server-default
      ├─ status: (HEALTHY) Healthy: communicating with pid '23'
      ├─ fleet-server-default
      │  └─ status: (STARTING) Starting
      └─ fleet-server-default-fleet-server
         └─ status: (STARTING) Starting

But the process never leaves this point.

soularius · September 10, 2024, 11:41pm

Added fleet

leandrojmp · September 11, 2024, 1:10am

This error means that the Agent cannot connect with Elasticsearch, this is a connectivity issue you need to validate that your containers are able to connect to your Elasticsearch.

You are running everything inside a docker compose, any reason to use a public dns?

In your Kibana container you are using https://elasticsearch:9200 as the host, try to do the same in your Elastic Agent container in both the hosts for elasticsearch and kibana.

Just replace https://dominio.com:9200 with https://elasticsearch:9200 and https://dominio.com:5601 with https://kibana:5601.

soularius · September 11, 2024, 3:49am

In fact, I also tried it this way but the error continues and it is not a network problem or communication between containers since I make short requests between containers and they work correctly.

elastic-agent:
    image: docker.elastic.co/beats/elastic-agent-complete:8.15.1
    container_name: elastic-agent
    environment:
      - URL=https://dominio.com:8220
      - FLEET_ENROLLMENT_TOKEN="TOKEN"
      - CERTIFICATE_AUTHORITIES=/usr/share/elastic-agent/config/certs/chain.pem
      - FLEET_CA=/usr/share/elastic-agent/config/certs/fullchain.pem

      - FLEET_URL=https://dominio.com:8220
      - FLEET_ENROLL=1
      - FLEET_SERVER_ENABLE=true
      - FLEET_SERVER_PORT=8220

      - FLEET_SERVER_ES=https://elasticsearch:9200 
      - FLEET_SERVER_ES_CA=/usr/share/elastic-agent/config/certs/chain.pem

      - FLEET_SERVER_SERVICE_TOKEN="TOKEN"
      - FLEET_SERVER_POLICY=fleet-server-policy      
      - FLEET_SERVER_CERT=/usr/share/elastic-agent/config/certs/fullchain.pem
      - FLEET_SERVER_CERT_KEY=/usr/share/elastic-agent/config/certs/privkey.pem


      - FLEET_SERVER_ES_CERT=/usr/share/elastic-agent/config/certs/fullchain.pem
      - FLEET_SERVER_ES_CERT_KEY=/usr/share/elastic-agent/config/certs/privkey.pem
      - FLEET_SERVER_CLIENT_AUTH=required
      
      - KIBANA_FLEET_SETUP=1
      - KIBANA_FLEET_HOST=https://kibana:5601
      - KIBANA_FLEET_USERNAME="elastic"
      - KIBANA_FLEET_PASSWORD="PASS"
      - KIBANA_FLEET_CA=/usr/share/elastic-agent/config/certs/chain.pem

      - KIBANA_HOST=https://kibana:5601
      - KIBANA_USERNAME="elastic"
      - KIBANA_PASSWORD="PASS"
      - KIBANA_CA=/usr/share/elastic-agent/config/certs/chain.pem

      - ELASTICSEARCH_HOST=https://elasticsearch:9200 
      - ELASTICSEARCH_USERNAME="kibana_user"
      - ELASTICSEARCH_PASSWORD="PASS"
      - ELASTICSEARCH_CA=/usr/share/elastic-agent/config/certs/chain.pem

    volumes:
      - logstash-certs:/usr/share/elastic-agent/config/certs
      #- ./fleet/config/certs/:/usr/share/elastic-agent/config/certs/
    ports:
      - "8220:8220"  # Puerto para el Fleet Server
    networks:
      - bridge-net
      - dev1_laravel-search-history-service-net
      - dev2_laravel-search-history-service-net
      - prod_laravel-search-history-service-net
      - qa_laravel-search-history-service-net
    mem_limit: 2G

Logs elastic-agent

Policy selected for enrollment: fleet-server-policy

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.334Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).startAgent","file.name":"cmd/enroll_cmd.go","file.line":655},"message":"Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.404Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":195},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":28,"agent.version":"8.15.1","agent.unprivileged":true,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.613Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":144},"message":"agent is not upgradable, not starting watcher","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.615Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":284},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.616Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":65},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.629Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":71},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["windows/metrics","prometheus/metrics","aws-s3","azure-eventhub","filestream","websocket","kibana/metrics","uwsgi/metrics","osquery","packet","cloudbeat/vuln_mgmt_aws","enterprisesearch/metrics","kafka/metrics","cometd","lumberjack","netflow","syslog","udp","winlog","rabbitmq/metrics","stan/metrics","docker","gcp-pubsub","gcs","http_endpoint","logstash/metrics","apm","pf-elastic-collector","container","journald","linux/metrics","http/metrics","nats/metrics","endpoint","pf-host-agent","salesforce","unix","beat/metrics","containerd/metrics","mssql/metrics","nginx/metrics","vsphere/metrics","pf-elastic-symbolizer","cloudbeat/cis_gcp","cel","etw","synthetics/http","fleet-server","aws-cloudwatch","cloudfoundry","tcp","synthetics/icmp","activemq/metrics","entity-analytics","aws/metrics","syncgateway/metrics","gcp/metrics","cloud_defend/control","cloudbeat/cis_aws","audit/auditd","benchmark","o365audit","sql/metrics","cloudbeat","log","mqtt","mysql/metrics","awsfargate/metrics","oracle/metrics","haproxy/metrics","cloudbeat/cis_eks","audit/system","kubernetes/metrics","iis/metrics","memcached/metrics","zookeeper/metrics","traefik/metrics","cloudbeat/cis_azure","audit/file_integrity","azure-blob-storage","kafka","redis","mongodb/metrics","system/metrics","cloudbeat/asset_inventory_aws","cloudfoundry/metrics","apache/metrics","httpjson","synthetics/tcp","docker/metrics","postgresql/metrics","redis/metrics","azure/metrics","jolokia/metrics","cloudbeat/cis_k8s","synthetics/browser","elasticsearch/metrics","etcd/metrics","statsd/metrics"],"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.629Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/capabilities.LoadFile","file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.629Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":77},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.629Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":92},"message":"Loading baseline config from /usr/share/elastic-agent/state/elastic-agent.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.839Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.NewManager","file.name":"runtime/manager.go","file.line":180},"message":"GRPC comms socket listening at localhost:6789","log":{"source":"elastic-agent"},"address":"localhost:6789","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.850Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":167},"message":"Parsed configuration and determined agent is in Fleet Server bootstrap mode","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.851Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/control/v2/server.(*Server).Start","file.name":"server/server.go","file.line":88},"message":"GRPC control socket listening at unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","log":{"source":"elastic-agent"},"address":"unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.852Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).Run","file.name":"runtime/manager.go","file.line":246},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.852Z","log.logger":"composable.providers.docker","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker.(*dynamicProvider).Run","file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.853Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":123},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.853Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:\"\", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc0024cc030), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:\"\", APIKey:\"\", SecretToken:\"\", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.853Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":90},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.853Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.853Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:44.961Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1272},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.143Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":627},"message":"Spawned new component fleet-server-default: Starting: spawned pid '44'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.143Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit fleet-server-default-fleet-server: Starting: spawned pid '44'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.143Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit fleet-server-default: Starting: spawned pid '44'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.167Z","message":"Boot fleet-server","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","version":"8.15.1","commit":"13e2f247","ppid":28,"exe":"/usr/share/elastic-agent/data/elastic-agent-ecab0c/components/fleet-server","ecs.version":"1.6.0","pid":44,"args":["--agent-mode","-E","logging.level=debug","-E","logging.to_stderr=true","-E","path.data=/usr/share/elastic-agent/state/data/run/fleet-server-default"],"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.167Z","message":"starting communication connection back to Elastic Agent","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.169Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1478},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"count":1},"outputs":{}},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.169Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1272},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.174Z","log.logger":"component.runtime.fleet-server-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":715},"message":"control checkin v2 protocol has chunking enabled","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.174Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":647},"message":"Component state changed fleet-server-default (STARTING->HEALTHY): Healthy: communicating with pid '44'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-09-11T03:44:45.177Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2024-09-11T03:44:45.177Z","message":"Failed Elasticsearch output configuration test, using bootstrap values.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","output":{"hosts":["localhost:9200"],"protocol":"https","proxy_disable":false,"proxy_headers":{},"service_token":"AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MjU5NjMxMzMzMDk6elBSZWhmV01TUmlfaWpDZUFjNng2UQ","ssl":{"certificate":"/usr/share/elastic-agent/config/certs/fullchain.pem","certificate_authorities":["/usr/share/elastic-agent/config/certs/chain.pem"],"key":"/usr/share/elastic-agent/config/certs/privkey.pem","verification_mode":"full"},"type":"elasticsearch"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.179Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","memory_mb":24027,"recommended_mb":16192,"ecs.version":"1.6.0","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.179Z","message":"Setting cache config options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"cfg":{"APIKeyJitter":300000000000,"APIKeyTTL":900000000000,"ActionTTL":300000000000,"ArtifactTTL":86400000000000,"EnrollKeyTTL":60000000000,"MaxCost":268435456,"NumCounters":1600000},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.182Z","message":"Starting","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.182Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","memory_mb":24027,"recommended_mb":16192,"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.182Z","message":"initial server configuration","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","new":{"Fleet":{"Agent":{"ID":"","Logging":{"Level":""},"Version":"8.15.1"},"Host":{"ID":"","Name":""}},"HTTP":{"Enabled":false,"Host":"localhost","Port":5066,"SecurityDescriptor":"","User":""},"Inputs":[{"Cache":{"APIKeyJitter":0,"APIKeyTTL":0,"ActionTTL":0,"ArtifactTTL":0,"EnrollKeyTTL":0,"MaxCost":0,"NumCounters":0},"Monitor":{"FetchSize":0,"PolicyDebounceTime":0,"PollTimeout":0},"Policy":{"ID":""},"Server":{"Bulk":{"FlushInterval":250000000,"FlushMaxPending":8,"FlushThresholdCount":2048,"FlushThresholdSize":1048576},"CompressionLevel":1,"CompressionThresh":1024,"GC":{"CleanupAfterExpiredInterval":"30d","ScheduleInterval":3600000000000},"Host":"0.0.0.0","Instrumentation":{"APIKey":"","APIKeyPath":"","Enabled":false,"Environment":"","GlobalLabels":"","Hosts":null,"SecretToken":"","SecretTokenPath":"","TLS":{"ServerCA":"","ServerCertificate":"","SkipVerify":false},"TransactionSampleRate":""},"InternalPort":8221,"Limits":{"AckLimit":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":2097152},"ActionLimit":{"Burst":100,"Interval":500000,"Max":0,"MaxBody":0},"ArtifactLimit":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":0},"CheckinLimit":{"Burst":4000,"Interval":500000,"Max":40000,"MaxBody":1048576},"DeliverFileLimit":{"Burst":40,"Interval":100000000,"Max":80,"MaxBody":0},"EnrollLimit":{"Burst":100,"Interval":10000000,"Max":200,"MaxBody":524288},"GetPGPKey":{"Burst":25,"Interval":5000000,"Max":50,"MaxBody":0},"MaxAgents":0,"MaxConnections":0,"MaxHeaderByteSize":8192,"PolicyLimit":{"Burst":1,"Interval":2000000,"Max":0,"MaxBody":0},"PolicyThrottle":0,"StatusLimit":{"Burst":200,"Interval":5000000,"Max":400,"MaxBody":0},"UploadChunkLimit":{"Burst":40,"Interval":3000000,"Max":80,"MaxBody":4194304},"UploadEndLimit":{"Burst":40,"Interval":2000000000,"Max":80,"MaxBody":1024},"UploadStartLimit":{"Burst":40,"Interval":2000000000,"Max":80,"MaxBody":5242880}},"PGP":{"Dir":"/usr/share/elastic-agent/data/elastic-agent-ecab0c/components/elastic-agent-upgrade-keys","UpstreamURL":"https://artifacts.elastic.co/GPG-KEY-elastic-agent"},"Port":8220,"Profiler":{"Bind":"localhost:6060","Enabled":false},"Runtime":{"GCPercent":0,"MemoryLimit":0},"StaticPolicyTokens":{"Enabled":false,"PolicyTokens":null},"TLS":{"CASha256":null,"CAs":["/usr/share/elastic-agent/config/certs/fullchain.pem"],"Certificate":{"Certificate":"/usr/share/elastic-agent/config/certs/fullchain.pem","Key":"[redacted]","Passphrase":"","PassphrasePath":""},"CipherSuites":null,"ClientAuth":"required","CurveTypes":null,"Enabled":null,"VerificationMode":"full","Versions":null},"Timeouts":{"CheckinJitter":30000000000,"CheckinLongPoll":300000000000,"CheckinMaxPoll":3600000000000,"CheckinTimestamp":30000000000,"Drain":10000000000,"Idle":30000000000,"Read":60000000000,"ReadHeader":5000000000,"Write":600000000000}},"Type":""}],"Logging":{"Files":null,"Level":"info","Pretty":false,"ToFiles":true,"ToStderr":true},"Output":{"Elasticsearch":{"Headers":null,"Hosts":["elasticsearch:9200"],"MaxConnPerHost":128,"MaxContentLength":104857600,"MaxRetries":3,"Path":"","Protocol":"https","ProxyDisable":false,"ProxyHeaders":{},"ProxyURL":"","ServiceToken":"[redacted]","ServiceTokenPath":"","TLS":{"CASha256":null,"CATrustedFingerprint":"","CAs":["/usr/share/elastic-agent/config/certs/chain.pem"],"Certificate":{"Certificate":"/usr/share/elastic-agent/config/certs/fullchain.pem","Key":"[redacted]","Passphrase":"","PassphrasePath":""},"CipherSuites":null,"CurveTypes":null,"Enabled":null,"Renegotiation":"never","VerificationMode":"full","Versions":null},"Timeout":90000000000},"Extra":null}},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.182Z","message":"starting server on configuration change","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.183Z","message":"Run bulker with options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","opts":{"apikeyMaxParallel":120,"apikeyMaxReqSize":104857600,"blockQueueSz":32,"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.201Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"elasticsearch_version":"8.15.0","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"8.15.1","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.210Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","index":".fleet-policies","ctx":"index monitor","@timestamp":"2024-09-11T03:44:45.21Z","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.211Z","message":"run policy monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","ctx":"policy agent monitor","ecs.version":"1.6.0","service.name":"fleet-server","burst":1,"event_rate":500,"@timestamp":"2024-09-11T03:44:45.21Z","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.211Z","message":"upload limits","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","limits":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":0},"maxFileSize":104857600,"@timestamp":"2024-09-11T03:44:45.21Z","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.211Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ctx":"index monitor","@timestamp":"2024-09-11T03:44:45.21Z","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","index":".fleet-actions","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.211Z","message":"upload limits","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","limits":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":0},"maxFileSize":104857600,"ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2024-09-11T03:44:45.211Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ctx":"policy leader manager","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.211Z","message":"server hard connection limiter disabled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.211Z","message":"server hard connection limiter disabled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.212Z","message":"Listening on 0.0.0.0:8220","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.212Z","message":"Listening on localhost:8221","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.730Z","message":"Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"@timestamp":"2024-09-11T03:44:45.73Z","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"DEGRADED","ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2024-09-11T03:44:45.730Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->DEGRADED): Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2024-09-11T03:44:45.730Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default (STARTING->DEGRADED): Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:46.337Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":805},"message":"Fleet Server - Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T03:44:46.349Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":518},"message":"Starting enrollment to URL: https://increci-kibana.opperweb.com:8220/","ecs.version":"1.6.0"}

leandrojmp · September 11, 2024, 4:25am

The error you shared in your first post was a network issue, you were using a public endpoint and your Elastic Agent container could not connect to it.

You can confirm it in this error line:

Error - failed version compatibility check with elasticsearch: dial tcp your-public-ip:9200: i/o timeout

In the new log you shared, you do not have this issue as you can check from this log line:

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.201Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"elasticsearch_version":"8.15.0","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"8.15.1","ecs.version":"1.6.0"}

And your Fleet Server seems to be listening as well:

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.212Z","message":"Listening on 0.0.0.0:8220","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

And it started the enrollment process:

{"log.level":"info","@timestamp":"2024-09-11T03:44:46.349Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":518},"message":"Starting enrollment to URL: https://your-public-domain:8220/","ecs.version":"1.6.0"}

It is not clear what is your issue now, it is not the same from your original post.

What is your current issue now? Please provide more logs.

You are still using public domains in some of the configuration in your compose, your containers may be having issues to connect to them.

Also, you may want to edit your posts to redact your domain.

soularius · September 11, 2024, 4:51am

Of course, initially there were 3 errors, but only 2 remain, which are the ones I mentioned:

soularius:

{"log.level":"error","@timestamp":"2024-09-11T03:44:45.177Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2024-09-11T03:44:45.177Z","message":"Failed Elasticsearch output configuration test, using bootstrap values.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","output":{"hosts":["localhost:9200"],"protocol":"https","proxy_disable":false,"proxy_headers":{},"service_token":"AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MjU5NjMxMzMzMDk6elBSZWhmV01TUmlfaWpDZUFjNng2UQ","ssl":{"certificate":"/usr/share/elastic-agent/config/certs/fullchain.pem","certificate_authorities":["/usr/share/elastic-agent/config/certs/chain.pem"],"key":"/usr/share/elastic-agent/config/certs/privkey.pem","verification_mode":"full"},"type":"elasticsearch"},"ecs.version":"1.6.0"}

leandrojmp · September 11, 2024, 4:56am

This error was already solved, check further in the logs and you will see this:

{"log.level":"info","@timestamp":"2024-09-11T03:44:45.201Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"elasticsearch_version":"8.15.0","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"8.15.1","ecs.version":"1.6.0"}

Your docker compose will not wait for the elasticsearch container to start before starting the other containers, so the elastic agent container may start before elasticsearch is ready to receive request, so this is expected.

If you check further in the logs you will see that it was able to connect to Elasticsearch.

This error here:

Failed Elasticsearch output configuration test, using bootstrap values.

Means that your Elasticsearch Output in the Fleet Settings UI is pointing to https://localhost:9200, you need to change it in the Feet Settings to point to https://elasticsearch:9200

Do you have any new errors?

soularius · September 11, 2024, 5:05am

I share my configuration with you. I have changed it to the name of the container but the error continues.

{"log.level":"error","@timestamp":"2024-09-11T05:00:35.415Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2024-09-11T05:00:35.415Z","message":"Failed Elasticsearch output configuration test, using bootstrap values.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"output":{"hosts":["localhost:9200"],"protocol":"https","proxy_disable":false,"proxy_headers":{},"service_token":"AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MjYwMzA0MzM3ODY6YVRYT3VZTGpTS3UyVFkteGt0V2otUQ","ssl":{"certificate":"/usr/share/elastic-agent/config/certs/fullchain.pem","certificate_authorities":["/usr/share/elastic-agent/config/certs/chain.pem"],"key":"/usr/share/elastic-agent/config/certs/privkey.pem","verification_mode":"full"},"type":"elasticsearch"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0"}

Also can't complete registration in kibana

This is now my status

# docker exec -it elastic-agent bash

elastic-agent@10ae7cd98e83:~$ elastic-agent status
┌─ fleet
│  └─ status: (STARTING) 
└─ elastic-agent
   ├─ status: (DEGRADED) 1 or more components/units in a degraded state
   └─ fleet-server-default
      ├─ status: (HEALTHY) Healthy: communicating with pid '24'
      ├─ fleet-server-default
      │  └─ status: (DEGRADED) Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)
      └─ fleet-server-default-fleet-server
         └─ status: (DEGRADED) Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)

leandrojmp · September 11, 2024, 5:12am

Did you change these 2 settings to use the container name?

- URL=https://increci-kibana.opperweb.com:8220`

- FLEET_URL=https://dominio.com:8220

You need to ignore this error, until your Fleet Server is Healthy, it is not Healthy, which means that it was not able to enroll itself.

soularius · September 11, 2024, 5:18am

Yes i change too


  elastic-agent:
    image: docker.elastic.co/beats/elastic-agent-complete:8.15.1
    container_name: elastic-agent
    environment:
      - URL=https://elastic-agent:8220
      - FLEET_ENROLLMENT_TOKEN="TOKEN"
      - CERTIFICATE_AUTHORITIES=/usr/share/elastic-agent/config/certs/chain.pem
      - FLEET_CA=/usr/share/elastic-agent/config/certs/fullchain.pem

      - FLEET_URL=https://elastic-agent:8220
      - FLEET_ENROLL=1
      - FLEET_SERVER_ENABLE=true
      - FLEET_SERVER_PORT=8220

      - FLEET_SERVER_ES=https://elasticsearch:9200 
      - FLEET_SERVER_ES_CA=/usr/share/elastic-agent/config/certs/chain.pem

      - FLEET_SERVER_SERVICE_TOKEN="TOKEN"
      - FLEET_SERVER_POLICY=fleet-server-policy      
      - FLEET_SERVER_CERT=/usr/share/elastic-agent/config/certs/fullchain.pem
      - FLEET_SERVER_CERT_KEY=/usr/share/elastic-agent/config/certs/privkey.pem


      - FLEET_SERVER_ES_CERT=/usr/share/elastic-agent/config/certs/fullchain.pem
      - FLEET_SERVER_ES_CERT_KEY=/usr/share/elastic-agent/config/certs/privkey.pem
      - FLEET_SERVER_CLIENT_AUTH=required
      
      - KIBANA_FLEET_SETUP=1
      - KIBANA_FLEET_HOST=https://kibana:5601
      - KIBANA_FLEET_USERNAME="elastic"
      - KIBANA_FLEET_PASSWORD="PASS"
      - KIBANA_FLEET_CA=/usr/share/elastic-agent/config/certs/chain.pem

      - KIBANA_HOST=https://kibana:5601
      - KIBANA_USERNAME="elastic"
      - KIBANA_PASSWORD="PASS"
      - KIBANA_CA=/usr/share/elastic-agent/config/certs/chain.pem

      - ELASTICSEARCH_HOST=https://elasticsearch:9200 
      - ELASTICSEARCH_USERNAME="kibana_user"
      - ELASTICSEARCH_PASSWORD="PASS"
      - ELASTICSEARCH_CA=/usr/share/elastic-agent/config/certs/chain.pem

    volumes:
      - logstash-certs:/usr/share/elastic-agent/config/certs
      #- ./fleet/config/certs/:/usr/share/elastic-agent/config/certs/
    ports:
      - "8220:8220"  # Puerto para el Fleet Server
    networks:
      - bridge-net
      - dev1_laravel-search-history-service-net
      - dev2_laravel-search-history-service-net
      - prod_laravel-search-history-service-net
      - qa_laravel-search-history-service-net
    #depends_on:
    #  - elasticsearch
    #  - kibana
    mem_limit: 2G

This is my full log:


{"log.level":"info","@timestamp":"2024-09-11T05:15:11.386Z","message":"Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","state":"DEGRADED","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:13.470Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":383},"message":"signal \"terminated\" received","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:13.470Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":395},"message":"Shutting down Elastic Agent and sending last events...","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:13.470Z","message":"On signal","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"@timestamp":"2024-09-11T05:15:13.47Z","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","sig":"terminated","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:13.671Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":404},"message":"Shutting down completed.","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:13.671Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Stop","file.name":"reload/reload.go","file.line":74},"message":"Stopping monitoring server","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:13.671Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":90},"message":"Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: use of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

Error: context canceled

For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.15/fleet-troubleshooting.html

{"log.level":"info","@timestamp":"2024-09-11T05:15:15.729Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":195},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":6,"agent.version":"8.15.1","agent.unprivileged":true,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:15.937Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":144},"message":"agent is not upgradable, not starting watcher","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:15.939Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":284},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:15.940Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":65},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:15.953Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":71},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["pf-elastic-collector","audit/auditd","cometd","etcd/metrics","prometheus/metrics","cloudbeat/cis_aws","azure-eventhub","benchmark","cel","log","o365audit","synthetics/tcp","haproxy/metrics","rabbitmq/metrics","gcs","kafka","redis","kibana/metrics","nginx/metrics","stan/metrics","pf-host-agent","lumberjack","enterprisesearch/metrics","uwsgi/metrics","mssql/metrics","cloud_defend/control","cloudbeat","cloudbeat/cis_gcp","azure-blob-storage","gcp-pubsub","winlog","docker/metrics","oracle/metrics","pf-elastic-symbolizer","aws-cloudwatch","etw","netflow","websocket","synthetics/http","elasticsearch/metrics","mongodb/metrics","system/metrics","apache/metrics","osquery","fleet-server","container","http_endpoint","journald","syslog","jolokia/metrics","apm","iis/metrics","memcached/metrics","cloudbeat/cis_k8s","audit/system","mqtt","logstash/metrics","mysql/metrics","postgresql/metrics","statsd/metrics","vsphere/metrics","aws-s3","docker","httpjson","synthetics/icmp","kubernetes/metrics","azure/metrics","cloudfoundry/metrics","packet","cloudbeat/vuln_mgmt_aws","endpoint","entity-analytics","tcp","syncgateway/metrics","cloudbeat/asset_inventory_aws","cloudfoundry","salesforce","unix","filestream","synthetics/browser","activemq/metrics","nats/metrics","cloudbeat/cis_eks","udp","redis/metrics","containerd/metrics","zookeeper/metrics","linux/metrics","windows/metrics","awsfargate/metrics","gcp/metrics","kafka/metrics","cloudbeat/cis_azure","beat/metrics","http/metrics","sql/metrics","audit/file_integrity","aws/metrics","traefik/metrics"],"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:15.953Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/capabilities.LoadFile","file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:15.953Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":77},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:15.953Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":92},"message":"Loading baseline config from /usr/share/elastic-agent/state/elastic-agent.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.138Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.NewManager","file.name":"runtime/manager.go","file.line":180},"message":"GRPC comms socket listening at localhost:6789","log":{"source":"elastic-agent"},"address":"localhost:6789","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.146Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":167},"message":"Parsed configuration and determined agent is in Fleet Server bootstrap mode","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.147Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/control/v2/server.(*Server).Start","file.name":"server/server.go","file.line":88},"message":"GRPC control socket listening at unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","log":{"source":"elastic-agent"},"address":"unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.148Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).Run","file.name":"runtime/manager.go","file.line":246},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.148Z","log.logger":"composable.providers.docker","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker.(*dynamicProvider).Run","file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.149Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":123},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.149Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:\"\", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc001779110), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:\"\", APIKey:\"\", SecretToken:\"\", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.149Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":90},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.149Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.149Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.257Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1272},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.349Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":627},"message":"Spawned new component fleet-server-default: Starting: spawned pid '22'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.350Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit fleet-server-default-fleet-server: Starting: spawned pid '22'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.350Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit fleet-server-default: Starting: spawned pid '22'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.371Z","message":"Boot fleet-server","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","version":"8.15.1","pid":22,"args":["--agent-mode","-E","logging.level=debug","-E","logging.to_stderr=true","-E","path.data=/usr/share/elastic-agent/state/data/run/fleet-server-default"],"service.type":"fleet-server","commit":"13e2f247","ppid":6,"exe":"/usr/share/elastic-agent/data/elastic-agent-ecab0c/components/fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.371Z","message":"starting communication connection back to Elastic Agent","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.377Z","log.logger":"component.runtime.fleet-server-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":715},"message":"control checkin v2 protocol has chunking enabled","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.377Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":647},"message":"Component state changed fleet-server-default (STARTING->HEALTHY): Healthy: communicating with pid '22'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-09-11T05:15:16.381Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","@timestamp":"2024-09-11T05:15:16.38Z","ecs.version":"1.6.0","ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2024-09-11T05:15:16.381Z","message":"Failed Elasticsearch output configuration test, using bootstrap values.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"output":{"hosts":["localhost:9200"],"protocol":"https","proxy_disable":false,"proxy_headers":{},"service_token":"AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MjYwMzA0MzM3ODY6YVRYT3VZTGpTS3UyVFkteGt0V2otUQ","ssl":{"certificate":"/usr/share/elastic-agent/config/certs/fullchain.pem","certificate_authorities":["/usr/share/elastic-agent/config/certs/chain.pem"],"key":"/usr/share/elastic-agent/config/certs/privkey.pem","verification_mode":"full"},"type":"elasticsearch"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.382Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","memory_mb":24027,"recommended_mb":16192,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.382Z","message":"Setting cache config options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","cfg":{"APIKeyJitter":300000000000,"APIKeyTTL":900000000000,"ActionTTL":300000000000,"ArtifactTTL":86400000000000,"EnrollKeyTTL":60000000000,"MaxCost":268435456,"NumCounters":1600000},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.383Z","message":"Starting","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.383Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","memory_mb":24027,"recommended_mb":16192,"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.384Z","message":"initial server configuration","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","new":{"Fleet":{"Agent":{"ID":"","Logging":{"Level":""},"Version":"8.15.1"},"Host":{"ID":"","Name":""}},"HTTP":{"Enabled":false,"Host":"localhost","Port":5066,"SecurityDescriptor":"","User":""},"Inputs":[{"Cache":{"APIKeyJitter":0,"APIKeyTTL":0,"ActionTTL":0,"ArtifactTTL":0,"EnrollKeyTTL":0,"MaxCost":0,"NumCounters":0},"Monitor":{"FetchSize":0,"PolicyDebounceTime":0,"PollTimeout":0},"Policy":{"ID":""},"Server":{"Bulk":{"FlushInterval":250000000,"FlushMaxPending":8,"FlushThresholdCount":2048,"FlushThresholdSize":1048576},"CompressionLevel":1,"CompressionThresh":1024,"GC":{"CleanupAfterExpiredInterval":"30d","ScheduleInterval":3600000000000},"Host":"0.0.0.0","Instrumentation":{"APIKey":"","APIKeyPath":"","Enabled":false,"Environment":"","GlobalLabels":"","Hosts":null,"SecretToken":"","SecretTokenPath":"","TLS":{"ServerCA":"","ServerCertificate":"","SkipVerify":false},"TransactionSampleRate":""},"InternalPort":8221,"Limits":{"AckLimit":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":2097152},"ActionLimit":{"Burst":100,"Interval":500000,"Max":0,"MaxBody":0},"ArtifactLimit":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":0},"CheckinLimit":{"Burst":4000,"Interval":500000,"Max":40000,"MaxBody":1048576},"DeliverFileLimit":{"Burst":40,"Interval":100000000,"Max":80,"MaxBody":0},"EnrollLimit":{"Burst":100,"Interval":10000000,"Max":200,"MaxBody":524288},"GetPGPKey":{"Burst":25,"Interval":5000000,"Max":50,"MaxBody":0},"MaxAgents":0,"MaxConnections":0,"MaxHeaderByteSize":8192,"PolicyLimit":{"Burst":1,"Interval":2000000,"Max":0,"MaxBody":0},"PolicyThrottle":0,"StatusLimit":{"Burst":200,"Interval":5000000,"Max":400,"MaxBody":0},"UploadChunkLimit":{"Burst":40,"Interval":3000000,"Max":80,"MaxBody":4194304},"UploadEndLimit":{"Burst":40,"Interval":2000000000,"Max":80,"MaxBody":1024},"UploadStartLimit":{"Burst":40,"Interval":2000000000,"Max":80,"MaxBody":5242880}},"PGP":{"Dir":"/usr/share/elastic-agent/data/elastic-agent-ecab0c/components/elastic-agent-upgrade-keys","UpstreamURL":"https://artifacts.elastic.co/GPG-KEY-elastic-agent"},"Port":8220,"Profiler":{"Bind":"localhost:6060","Enabled":false},"Runtime":{"GCPercent":0,"MemoryLimit":0},"StaticPolicyTokens":{"Enabled":false,"PolicyTokens":null},"TLS":{"CASha256":null,"CAs":["/usr/share/elastic-agent/config/certs/fullchain.pem"],"Certificate":{"Certificate":"/usr/share/elastic-agent/config/certs/fullchain.pem","Key":"[redacted]","Passphrase":"","PassphrasePath":""},"CipherSuites":null,"ClientAuth":"required","CurveTypes":null,"Enabled":null,"VerificationMode":"full","Versions":null},"Timeouts":{"CheckinJitter":30000000000,"CheckinLongPoll":300000000000,"CheckinMaxPoll":3600000000000,"CheckinTimestamp":30000000000,"Drain":10000000000,"Idle":30000000000,"Read":60000000000,"ReadHeader":5000000000,"Write":600000000000}},"Type":""}],"Logging":{"Files":null,"Level":"info","Pretty":false,"ToFiles":true,"ToStderr":true},"Output":{"Elasticsearch":{"Headers":null,"Hosts":["elasticsearch:9200"],"MaxConnPerHost":128,"MaxContentLength":104857600,"MaxRetries":3,"Path":"","Protocol":"https","ProxyDisable":false,"ProxyHeaders":{},"ProxyURL":"","ServiceToken":"[redacted]","ServiceTokenPath":"","TLS":{"CASha256":null,"CATrustedFingerprint":"","CAs":["/usr/share/elastic-agent/config/certs/chain.pem"],"Certificate":{"Certificate":"/usr/share/elastic-agent/config/certs/fullchain.pem","Key":"[redacted]","Passphrase":"","PassphrasePath":""},"CipherSuites":null,"CurveTypes":null,"Enabled":null,"Renegotiation":"never","VerificationMode":"full","Versions":null},"Timeout":90000000000},"Extra":null}},"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.384Z","message":"starting server on configuration change","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.385Z","message":"Run bulker with options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"opts":{"apikeyMaxParallel":120,"apikeyMaxReqSize":104857600,"blockQueueSz":32,"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.391Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"8.15.1","elasticsearch_version":"8.15.0","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.399Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","index":".fleet-policies","ctx":"index monitor","ecs.version":"1.6.0","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.399Z","message":"upload limits","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","limits":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":0},"maxFileSize":104857600,"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.399Z","message":"run policy monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","burst":1,"ctx":"policy agent monitor","event_rate":500,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.399Z","message":"upload limits","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"maxFileSize":104857600,"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","limits":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":0},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.399Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","index":".fleet-actions","ctx":"index monitor","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.399Z","message":"server hard connection limiter disabled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.399Z","message":"server hard connection limiter disabled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2024-09-11T05:15:16.399Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ctx":"policy leader manager","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.400Z","message":"Listening on localhost:8221","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","@timestamp":"2024-09-11T05:15:16.4Z","ecs.version":"1.6.0","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.400Z","message":"Listening on 0.0.0.0:8220","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","@timestamp":"2024-09-11T05:15:16.4Z","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.464Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1478},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"count":1},"outputs":{}},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.464Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1272},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:16.912Z","message":"Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"DEGRADED","ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2024-09-11T05:15:16.912Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->DEGRADED): Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2024-09-11T05:15:16.912Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default (STARTING->DEGRADED): Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:22.420Z","message":"Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"DEGRADED","@timestamp":"2024-09-11T05:15:22.42Z","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-09-11T05:15:27.930Z","message":"Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"state":"DEGRADED","@timestamp":"2024-09-11T05:15:27.93Z","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}

leandrojmp · September 11, 2024, 5:31am

Your fleet server is still having some issues to start and become healthy, not sure why.

I would recommend that you uncomment these lines:

    #depends_on:
    #  - elasticsearch
    #  - kibana

The fleet server needs to be started only after Elasticsearch and Kibana, if they are not up it may not be able to enroll.

Also, you are using a lot of environment variables that seems to be unnecessary, only a couple are required for the fleet server.

You can check this docker-compose.yml for a working example.

Try to adapt it to your configuration.

soularius · September 11, 2024, 5:37am

I understand, thank you very much. In any case, I have already searched and I cannot find many examples. I have read the documentation and have investigated a lot, but I cannot get them to connect. I know that they must have the dependencies, but the other containers are running and active, so I know that is not the problem.

10ae7cd98e83   docker.elastic.co/beats/elastic-agent-complete:8.15.1              "/usr/bin/tini -- /u…"   39 minutes ago   Up 18 minutes         0.0.0.0:8220->8220/tcp                                               elastic-agent
14f0c4f587d9   docker.elastic.co/kibana/kibana:8.15.0                             "/bin/tini -- /usr/l…"   22 hours ago     Up 20 hours           0.0.0.0:5601->5601/tcp                                               kibana
3ca88278b189   docker.elastic.co/elasticsearch/elasticsearch:8.15.0               "/bin/tini -- /usr/l…"   22 hours ago     Up 21 hours           0.0.0.0:9200->9200/tcp, 9300/tcp                                     elasticsearch

If you could give me any other idea of what the error could be, I would appreciate it.

This is my Status:

elastic-agent@837b203b00ed:~$ elastic-agent status
┌─ fleet
│  └─ status: (STARTING) 
└─ elastic-agent
   ├─ status: (DEGRADED) 1 or more components/units in a degraded state
   └─ fleet-server-default
      ├─ status: (HEALTHY) Healthy: communicating with pid '23'
      ├─ fleet-server-default
      │  └─ status: (DEGRADED) Running on policy with Fleet Server integration: 286f9bd4-34e8-4569-972e-65d7d5a1d31b; missing config fleet.agent.id (expected during bootstrap process)
      └─ fleet-server-default-fleet-server
         └─ status: (DEGRADED) Running on policy with Fleet Server integration: 286f9bd4-34e8-4569-972e-65d7d5a1d31b; missing config fleet.agent.id (expected during bootstrap process)

leandrojmp · September 11, 2024, 12:56pm

I would not assume anything like that.

Have you uncommented the lines? The Elastic Agent depends on Elasticsearch and Kibana, so you need to make sure that it will only start after the other containers have started, in fact you should try to implement a health check to guarantee that it will only start after Elasticsearch and Kibana are healthy.

For example, if the container starts, but cannot communicate with Elasticsearch, it may not try again.

Also, try to use the same version, you are using 8.15.1 for the agents and 8.15.0 for Elastic and Kibana, change that to use 8.15.1 as well.

I shared a working compose from another project, take a look and try to change yours based on that.

soularius · September 11, 2024, 9:58pm

Hi,

I've checked the repository you shared with me and followed the indicated settings, keeping the versions the same as 8.15.1 and adjusting some details in the configuration. However, I'm still facing the same error and I haven't been able to get it working properly.

I'm sharing the current configuration with you so that, if possible, you can take a look at it. I'm also including the logs with the debug mode enabled and the current status of the service.

I really appreciate the support and I'm sorry for any inconvenience this may cause.

Regards and thanks again.

  elastic-agent:
    image: docker.elastic.co/beats/elastic-agent-complete:8.15.1
    container_name: elastic-agent
    environment:
      - FLEET_ENROLL=1
      - FLEET_SERVER_POLICY_ID=ID
      - FLEET_SERVER_ENABLE=1
      - KIBANA_FLEET_SETUP=1
      - KIBANA_HOST=https://elastic-agent:8220
      - FLEET_URL=https://elastic-agent:8220
      - FLEET_SERVER_ELASTICSEARCH_HOST=https://elasticsearch:9200
      - FLEET_CA=/usr/share/elastic-agent/config/certs/fullchain.pem
      - KIBANA_FLEET_USERNAME="kibana_user"
      - KIBANA_FLEET_PASSWORD="PASS"
      - FLEET_SERVER_CERT=/usr/share/elastic-agent/config/certs/fullchain.pem
      - FLEET_SERVER_CERT_KEY=/usr/share/elastic-agent/config/certs/privkey.pem 
      - FLEET_SERVER_ELASTICSEARCH_CA=/usr/share/elastic-agent/config/certs/chain.pem
      - KIBANA_FLEET_CA=/usr/share/elastic-agent/config/certs/chain.pem
      - FLEET_SERVER_SERVICE_TOKEN=TOKEN
    volumes:
      - logstash-certs:/usr/share/elastic-agent/config/certs
      - ./elastic-agent/elastic-agent.yml:/usr/share/elastic-agent/elastic-agent.yml
    ports:
      - "8220:8220"  # Puerto para el Fleet Server
    networks:
      - bridge-net
      - dev1_laravel-search-history-service-net
      - dev2_laravel-search-history-service-net
      - prod_laravel-search-history-service-net
      - qa_laravel-search-history-service-net
    depends_on:
      - elasticsearch
      - kibana
    mem_limit: 2G

{"log.level":"debug","@timestamp":"2024-09-11T21:50:48.659Z","message":"init es","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","cluster.addr":["localhost:9200"],"cluster.maxConnsPersHost":128,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:48.659Z","message":"check version compatibility with elasticsearch","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"fleet_version":"8.15.1","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:48.661Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","@timestamp":"2024-09-11T21:50:48.66Z","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-09-11T21:50:48.661Z","message":"Failed Elasticsearch output configuration test, using bootstrap values.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","output":{"hosts":["localhost:9200"],"protocol":"https","proxy_disable":false,"proxy_headers":{},"service_token":"AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyLXJlbW90ZS90b2tlbi0xNzI2MDg4MDQ5Mzc5Ok9hV0pJbXEyUmZpNFhreWxwOVpidGc","ssl":{"certificate_authorities":["/usr/share/elastic-agent/config/certs/chain.pem"],"verification_mode":"full"},"type":"elasticsearch"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-11T21:50:48.661Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","memory_mb":24027,"recommended_mb":16192,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-11T21:50:48.683Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","fleet_version":"8.15.1","elasticsearch_version":"8.15.1","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:48.683Z","message":"Migrations started","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:48.683Z","message":"applying migration to v7.15","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:48.685Z","message":"migration AgentMetadata failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","error.message":"migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","http.request.body.content":"{\"query\":{\"bool\":{\"must_not\":{\"exists\":{\"field\":\"agent.id\"}}}},\"script\":\"ctx._source.agent = [:]; ctx._source.agent.id = ctx._id;\"}","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:48.685Z","message":"Migrations exited","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","error.message":"v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:48.685Z","message":"Bulker exited","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-11T21:50:48.685Z","message":"Error - failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"FAILED","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:48.685Z","message":"Fleet Server failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","error.message":"failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:48.685Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default (STARTING->FAILED): Error - failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:48.685Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->FAILED): Error - failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:49.891Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":801},"message":"Waiting for Elastic Agent to start Fleet Server: FAILED - Error - failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-11T21:50:50.688Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default-fleet-server (FAILED->STARTING): Starting","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STARTING","old_state":"FAILED"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-11T21:50:50.688Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default (FAILED->STARTING): Starting","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING","old_state":"FAILED"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:50.693Z","message":"fetched elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","elasticsearch_version":"8.15.1","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-11T21:50:50.693Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"8.15.1","elasticsearch_version":"8.15.1","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:50.693Z","message":"Migrations started","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:50.693Z","message":"applying migration to v7.15","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:50.694Z","message":"migration AgentMetadata failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","http.request.body.content":"{\"query\":{\"bool\":{\"must_not\":{\"exists\":{\"field\":\"agent.id\"}}}},\"script\":\"ctx._source.agent = [:]; ctx._source.agent.id = ctx._id;\"}","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:50.694Z","message":"Migrations exited","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:50.694Z","message":"Bulker exited","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-11T21:50:50.694Z","message":"Error - failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"FAILED","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:50.694Z","message":"Fleet Server failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:50.695Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->FAILED): Error - failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:50.695Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default (STARTING->FAILED): Error - failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-11T21:50:52.694Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","memory_mb":24027,"recommended_mb":16192,"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-11T21:50:52.695Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default (FAILED->STARTING): Starting","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING","old_state":"FAILED"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:52.701Z","message":"fetched elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","elasticsearch_version":"8.15.1","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-11T21:50:52.701Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"8.15.1","elasticsearch_version":"8.15.1","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:52.701Z","message":"Migrations started","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:52.701Z","message":"applying migration to v7.15","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:52.703Z","message":"migration AgentMetadata failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","http.request.body.content":"{\"query\":{\"bool\":{\"must_not\":{\"exists\":{\"field\":\"agent.id\"}}}},\"script\":\"ctx._source.agent = [:]; ctx._source.agent.id = ctx._id;\"}","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:52.703Z","message":"Migrations exited","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:52.703Z","message":"Bulker exited","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-11T21:50:52.703Z","message":"Error - failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","state":"FAILED","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:52.703Z","message":"Fleet Server failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:52.703Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->FAILED): Error - failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-11T21:50:52.703Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default (STARTING->FAILED): Error - failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-09-11T21:50:53.893Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":801},"message":"Waiting for Elastic Agent to start Fleet Server: FAILED - Error - failed to run subsystems: v7.15.0 data migration failed: failed to apply migration \"AgentMetadata\": migrate AgentMetadata UpdateByQuery failed: [403 Forbidden] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/update/byquery] is unauthorized for service account [elastic/fleet-server-remote] on restricted indices [.fleet-agents], this action is granted by the index privileges [index,write,all]\"},\"status\":403}","ecs.version":"1.6.0"}

soularius · September 14, 2024, 7:49am

Why does Kibana give a version error saying that it uses 8.10 when I have 8.15 installed on my Docker?

Topic		Replies	Views
Elastic stack with Fleet Server - certificate issue Kibana docker , fleet	4	1170	May 14, 2024
Elastic Agent - Fleet Server Configuration yml Kibana docker	0	11	August 22, 2024
Error: Kibana server is not ready yet Kibana docker	19	8802	April 9, 2023
My Elastic search and Kibana functionalities are not working after container restart , i need to recreate the entire setup again to configure logs and apm, User Experience elastic-stack-monitoring	8	174	June 21, 2024
ECK Selfhosted k8's fleet -server wont connect Kibana Elastic Cloud on Kubernetes (ECK) elastic-stack-security	1	77	July 29, 2024

Error in fleet-server, kibana, elastic search

kibana.yml

elasticsearch.yml

But the process never leaves this point.

Logs elastic-agent

Related Topics