Cannot Enroll Elastic Agent to Fleet hosted on Elastic Cloud Enterprise

hallaoui · August 31, 2020, 1:54pm

Hi All,

I am trying to test Elastic Agent on my PC Windows laptop but when I run command to enroll it to Fleet I got below message error.

.\elastic-agent enroll https://ef13ea6b551540938cccfb4c482b50a0.10.30.30.204.ip.es.io:9243 MzdKVFJIUUJxT0k1UURMbzh1NGQ6NzJJRjBPYzVRaU9mMFNwcjVxVWNZdw==

fail to enroll: fail to execute request to Kibana: Post "https://ef13ea6b551540938cccfb4c482b50a0.10.30.30.204.ip.es.io:9243/api/ingest_manager/fleet/agents/enroll?": x509: certificate signed by unknown authority

For info my Elasticsearch and Kibana are deployed in ECE.

Thanks for your support.

Hamid

ruflin · September 1, 2020, 8:23am

Hi, it looks like you have the same issue as others with the self signed certificates. Have a look at this thread here: [Ingest management] Use insecure elasticsearch output managed in fleet mode for elastic agent

hallaoui · September 1, 2020, 10:42am

Hi Ruflin,

Thanks for your reply but in the thread you mentioned I cannot find any solution to my issue. I even tried the Standalone mode but not data shown in Datasets tab of Ingest Manager.
Any detailed procedure please to follow to make it working ?

Thanks for your support.
Hamid

ruflin · September 2, 2020, 9:24am

Could you share by chance the config you used in standalone mode? Under data you can find error logs for metricbeat / filebeat, could you share these as they might contain more information on what didn't work.

hallaoui · September 2, 2020, 10:07am

Hi Ruflin,

Thanks again for your support.

The configuration I used is the one given by Ingest Manager and you can find it below.

For the logs please find them attached to this message.

elastic-agent.yml:

id: 5fd56a50-eb7e-11ea-87cd-11a6a05e64d9
revision: 1
outputs:
default:
type: elasticsearch
hosts:

'' username: elastic password: 05b7jz5qldMlRX5tUAQnV8ff agent: monitoring: enabled: true use_output: default logs: true metrics: true inputs: - id: 6bce72c0-eb7e-11ea-87cd-11a6a05e64d9 name: system-1 type: logfile use_output: default meta: package: name: system version: 0.5.3 data_stream: namespace: default streams: - id: logfile-system.auth data_stream: dataset: system.auth type: logs paths: - /var/log/auth.log* - /var/log/secure* exclude_files: - .gz$ multiline: pattern: ^\s match: after processors: - add_locale: null - add_fields: target: '' fields: ecs.version: 1.5.0 - id: logfile-system.syslog data_stream: dataset: system.syslog type: logs paths: - /var/log/messages* - /var/log/syslog* exclude_files: - .gz$ multiline: pattern: ^\s match: after processors: - add_locale: null - add_fields: target: '' fields: ecs.version: 1.5.0 - id: 6bce72c0-eb7e-11ea-87cd-11a6a05e64d9 name: system-1 type: system/metrics use_output: default meta: package: name: system version: 0.5.3 data_stream: namespace: default streams: - id: system/metrics-system.cpu data_stream: dataset: system.cpu type: metrics metricsets: - cpu cpu.metrics: - percentages - normalized_percentages period: 10s - id: system/metrics-system.diskio data_stream: dataset: system.diskio type: metrics metricsets: - diskio diskio.include_devices: null period: 10s - id: system/metrics-system.load data_stream: dataset: system.load type: metrics metricsets: - load period: 10s - id: system/metrics-system.memory data_stream: dataset: system.memory type: metrics metricsets: - memory period: 10s - id: system/metrics-system.network data_stream: dataset: system.network type: metrics metricsets: - network period: 10s network.interfaces: null - id: system/metrics-system.process data_stream: dataset: system.process type: metrics metricsets: - process period: 10s process.include_top_n.by_cpu: 5 process.include_top_n.by_memory: 5 process.cmdline.cache.enabled: true process.cgroups.enabled: true processes: - .* - id: system/metrics-system.process_summary data_stream: dataset: system.process_summary type: metrics metricsets: - process_summary period: 10s - id: system/metrics-system.socket_summary data_stream: dataset: system.socket_summary type: metrics metricsets: - socket_summary period: 10s - id: system/metrics-system.uptime data_stream: dataset: system.uptime type: metrics metricsets: - uptime period: 10s

Thanks & Regards,

Hamid

(Attachment logs.tar.gz is missing)

hallaoui · September 2, 2020, 10:11am

Please find attached metricbeat / filebeat logs.

Thanks

(Attachment filebeat-json.log-2020-09-02-12-1 is missing)

(Attachment metricbeat-json.log-2020-09-02-12-2 is missing)

hallaoui · September 2, 2020, 10:13am

Apparently I cannot upload the logs files to the thread. Is there another way to share the logs?

Thanks.

ruflin · September 2, 2020, 12:10pm

Could you try to use ticks (`) around the pasted yaml so it is formatted correctly? I couldn't see a host specified in there. Under what hostname is Elasticsearch running.

For the logs, you could try a gist.

hallaoui · September 2, 2020, 1:01pm

id: 5fd56a50-eb7e-11ea-87cd-11a6a05e64d9
revision: 1
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://45cd934da9d443b187c79007c5bcb586.10.30.30.204.ip.es.io:9243'
    username: elastic
    password: 05b7jz5qldMlRX5tUAQnV8ff
agent:
  monitoring:
    enabled: true
    use_output: default
    logs: true
    metrics: true
inputs:
  - id: 6bce72c0-eb7e-11ea-87cd-11a6a05e64d9
    name: system-1
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 0.5.3
    data_stream:
      namespace: default
    streams:
      - id: logfile-system.auth
        data_stream:
          dataset: system.auth
          type: logs
        paths:
          - /var/log/auth.log*
          - /var/log/secure*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.5.0
      - id: logfile-system.syslog
        data_stream:
          dataset: system.syslog
          type: logs
        paths:
          - /var/log/messages*
          - /var/log/syslog*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.5.0
  - id: 6bce72c0-eb7e-11ea-87cd-11a6a05e64d9
    name: system-1
    type: system/metrics
    use_output: default
    meta:
      package:
        name: system
        version: 0.5.3
    data_stream:
      namespace: default
    streams:
      - id: system/metrics-system.cpu
        data_stream:
          dataset: system.cpu
          type: metrics
        metricsets:
          - cpu
        cpu.metrics:
          - percentages
          - normalized_percentages
        period: 10s
      - id: system/metrics-system.diskio
        data_stream:
          dataset: system.diskio
          type: metrics
        metricsets:
          - diskio
        diskio.include_devices: null
        period: 10s
      - id: system/metrics-system.load
        data_stream:
          dataset: system.load
          type: metrics
        metricsets:
          - load
        period: 10s
      - id: system/metrics-system.memory
        data_stream:
          dataset: system.memory
          type: metrics
        metricsets:
          - memory
        period: 10s
      - id: system/metrics-system.network
        data_stream:
          dataset: system.network
          type: metrics
        metricsets:
          - network
        period: 10s
        network.interfaces: null
      - id: system/metrics-system.process
        data_stream:
          dataset: system.process
          type: metrics
        metricsets:
          - process
        period: 10s
        process.include_top_n.by_cpu: 5
        process.include_top_n.by_memory: 5
        process.cmdline.cache.enabled: true
        process.cgroups.enabled: true
        processes:
          - .*
      - id: system/metrics-system.process_summary
        data_stream:
          dataset: system.process_summary
          type: metrics
        metricsets:
          - process_summary
        period: 10s
      - id: system/metrics-system.socket_summary
        data_stream:
          dataset: system.socket_summary
          type: metrics
        metricsets:
          - socket_summary
        period: 10s
      - id: system/metrics-system.uptime
        data_stream:
          dataset: system.uptime
          type: metrics
        metricsets:
          - uptime
        period: 10s

hallaoui · September 2, 2020, 1:13pm

For the logs please find them in below gist:

gist.github.com

https://gist.github.com/hamidallaoui/84bc7ea870fa46da7feaff8567eac335

filebeat-json.log-2020-09-02-12-1

{"log.level":"debug","@timestamp":"2020-09-02T12:00:05.452+0300","log.logger":"input","log.origin":{"file.name":"input/input.go","file.line":139},"message":"Run input","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:00:05.463+0300","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":205},"message":"Start next scan","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:00:05.464+0300","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":226},"message":"input states cleaned up. Before: 0, After: 0, Pending: 0","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:00:05.466+0300","log.logger":"input","log.origin":{"file.name":"input/input.go","file.line":139},"message":"Run input","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:00:05.466+0300","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":205},"message":"Start next scan","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:00:05.467+0300","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":226},"message":"input states cleaned up. Before: 0, After: 0, Pending: 0","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:00:15.467+0300","log.logger":"input","log.origin":{"file.name":"input/input.go","file.line":139},"message":"Run input","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:00:15.467+0300","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":205},"message":"Start next scan","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:00:15.467+0300","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":226},"message":"input states cleaned up. Before: 0, After: 0, Pending: 0","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:00:15.469+0300","log.logger":"input","log.origin":{"file.name":"input/input.go","file.line":139},"message":"Run input","ecs.version":"1.5.0"}

This file has been truncated. show original

metricbeat-json.log-2020-09-02-12-2

{"log.level":"info","@timestamp":"2020-09-02T12:56:57.844+0300","log.origin":{"file.name":"instance/beat.go","file.line":640},"message":"Home path: [C:\\Program Files\\Elastic-Agent\\data\\install\\metricbeat-7.9.0-windows-x86_64] Config path: [C:\\Program Files\\Elastic-Agent\\data\\install\\metricbeat-7.9.0-windows-x86_64] Data path: [C:\\Program Files\\Elastic-Agent\\data\\run\\default\\metricbeat--7.9.0] Logs path: [C:\\Program Files\\Elastic-Agent\\data\\install\\metricbeat-7.9.0-windows-x86_64\\logs]","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:56:57.880+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":692},"message":"Beat metadata path: C:\\Program Files\\Elastic-Agent\\data\\run\\default\\metricbeat--7.9.0\\meta.json","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-02T12:56:57.892+0300","log.origin":{"file.name":"instance/beat.go","file.line":648},"message":"Beat ID: 864913ac-c66b-45c8-ba7e-89feb383d023","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:56:58.373+0300","log.logger":"docker","log.origin":{"file.name":"docker/client.go","file.line":48},"message":"Docker client will negotiate the API version on the first request.","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:56:58.374+0300","log.logger":"add_docker_metadata","log.origin":{"file.name":"add_docker_metadata/add_docker_metadata.go","file.line":87},"message":"add_docker_metadata: docker environment not detected: protocol not available","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:56:58.374+0300","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":126},"message":"add_cloud_metadata: starting to fetch metadata, timeout=3s","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:56:58.377+0300","log.logger":"kubernetes","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":138},"message":"Could not create kubernetes client using in_cluster config: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:56:58.404+0300","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":162},"message":"add_cloud_metadata: received disposition for openstack after 4.0076ms. result=[provider:openstack, error=failed requesting openstack metadata: Get \"http://169.254.169.254/2009-04-04/meta-data/instance-id\": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network., metadata={}]","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:56:58.405+0300","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":162},"message":"add_cloud_metadata: received disposition for digitalocean after 30.0068ms. result=[provider:digitalocean, error=failed requesting digitalocean metadata: Get \"http://169.254.169.254/metadata/v1.json\": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network., metadata={}]","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-02T12:56:58.405+0300","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":162},"message":"add_cloud_metadata: received disposition for aws after 31.0038ms. result=[provider:aws, error=failed requesting aws metadata: Get \"http://169.254.169.254/2014-02-25/dynamic/instance-identity/document\": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network., metadata={}]","ecs.version":"1.5.0"}

This file has been truncated. show original

Thanks for your support.

ruflin · September 3, 2020, 7:08am

Thanks for all the details. I found the following entry in your log files:

{"log.level":"debug","@timestamp":"2020-09-02T12:57:27.817+0300","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":294},"message":"Ping request failed with: Get \"https://45cd934da9d443b187c79007c5bcb586.10.30.30.204.ip.es.io:9243\": x509: certificate signed by unknown authority","ecs.version":"1.5.0"}

Where are you running your ES cluster? It seems like a certificate issue.

hallaoui · September 3, 2020, 8:04am

Hi Ruflin,

Thanks for your support.

My ES is running in ECE deployed on remote linux machines and the Agent is running on my laptop.

Thanks.

Hamid

ruflin · September 4, 2020, 8:06am

I think we have a certificate issue here. Unfortunately its an issue quite a few users are hitting at the moment and we are working on fixing it. For an intermediate solution, have a look at these threads here: https://discuss.elastic.co/search?q=certificate%20tag%3Astack-ingest-management

hallaoui · September 6, 2020, 7:57am

Hi Ruflin,

Thank you for your support, by waiting your fix I will have a look at the threads you shared and see if it can help to fix my issue.

Thanks,
Hamid

system · September 20, 2020, 7:57am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.