Elasic-agent is online but don't send data

Hi,

I'm trying to use Elastic Agent and Ingest Manager. I have enroll 2 windows agent with fleet. Installation is ok. In Fleet Tab, Kibana, 2 agent is Online. But it don't send any data (Logs, Index Management, Dashboard - [Metrics Windows] Services ECS,Datasets...), elastic-agent.log from client show some below logs,

|2020-11-23T13:12:17.664+0700|INFO|log/reporter.go:40|2020-11-23T13:12:17+07:00: type: 'STATE': sub_type: 'RUNNING' message: Application: filebeat--7.9.3[283c8fc5-b8b1-42df-b9a3-e6a70ed69c33]: State changed to RUNNING: Running|
|2020-11-23T13:12:18.429+0700|INFO|log/reporter.go:40|2020-11-23T13:12:18+07:00: type: 'STATE': sub_type: 'RUNNING' message: Application: filebeat--7.9.3--36643631373035623733363936343635[283c8fc5-b8b1-42df-b9a3-e6a70ed69c33]: State changed to RUNNING: Running|
|2020-11-23T13:12:18.675+0700|INFO|log/reporter.go:40|2020-11-23T13:12:18+07:00: type: 'STATE': sub_type: 'RUNNING' message: Application: metricbeat--7.9.3[283c8fc5-b8b1-42df-b9a3-e6a70ed69c33]: State changed to DEGRADED: 1 error: 1 error: Error creating runner from config: 1 error: metricset 'system/load' not found|
|2020-11-23T13:12:19.398+0700|INFO|log/reporter.go:40|2020-11-23T13:12:19+07:00: type: 'STATE': sub_type: 'RUNNING' message: Application: metricbeat--7.9.3--36643631373035623733363936343635[283c8fc5-b8b1-42df-b9a3-e6a70ed69c33]: State changed to RUNNING: Running|
|2020-11-23T13:13:18.686+0700|DEBUG|application/action_dispatcher.go:77|No action to dispatch|
|2020-11-23T13:13:18.686+0700|DEBUG|application/fleet_gateway.go:162|FleetGateway is sleeping, next update in 1s|
|2020-11-23T13:13:19.928+0700|DEBUG|application/fleet_gateway.go:142|FleetGateway calling Checkin API|
|2020-11-23T13:13:19.935+0700|DEBUG|kibana/client.go:170|Request method: POST, path: /api/ingest_manager/fleet/agents/283c8fc5-b8b1-42df-b9a3-e6a70ed69c33/checkin|

Should I config elastic-agent.yml in client? In tutorial, just config when add agent by standalone mode. There are few documents about Elastic-Agent. Hope you guy help me. Thanks

My elasticsearch URL use SSL, but in setting is no place to fill user, password or something to authentication. I don't know if it was the cause?

elasticsearch.yml

 xpack.security.enabled: true
    xpack.security.authc.api_key.enabled: true
    #
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: "http.p12"
    xpack.security.http.ssl.client_authentication: optional
    #
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

Fleet automatically creates API Keys for each Agent for you. Could you check the logs of the filebeat or metricbeat process to see if there are more details in it on what went wrong? You should find these in data/elastic-agent-*/logs/default/metricbeat-json-*

Hi Ruflin,

Thanks for reply , I have check log data/logs/default/from client.

filebeat_monitor-json.log

{"log.level":"debug","@timestamp":"2020-11-24T16:07:01.941+0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":294},"message":"Ping request failed with: Get \"https://x.x.x.x:9200\": x509: certificate signed by unknown authority","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"input/input.go","file.line":139},"message":"Run input","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":205},"message":"Start next scan","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":439},"message":"Check file for harvesting: C:\\Program Files\\Elastic-Agent\\data\\logs\\elastic-agent-json.log","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":530},"message":"Update existing file for harvesting: C:\\Program Files\\Elastic-Agent\\data\\logs\\elastic-agent-json.log, offset: 35280","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":582},"message":"Harvester for file is still running: C:\\Program Files\\Elastic-Agent\\data\\logs\\elastic-agent-json.log","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":226},"message":"input states cleaned up. Before: 1, After: 1, Pending: 0","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-11-24T16:07:02.497+0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":10437},"total":{"ticks":25827,"time":{"ms":31},"value":25827},"user":{"ticks":15390,"time":{"ms":31}}},"handles":{"open":366},"info":{"ephemeral_id":"0aef9b88-8fa1-48e5-a7d0-6a0435d0ba5f","uptime":{"ms":18753138}},"memstats":{"gc_next":68267152,"memory_alloc":34320256,"memory_total":314236544},"runtime":{"goroutines":64}},"filebeat":{"harvester":{"files":{"675de6c1-bead-4c83-ab59-f6da6237010e":{"size":892}},"open_files":5,"running":3}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4119,"retry":50}}},"registrar":{"states":{"current":4}}},"ecs.version":"1.5.0"}}

metricbeat-json.log

{"log.level":"debug","@timestamp":"2020-11-24T16:13:35.658+0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":294},"message":"Ping request failed with: Get \"https://x.x.x.x:9200\": x509: certificate signed by unknown authority","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-11-24T16:14:02.337+0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":7015,"time":{"ms":15}},"total":{"ticks":19983,"time":{"ms":15},"value":19983},"user":{"ticks":12968}},"handles":{"open":471},"info":{"ephemeral_id":"884271c1-677a-4d4b-897c-7eaf398bf5e4","uptime":{"ms":19173657}},"memstats":{"gc_next":57211440,"memory_alloc":28935616,"memory_total":220847824,"rss":20480},"runtime":{"goroutines":76}},"libbeat":{"config":{"module":{"running":10}},"pipeline":{"clients":10,"events":{"active":4126,"retry":50}}}},"ecs.version":"1.5.0"}}
{"log.level":"error","@timestamp":"2020-11-24T16:14:23.544+0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":154},"message":"Failed to connect to backoff(elasticsearch(https://x.x.x.x:9200)): Get \"https://x.x.x.x:9200\": x509: certificate signed by unknown authority","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-11-24T16:14:23.544+0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":145},"message":"Attempting to reconnect to backoff(elasticsearch(https://x.x.x.x:9200)) with 436 reconnect attempt(s)","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-11-24T16:14:23.544+0700","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":213},"message":"retryer: send wait signal to consumer","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:14:23.544+0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":290},"message":"ES Ping(url=https://x.x.x.x:9200)","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-11-24T16:14:23.544+0700","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":217},"message":"  done","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:14:23.562+0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":294},"message":"Ping request failed with: Get \"https://x.x.x.x:9200\": x509: certificate signed by unknown authority","ecs.version":"1.5.0"}

It seems the cause is certificate.

Yes, the certificate seems to be the issue. If you are on 7.9, there are quite a few discuss topics around the certificate issue which should help you. Let me know if it doesn't work.

I have re-install agent, but fail to enroll

PS C:\Program Files\Elastic-Agent> .\elastic-agent enroll https://KIBANA_IP ENROLLMENT_KEY==
The Elastic Agent is currently in BETA and should not be used in production
This will replace your current settings. Do you want to continue? [Y/n]:y
fail to enroll: fail to execute request to Kibana: Post "https://KIBANA_IP:443/api/ingest_manager/fleet/agents/enroll?": x509: certificate has expired or is not yet valid:

Then I re-generate certificate and the results are still the same.

Now it's certificate has expired or is not yet valid , not certificate signed by unknown authority though I have not configured anything before.

*I generate self-signed certificate with this tutorial https://techexpert.tips/elasticsearch/elasticsearch-enable-tls-https/

You can enroll with --insecure argument, but then there is no Datasets :pensive: I have the same problem.

First, I didn't config SSL to Kibana and the url was KIBANA_IP:5601, so I tried enroll with --insecure as PowerShell suggests. It didn't send data as you say. Then I config SSL to Kibana, enroll in PowerShell didn't warn , suggest use --insecure anymore.

@aloalo2242 Looking at the previous discussion you are using a self-signed certificate, I presume the host doesn't have the Certificate Authority installed locally to valid the remote certificate. Have you tried to give the Certificate authority as an argument to the enroll subcommand using --certificate-authorities /my/path/to/my/ca

Hi Pier,

Thank for your suggestion, I think it's very helpful. But in my case, the CA has expired though I generated it a days ago and set "How long should your certificate be valid" is 5 years , seem weird

x509: certificate has expired or is not yet valid: current time 2020-11-26T09:06:32+07:00 is after 2020-11-24T09:10:38Z
Error: enroll command failed with exit code: 1

Well, from what I see this is indeed the problem, have you tried to regenerate the cert?
if you are under linux you can use the following command on the certificate would give you more information.

openssl x509 -in certificate.crt -text -noout

Hi Pier,

I summarize the process as follows,

  1. generate the CA /usr/share/elasticsearch/bin/elasticsearch-certutil ca and I got elastic-stack-ca.p12

  2. generate Elastic Certificate by the CA /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 and got elastic-certificates.p12

  3. generate Client Certificate by the CA /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca config/certs/elastic-stack-ca.p12 -name "CN=something,OU=Consulting Team,DC=mydomain,DC=com" set output file is client.p12

  4. Generate:
    Private Key
    openssl pkcs12 -in client.p12 -nocerts -nodes > client.key
    Public Certificate
    openssl pkcs12 -in client.p12 -clcerts -nokeys > client.cer
    CA Certificate
    openssl pkcs12 -in client.p12 -cacerts -nokeys -chain > client-ca.cer

  5. Copy them to the equivalent directory, and config yml file

elasticsearch.yml

xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true
#
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: elastic-certificates.p12
xpack.security.http.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional
#
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

kibana.yml

elasticsearch.hosts: ["https://localhost:9200"]

xpack.fleet.enabled: true
xpack.fleet.agents.enabled: true
xpack.fleet.agents.tlsCheckDisabled: true
xpack.encryptedSavedObjects.encryptionKey: "something32characters"
xpack.security.enabled: true
elasticsearch.username: "elastic"
elasticsearch.password: "password"
elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.certificateAuthorities: /etc/kibana/client-ca.cer
elasticsearch.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
  1. Convert to .pem format and copy to client openssl pkcs12 -in "client.p12" -out "client.pem" -clcerts -nokeys

  2. Enroll and get error
    .\elastic-agent enroll https://KIBANA_IP ENROLLMENT_KEY== --certificate-authorities C:\Test\client.pem

Check client.pem

[root@localhost elasticsearch]# openssl x509 -in client.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            78:fe:a8:e3:c1:82:78:77:52:b5:7d:ac:ba:b9:8c:88:b9:26:04:6f
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Elastic Certificate Tool Autogenerated CA
        Validity
            Not Before: Nov 30 02:27:33 2020 GMT
            Not After : Nov 30 02:27:33 2023 GMT
        Subject: DC=com, DC=mydomain, OU=Consulting, CN=something
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:94:50:ad:d8:db:09:f0:84:1d:f7:93:14:e6:3b:
                    c1:5d:e2:6f:4f:59:4e:ab:b1:e2:81:d6:00:17:6f:
                    b0:a9:89:98:94:47:b0:d2:c0:eb:9e:58:0a:57:65:
                    9e:4d:4e:5e:e5:28:04:b7:3f:9c:44:24:9c:8e:b8:
                    06:8e:a3:d8:33:3c:79:ec:1f:bf:cf:05:48:68:78:
                    b6:cd:7c:26:88:74:fb:a4:e8:6d:6e:56:bd:af:c4:
                    6e:de:f3:70:52:34:e7:69:2a:67:fa:7e:7d:02:66:
                    13:07:1f:4c:05:47:82:e1:15:c7:48:d6:64:65:29:
                    c5:02:40:14:c9:5b:f0:c1:cb:4b:22:cf:42:78:e2:
                    5f:b0:4f:10:c4:7c:bf:f9:f7:b1:f7:f5:84:ae:55:
                    76:5b:43:66:19:d6:09:4d:12:60:a5:1a:c7:5a:b6:
                    22:3f:ed:58:c2:e8:76:4c:f7:14:39:a7:b5:b1:77:
                    6f:98:c6:29:7d:bb:95:98:92:3c:d6:6c:d9:ea:6d:
                    a8:f6:c0:4d:66:bc:3e:9b:c0:bd:ba:1d:2c:42:54:
                    e4:7a:0e:49:25:8f:6f:b4:b2:71:f9:17:6d:50:b2:
                    9e:dd:51:99:90:fb:b8:c0:1c:f4:7b:02:03:dc:2f:
                    ed:f3:c6:1f:c1:53:c7:18:a2:39:56:2b:12:06:cb:
                    6f:0d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                21:EC:18:A1:60:07:E7:CE:03:02:4C:BF:CE:9B:19:80:49:F3:41:08
            X509v3 Authority Key Identifier:
                keyid:0C:4C:02:81:35:2D:46:60:C0:03:74:8D:21:7E:EA:B9:7D:0A:BA:82

            X509v3 Basic Constraints:
                CA:FALSE
    Signature Algorithm: sha256WithRSAEncryption
         03:d4:f3:6d:89:8e:a5:de:35:b6:49:12:c4:a3:fd:7f:81:b4:
         ff:cf:74:38:48:90:c3:9a:5b:92:44:03:ca:04:2a:b1:70:1d:
         bf:ff:c0:ae:e0:5f:5d:cf:19:6c:e6:84:46:3d:d7:72:f5:95:
         ab:bf:93:7c:05:45:b8:ad:59:e3:18:97:9e:89:7a:02:ce:b9:
         17:66:de:25:88:62:6e:6a:ae:5c:1f:4a:c0:f8:25:61:41:10:
         00:cd:c7:f6:e5:a6:74:f4:24:fd:c3:79:81:77:2e:43:6c:ca:
         c9:89:b6:21:35:fd:72:b0:64:76:ce:9d:ab:d9:3c:d8:a4:3c:
         37:3f:e5:b9:2b:9e:cd:70:26:7e:1d:fb:4f:be:89:dd:e4:fb:
         ac:f4:71:58:14:fc:f7:cd:12:a5:4a:fd:a1:62:2c:ea:4d:51:
         70:23:63:82:09:3c:0d:84:3f:f2:8d:ad:17:f9:c8:2e:dc:97:
         57:bd:00:c8:de:82:3b:fe:ee:79:69:90:92:ee:7a:8b:77:71:
         96:9c:8a:84:e4:3c:60:30:cf:87:e5:f0:8b:63:05:ea:5a:04:
         c5:3f:46:1c:aa:55:1a:4f:86:98:ee:06:74:6e:ce:9f:4b:1e:
         f3:61:9f:5a:73:12:a8:6c:19:c1:be:af:c2:df:d7:18:20:ed:
         e1:67:77:ab

Every certificate that I generated has no password,... just Enter Enter :sweat_smile: