Elasic-agent is online but don't send data

aloalo2242 · November 23, 2020, 2:16am

Hi,

I'm trying to use Elastic Agent and Ingest Manager. I have enroll 2 windows agent with fleet. Installation is ok. In Fleet Tab, Kibana, 2 agent is Online. But it don't send any data (Logs, Index Management, Dashboard - [Metrics Windows] Services ECS,Datasets...), elastic-agent.log from client show some below logs,

|2020-11-23T13:12:17.664+0700|INFO|log/reporter.go:40|2020-11-23T13:12:17+07:00: type: 'STATE': sub_type: 'RUNNING' message: Application: filebeat--7.9.3[283c8fc5-b8b1-42df-b9a3-e6a70ed69c33]: State changed to RUNNING: Running|
|2020-11-23T13:12:18.429+0700|INFO|log/reporter.go:40|2020-11-23T13:12:18+07:00: type: 'STATE': sub_type: 'RUNNING' message: Application: filebeat--7.9.3--36643631373035623733363936343635[283c8fc5-b8b1-42df-b9a3-e6a70ed69c33]: State changed to RUNNING: Running|
|2020-11-23T13:12:18.675+0700|INFO|log/reporter.go:40|2020-11-23T13:12:18+07:00: type: 'STATE': sub_type: 'RUNNING' message: Application: metricbeat--7.9.3[283c8fc5-b8b1-42df-b9a3-e6a70ed69c33]: State changed to DEGRADED: 1 error: 1 error: Error creating runner from config: 1 error: metricset 'system/load' not found|
|2020-11-23T13:12:19.398+0700|INFO|log/reporter.go:40|2020-11-23T13:12:19+07:00: type: 'STATE': sub_type: 'RUNNING' message: Application: metricbeat--7.9.3--36643631373035623733363936343635[283c8fc5-b8b1-42df-b9a3-e6a70ed69c33]: State changed to RUNNING: Running|
|2020-11-23T13:13:18.686+0700|DEBUG|application/action_dispatcher.go:77|No action to dispatch|
|2020-11-23T13:13:18.686+0700|DEBUG|application/fleet_gateway.go:162|FleetGateway is sleeping, next update in 1s|
|2020-11-23T13:13:19.928+0700|DEBUG|application/fleet_gateway.go:142|FleetGateway calling Checkin API|
|2020-11-23T13:13:19.935+0700|DEBUG|kibana/client.go:170|Request method: POST, path: /api/ingest_manager/fleet/agents/283c8fc5-b8b1-42df-b9a3-e6a70ed69c33/checkin|

Should I config elastic-agent.yml in client? In tutorial, just config when add agent by standalone mode. There are few documents about Elastic-Agent. Hope you guy help me. Thanks

aloalo2242 · November 24, 2020, 3:59am

My elasticsearch URL use SSL, but in setting is no place to fill user, password or something to authentication. I don't know if it was the cause?

elasticsearch.yml

 xpack.security.enabled: true
    xpack.security.authc.api_key.enabled: true
    #
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: "http.p12"
    xpack.security.http.ssl.client_authentication: optional
    #
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

ruflin · November 24, 2020, 8:38am

Fleet automatically creates API Keys for each Agent for you. Could you check the logs of the filebeat or metricbeat process to see if there are more details in it on what went wrong? You should find these in data/elastic-agent-*/logs/default/metricbeat-json-*

aloalo2242 · November 24, 2020, 9:18am

Hi Ruflin,

Thanks for reply , I have check log data/logs/default/from client.

filebeat_monitor-json.log

{"log.level":"debug","@timestamp":"2020-11-24T16:07:01.941+0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":294},"message":"Ping request failed with: Get \"https://x.x.x.x:9200\": x509: certificate signed by unknown authority","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"input/input.go","file.line":139},"message":"Run input","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":205},"message":"Start next scan","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":439},"message":"Check file for harvesting: C:\\Program Files\\Elastic-Agent\\data\\logs\\elastic-agent-json.log","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":530},"message":"Update existing file for harvesting: C:\\Program Files\\Elastic-Agent\\data\\logs\\elastic-agent-json.log, offset: 35280","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":582},"message":"Harvester for file is still running: C:\\Program Files\\Elastic-Agent\\data\\logs\\elastic-agent-json.log","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:07:02.266+0700","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":226},"message":"input states cleaned up. Before: 1, After: 1, Pending: 0","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-11-24T16:07:02.497+0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":10437},"total":{"ticks":25827,"time":{"ms":31},"value":25827},"user":{"ticks":15390,"time":{"ms":31}}},"handles":{"open":366},"info":{"ephemeral_id":"0aef9b88-8fa1-48e5-a7d0-6a0435d0ba5f","uptime":{"ms":18753138}},"memstats":{"gc_next":68267152,"memory_alloc":34320256,"memory_total":314236544},"runtime":{"goroutines":64}},"filebeat":{"harvester":{"files":{"675de6c1-bead-4c83-ab59-f6da6237010e":{"size":892}},"open_files":5,"running":3}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4119,"retry":50}}},"registrar":{"states":{"current":4}}},"ecs.version":"1.5.0"}}

metricbeat-json.log

{"log.level":"debug","@timestamp":"2020-11-24T16:13:35.658+0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":294},"message":"Ping request failed with: Get \"https://x.x.x.x:9200\": x509: certificate signed by unknown authority","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-11-24T16:14:02.337+0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":7015,"time":{"ms":15}},"total":{"ticks":19983,"time":{"ms":15},"value":19983},"user":{"ticks":12968}},"handles":{"open":471},"info":{"ephemeral_id":"884271c1-677a-4d4b-897c-7eaf398bf5e4","uptime":{"ms":19173657}},"memstats":{"gc_next":57211440,"memory_alloc":28935616,"memory_total":220847824,"rss":20480},"runtime":{"goroutines":76}},"libbeat":{"config":{"module":{"running":10}},"pipeline":{"clients":10,"events":{"active":4126,"retry":50}}}},"ecs.version":"1.5.0"}}
{"log.level":"error","@timestamp":"2020-11-24T16:14:23.544+0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":154},"message":"Failed to connect to backoff(elasticsearch(https://x.x.x.x:9200)): Get \"https://x.x.x.x:9200\": x509: certificate signed by unknown authority","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-11-24T16:14:23.544+0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":145},"message":"Attempting to reconnect to backoff(elasticsearch(https://x.x.x.x:9200)) with 436 reconnect attempt(s)","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-11-24T16:14:23.544+0700","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":213},"message":"retryer: send wait signal to consumer","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:14:23.544+0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":290},"message":"ES Ping(url=https://x.x.x.x:9200)","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-11-24T16:14:23.544+0700","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":217},"message":"  done","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-11-24T16:14:23.562+0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":294},"message":"Ping request failed with: Get \"https://x.x.x.x:9200\": x509: certificate signed by unknown authority","ecs.version":"1.5.0"}

It seems the cause is certificate.

ruflin · November 24, 2020, 3:37pm

Yes, the certificate seems to be the issue. If you are on 7.9, there are quite a few discuss topics around the certificate issue which should help you. Let me know if it doesn't work.

aloalo2242 · November 25, 2020, 3:48am

I have re-install agent, but fail to enroll

PS C:\Program Files\Elastic-Agent> .\elastic-agent enroll https://KIBANA_IP ENROLLMENT_KEY==
The Elastic Agent is currently in BETA and should not be used in production
This will replace your current settings. Do you want to continue? [Y/n]:y
fail to enroll: fail to execute request to Kibana: Post "https://KIBANA_IP:443/api/ingest_manager/fleet/agents/enroll?": x509: certificate has expired or is not yet valid:

Then I re-generate certificate and the results are still the same.

Now it's certificate has expired or is not yet valid , not certificate signed by unknown authority though I have not configured anything before.

*I generate self-signed certificate with this tutorial https://techexpert.tips/elasticsearch/elasticsearch-enable-tls-https/

Mikel_U · November 25, 2020, 8:01am

You can enroll with --insecure argument, but then there is no Datasets I have the same problem.

aloalo2242 · November 25, 2020, 8:33am

First, I didn't config SSL to Kibana and the url was KIBANA_IP:5601, so I tried enroll with --insecure as PowerShell suggests. It didn't send data as you say. Then I config SSL to Kibana, enroll in PowerShell didn't warn , suggest use --insecure anymore.

pierhugues · November 25, 2020, 2:06pm

@aloalo2242 Looking at the previous discussion you are using a self-signed certificate, I presume the host doesn't have the Certificate Authority installed locally to valid the remote certificate. Have you tried to give the Certificate authority as an argument to the enroll subcommand using --certificate-authorities /my/path/to/my/ca

aloalo2242 · November 26, 2020, 2:13am

Hi Pier,

Thank for your suggestion, I think it's very helpful. But in my case, the CA has expired though I generated it a days ago and set "How long should your certificate be valid" is 5 years , seem weird

x509: certificate has expired or is not yet valid: current time 2020-11-26T09:06:32+07:00 is after 2020-11-24T09:10:38Z
Error: enroll command failed with exit code: 1

pierhugues · November 30, 2020, 6:32pm

Well, from what I see this is indeed the problem, have you tried to regenerate the cert?
if you are under linux you can use the following command on the certificate would give you more information.

openssl x509 -in certificate.crt -text -noout

aloalo2242 · December 1, 2020, 2:59am

Hi Pier,

I summarize the process as follows,

generate the CA /usr/share/elasticsearch/bin/elasticsearch-certutil ca and I got elastic-stack-ca.p12
generate Elastic Certificate by the CA /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 and got elastic-certificates.p12
generate Client Certificate by the CA /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca config/certs/elastic-stack-ca.p12 -name "CN=something,OU=Consulting Team,DC=mydomain,DC=com" set output file is client.p12
Generate:
Private Key
openssl pkcs12 -in client.p12 -nocerts -nodes > client.key
Public Certificate
openssl pkcs12 -in client.p12 -clcerts -nokeys > client.cer
CA Certificate
openssl pkcs12 -in client.p12 -cacerts -nokeys -chain > client-ca.cer
Copy them to the equivalent directory, and config yml file

elasticsearch.yml

xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true
#
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: elastic-certificates.p12
xpack.security.http.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional
#
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

kibana.yml

elasticsearch.hosts: ["https://localhost:9200"]

xpack.fleet.enabled: true
xpack.fleet.agents.enabled: true
xpack.fleet.agents.tlsCheckDisabled: true
xpack.encryptedSavedObjects.encryptionKey: "something32characters"
xpack.security.enabled: true
elasticsearch.username: "elastic"
elasticsearch.password: "password"
elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.certificateAuthorities: /etc/kibana/client-ca.cer
elasticsearch.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

Convert to .pem format and copy to client openssl pkcs12 -in "client.p12" -out "client.pem" -clcerts -nokeys
Enroll and get error
.\elastic-agent enroll https://KIBANA_IP ENROLLMENT_KEY== --certificate-authorities C:\Test\client.pem

Check client.pem

[root@localhost elasticsearch]# openssl x509 -in client.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            78:fe:a8:e3:c1:82:78:77:52:b5:7d:ac:ba:b9:8c:88:b9:26:04:6f
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Elastic Certificate Tool Autogenerated CA
        Validity
            Not Before: Nov 30 02:27:33 2020 GMT
            Not After : Nov 30 02:27:33 2023 GMT
        Subject: DC=com, DC=mydomain, OU=Consulting, CN=something
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:94:50:ad:d8:db:09:f0:84:1d:f7:93:14:e6:3b:
                    c1:5d:e2:6f:4f:59:4e:ab:b1:e2:81:d6:00:17:6f:
                    b0:a9:89:98:94:47:b0:d2:c0:eb:9e:58:0a:57:65:
                    9e:4d:4e:5e:e5:28:04:b7:3f:9c:44:24:9c:8e:b8:
                    06:8e:a3:d8:33:3c:79:ec:1f:bf:cf:05:48:68:78:
                    b6:cd:7c:26:88:74:fb:a4:e8:6d:6e:56:bd:af:c4:
                    6e:de:f3:70:52:34:e7:69:2a:67:fa:7e:7d:02:66:
                    13:07:1f:4c:05:47:82:e1:15:c7:48:d6:64:65:29:
                    c5:02:40:14:c9:5b:f0:c1:cb:4b:22:cf:42:78:e2:
                    5f:b0:4f:10:c4:7c:bf:f9:f7:b1:f7:f5:84:ae:55:
                    76:5b:43:66:19:d6:09:4d:12:60:a5:1a:c7:5a:b6:
                    22:3f:ed:58:c2:e8:76:4c:f7:14:39:a7:b5:b1:77:
                    6f:98:c6:29:7d:bb:95:98:92:3c:d6:6c:d9:ea:6d:
                    a8:f6:c0:4d:66:bc:3e:9b:c0:bd:ba:1d:2c:42:54:
                    e4:7a:0e:49:25:8f:6f:b4:b2:71:f9:17:6d:50:b2:
                    9e:dd:51:99:90:fb:b8:c0:1c:f4:7b:02:03:dc:2f:
                    ed:f3:c6:1f:c1:53:c7:18:a2:39:56:2b:12:06:cb:
                    6f:0d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                21:EC:18:A1:60:07:E7:CE:03:02:4C:BF:CE:9B:19:80:49:F3:41:08
            X509v3 Authority Key Identifier:
                keyid:0C:4C:02:81:35:2D:46:60:C0:03:74:8D:21:7E:EA:B9:7D:0A:BA:82

            X509v3 Basic Constraints:
                CA:FALSE
    Signature Algorithm: sha256WithRSAEncryption
         03:d4:f3:6d:89:8e:a5:de:35:b6:49:12:c4:a3:fd:7f:81:b4:
         ff:cf:74:38:48:90:c3:9a:5b:92:44:03:ca:04:2a:b1:70:1d:
         bf:ff:c0:ae:e0:5f:5d:cf:19:6c:e6:84:46:3d:d7:72:f5:95:
         ab:bf:93:7c:05:45:b8:ad:59:e3:18:97:9e:89:7a:02:ce:b9:
         17:66:de:25:88:62:6e:6a:ae:5c:1f:4a:c0:f8:25:61:41:10:
         00:cd:c7:f6:e5:a6:74:f4:24:fd:c3:79:81:77:2e:43:6c:ca:
         c9:89:b6:21:35:fd:72:b0:64:76:ce:9d:ab:d9:3c:d8:a4:3c:
         37:3f:e5:b9:2b:9e:cd:70:26:7e:1d:fb:4f:be:89:dd:e4:fb:
         ac:f4:71:58:14:fc:f7:cd:12:a5:4a:fd:a1:62:2c:ea:4d:51:
         70:23:63:82:09:3c:0d:84:3f:f2:8d:ad:17:f9:c8:2e:dc:97:
         57:bd:00:c8:de:82:3b:fe:ee:79:69:90:92:ee:7a:8b:77:71:
         96:9c:8a:84:e4:3c:60:30:cf:87:e5:f0:8b:63:05:ea:5a:04:
         c5:3f:46:1c:aa:55:1a:4f:86:98:ee:06:74:6e:ce:9f:4b:1e:
         f3:61:9f:5a:73:12:a8:6c:19:c1:be:af:c2:df:d7:18:20:ed:
         e1:67:77:ab

Every certificate that I generated has no password,... just Enter Enter

Nirmal_Unagar · July 13, 2022, 9:40am

whne i have used --insecure, am not able to fetch data from client. is there any soln?