Missing Elastic Security and endpoint integration data

Elastic Security
1

Hi Team.
Great work with the continued developments of Elastic, love the product.
So I've upgraded to v7.9 and eager to test out the new Elastic Agent/Endpoint etc.

I've enrolled a test agent insecurely, i.e. -i and the Kibana UI states the agent is enrolled. I've also setup 3 integrations in the Ingest Manager and pushed them to the agent, those being;

  • Elastic Endpoint Security

  • System

  • Windows

Unfortunately, I'm not receiving any data, and I'm scratching my head a little.

I've noticed in the elastic-endpoint.yaml file there is the following section at the bottom.

output:
     elasticsearch:
      api_key: ***************************
      hosts:
      - http://localhost:9200
    revision: 5

I've manually modified the above to be;
- https://fqdn:9200

This is what I'd expect, however restarting the elastic-agent seems to overwrite the value. I'm assuming this is the issue, though clearly I'm approaching the resolution incorrectly. Your assistance would be appreciated

Should this be defined centrally? or a bug maybe?

Thanks

2

Hello,
This seems a bug on our side, could you add the content of the action_store.yml from the elastic-agent?

3

Hello @OntheHighSeas

In order to change the value in that file you'll need to do it through the ingest manager section in Kibana. I'm not sure if you've done that or not, but there is also a bug that makes the data not propagate down as it is supposed to. We've filed an issue to fix this here: https://github.com/elastic/kibana/issues/76136

In the meantime in order to update the elastic search address endpoint is using please try the following workaround:

  • In ingest manager, under the main settings menu, you can update,add,change the Kibana and Elasticsearch URLs. Click save.
  • Afterward, under the Configurations tab of ingest manager, click on the Configuration assigned to the endpoint you want to update.
  • On the Configuration page, in the integrations tab, click the actions "..." for the Elastic Endpoint Security integration and select "Edit integration"
  • On this next page, click "Save integration" in the bottom right (you do not need to make any changes).

This should trigger an update to the configuration for the agent which will propagate down with the new global settings applies.

Please let me know if this doesn't resolve the issue for you.

-Nick Fritts

4

@OntheHighSeas

Based on the other thread:

You've updated the URLs in the global settings and edited/toucehd the configs so that it would propagate to the endpoint.

What status does your endpoint show in kibana?
You said you're using self-signed certs. Have you added them to the local machine certificate store?
Are there any errors in the log files located at: C:\Program Files\Elastic\Endpoint\state\log

-nf

5

Thanks for the continued help.

I wasn't aware of the state\log file so that's certainly a help. Looking through the file I've found I've still got a reference to localhost. As shown in the dump below. I have been through and touched the configs, I guess I missed one???

{"@timestamp":"2020-08-28T17:43:44.62428100Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://my.fqdn.com:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:43:44.87056100Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:43:49.87599100Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [http://localhost:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:43:50.93932600Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:43:55.95041900Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://my.fqdn.com:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:43:55.35233500Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:00.35868400Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [http://localhost:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:01.40025500Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:06.41148300Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://my.fqdn.com:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:06.67198900Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:11.67265600Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [http://localhost:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:13.70234200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:18.71342100Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://my.fqdn.com:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:18.97324500Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:23.97390900Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [http://localhost:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:24.99543000Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:29.99819200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://my.fqdn.com:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:29.15875100Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:34.16075800Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [http://localhost:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:35.18779500Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:40.19855300Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://my.fqdn.com:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:40.44491900Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":12728,"thread":{"id":11872}}}
{"@timestamp":"2020-08-28T17:44:45.44767900Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [http://localhost:9200/_cluster/health]","process":{"pid":12728,"thread":{"id":11872}}}

Edit, I've checked the elastic-endpoint.yaml file and there is no localhost referenced.

6

I've been through and touched all the configs again, also removed replaced the IP address for Kibana under settings for a FQDN. This seems to have removed the localhost, but still not data.

Below are more recent log lines.

{"@timestamp":"2020-09-02T16:47:57.74402600Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4688]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:47:57.35301300Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":661,"name":"File.cpp"}}},"message":"File.cpp:661 ioStatusBlock.Status=0, status=0x0","process":{"pid":5380,"thread":{"id":1388}}}
{"@timestamp":"2020-09-02T16:47:57.45802200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":661,"name":"File.cpp"}}},"message":"File.cpp:661 ioStatusBlock.Status=0, status=0x0","process":{"pid":5380,"thread":{"id":4704}}}
{"@timestamp":"2020-09-02T16:47:57.8279400Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":661,"name":"File.cpp"}}},"message":"File.cpp:661 ioStatusBlock.Status=0, status=0x0","process":{"pid":5380,"thread":{"id":1388}}}
{"@timestamp":"2020-09-02T16:47:57.23921800Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":661,"name":"File.cpp"}}},"message":"File.cpp:661 ioStatusBlock.Status=0, status=0x0","process":{"pid":5380,"thread":{"id":1388}}}
{"@timestamp":"2020-09-02T16:47:58.10959200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://my.fqdn.com:9200/_cluster/health]","process":{"pid":5380,"thread":{"id":12740}}}
{"@timestamp":"2020-09-02T16:47:58.42233100Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":5380,"thread":{"id":12740}}}
{"@timestamp":"2020-09-02T16:47:58.4025200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4673]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:47:58.5023100Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4673]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:03.50425100Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://my.fqdn.com:9200/_cluster/health]","process":{"pid":5380,"thread":{"id":12740}}}
{"@timestamp":"2020-09-02T16:48:03.66065200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":5380,"thread":{"id":12740}}}
{"@timestamp":"2020-09-02T16:48:06.14859200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":661,"name":"File.cpp"}}},"message":"File.cpp:661 ioStatusBlock.Status=0, status=0x0","process":{"pid":5380,"thread":{"id":10016}}}
{"@timestamp":"2020-09-02T16:48:06.46093600Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":661,"name":"File.cpp"}}},"message":"File.cpp:661 ioStatusBlock.Status=0, status=0x0","process":{"pid":5380,"thread":{"id":3348}}}
{"@timestamp":"2020-09-02T16:48:06.19744200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4611]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.19744200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4673]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.19744200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4627]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.19744200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4627]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.19744200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4611]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.19744200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4673]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.19744200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4688]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.19744200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4688]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.19744200Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4627]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.26708600Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4627]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.26708600Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4673]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.26708600Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4688]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.26708600Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4673]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:06.26708600Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1051,"name":"EventUpdater.cpp"}}},"message":"EventUpdater.cpp:1051 Unsupported Security Event type: [4688]","process":{"pid":5380,"thread":{"id":456}}}
{"@timestamp":"2020-09-02T16:48:08.68394400Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://my.fqdn.com:9200/_cluster/health]","process":{"pid":5380,"thread":{"id":12740}}}
{"@timestamp":"2020-09-02T16:48:08.85080300Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":5380,"thread":{"id":12740}}}
{"@timestamp":"2020-09-02T16:48:13.99174800Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://my.fqdn.com:9200/_cluster/health]","process":{"pid":5380,"thread":{"id":12740}}}
{"@timestamp":"2020-09-02T16:48:13.14798400Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":5380,"thread":{"id":12740}}}
{"@timestamp":"2020-09-02T16:48:17.23500900Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":661,"name":"File.cpp"}}},"message":"File.cpp:661 ioStatusBlock.Status=0, status=0x0","process":{"pid":5380,"thread":{"id":12024}}}
{"@timestamp":"2020-09-02T16:48:18.27818400Z","agent":{"id":"ab3850de-22a0-4e2d-862e-0aeb4ffc6f65","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://my.fqdn.com:9200/_cluster/health]","process":{"pid":5380,"thread":{"id":12740}}}
7

@OntheHighSeas

Can you check your Elasticsearch logs? There is an issue with self signed certs where they're not being provided to and taken in to account by the endpoint properly like they are for beats. Its talked about some in this post: Elastic Agent not sending Data

Have you added your self-signed certificates to the trusted roots in windows?

I'm looking to see if I can find anything that would cause the localhost to pop up like that.

8

There does seem to be a certificate issue, not sure why.

[2020-09-03T13:41:12,829][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [my_hostname] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.168.0.32:9200, remoteAddress=/192.168.0.164:56105}
[2020-09-03T13:41:13,636][WARN ][o.e.h.AbstractHttpServerTransport] [naboo] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/192.168.0.32:9200, remoteAddress=/192.168.2.13:23509}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]

My cluster SSL configuration is as follows

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.authc.api_key.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: elastic-certificates.p12
xpack.security.http.ssl.truststore.path: elastic-certificates.p12
9

@OntheHighSeas its an issue with the certificates getting passed to Endpoint and it not trusting them properly. Currently the fix is to add the CA for your certs to the endpoint's local machine trusted CA certificate store. We're planning on fixing it, but I don't have a link to the github issue handy right now. I'll see if I can track it down and add it in an edit.

10

I've imported the CA into the trusted root, still not working, I'm wondering if my cert setup is wrong some how.

11

I've been through the process of regenerating my certs and refreshed them on Elastic and Logstash. I've reenrolled the agent and and reinstalled the ca into my local pc. Still nothing...... I'm running out of ideas.

I imported the .p12 file into my pc is that correct?

12

@OntheHighSeas I've been working with @NickFritts on this issue. We are trying to understand the difference between an ssl setup using .p12 certificates vs certificate authorities. We had assumed importing the .p12 into the endpoint pc would work.

Please confirm that your output.elasticsearch remains

output:
     elasticsearch:
      api_key: ***************************
      hosts:
      - http://localhost:9200

Particularly that there is no ssl configuration.

If you have the flexibility to make tls setup changes, the setup from this blog : https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash is known to work, after adding the self-signed ca-cert (xpack.security.http.ssl.certificate_authorities: certs/ca.crt) to the endpoint's trusted system certificates.

Regardless, I will report back about .p12 certificate support.

13

So I've got it working now, but I'm not entirely sure which part of my changes specifically fixed the issue.
I followed this guide when trying again, so I guess the step which made the difference is in there.
https://www.elastic.co/guide/en/elasticsearch/reference/7.9/configuring-tls.html#node-certificates
I'd also suggest it was to do with generating the specific HTTP certs using

bin/elasticsearch-certutil http

I've wildcarded the HTTP certs generated and supplied the CA generated when using the following command as normal.

bin/elasticsearch-certutil ca

My Elasticsearch.yml now looks like this, the HTTP sections have changed.

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

xpack.security.authc.api_key.enabled: true

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
#xpack.security.http.ssl.truststore.path: certs/http.p12

Once I restarted the cluster the endpoint starting working, I guess because the CA was still correct.

I also updated to 7.9.1 while the cluster was down. I just need to get Logstash connected, just not tried yet.

14

@OntheHighSeas Glad you got it working.

I wonder if the key was commenting out this line
#xpack.security.http.ssl.truststore.path: certs/http.p12

Perhaps, that instructed elasticsearch to not expect a certificate from the http client.

15

I'm not sure. There is a lot of documentation from Elastic on the topic, and I've seen configs in those docs where the http.ssl.truststore is provided. My earlier configuration for example included it, and Logstash etc worked correctly.

Given it wasn't required in the specific instructions I was following I opted to remove it, and everything worked. Might not be the answer though, hope the insight helps though,

16

FWIW, on an Ubuntu Linux VM, I was able to connect up an endpoint to elasticsearch using pki certificates. An issue I noticed was that I needed to restart endpoint in order for it to pick up changes in the system's trusted certificates.

I ran the following two commands in a root shell to add the certificate:

$ openssl pkcs12 -in certs/http.p12 -nokeys -out /usr/local/share/ca-certificates/elasticsearch-http.crt
$ update-ca-certificates

After restarting the endpoint, it was able to connect up fine.

Also, this particular line in elasticsearch's config

 xpack.security.http.ssl.truststore.path: certs/http.p12

did not affect the ability to connect, whether commented out or not.