Elastic Agent healthy, but no data streams

Elastic Stack Beats
,
#1

The Elastic Agent (7.11.1) is installed on a single Windows 10 test system and showing as 'Healthy' in Fleet, but I have no data appearing in streams.

I have xpack security working with self generated certs - I have loaded the Elastic ca cert in to the trusted root authorities store of the test computer.

The metricbeat_monitor-json.log on the test machine contains multiple entries regarding unknown certificates:

"Failed to connect to backoff(elasticsearch(https://10.1.140.50:9200)): Get '\"https://10.1.140.50:9200\": x509: certificate signed by unknown authority","ecs.version":"1.6.0"}
'{"file.name":"pipeline/output.go","file.line":145},"message":"Attempting to reconnect to 'backoff(elasticsearch(https://10.1.140.50:9200)) with 3 reconnect attempt(s)","ecs.version":"1.6.0"}

What have I missed?

Elastic-Agent not providing data-stream
#2

I have xpack security working with self generated certs - I have loaded the Elastic ca cert in to the trusted root authorities store of the test computer.

It should be fine, did you try to restart the Agent? I assume that certs have been installed on that host, right?

#3

Hi Marcin,
Yes - I've restarted the Agent, and even rebooted the computer - but the issue remains.
Certs? - the root CA cert has been installed to 'local computer trusted root authorities', is there another certificate that I should install?

#4

What happens if you open the https://10.1.140.50:9200 with the browser? Is the certificate considered as trusted?

#5

Oh! That's a great question.
So, with port 5601 for Kibana, no probelm, but on 9200 it fails with a security exception.

Note that the test machine has firefox synchornised internet settings, so my main machine which trusts Kibana will have synchronised that setting with the test machine. So Kibana being okay might not be related to the certificate trust.

#6

I've tried importing the self generated root ca certificate from my single linux elastic node to the local machine trusted root store on the Windows test machine, as a .crt, a .pem and .p12 - but no matter what format I import this, the local machine does not trust the elastic url https://10.1.140.50:9600 or https://10.1.140.50:5601.

I'm at a loss to know how to get this to work. By accepting the scurity exception in the browser, I have been masking the problem.

Any help you can provide is greatly appreciated.

#7

A small update, but still no solution.
I've removed all of the elastic node ca certifcates that I can find in my test workstation (there were a few old ones present from previous testing) and I've then "Cleared the SSL Cache" from within Internet settings in the Windows control panel. Then I re-added the root ca certificate from my elastic node to the local machine "Trusted Root Authorities" store on the test machine.

Now when I open Chrome and navigate to Kibana at https://10.1.140.50:5601 I get a login prompt as expected, but now I have a solid padlock icon in the URL bar and Chrome says "This connection is secure".

However, when I connect to the same URL using Firefox, I get a security exception and a message that "Error code: SEC_ERROR_UNKNOWN_ISSUER".

Also, the Endpoint data streams and logs are still empty.

I really don't know what to do next to get this working - please help!

#8

Okay - so I got the certificate side of things sorted out.
First of all, make sure you clear the 'ssl cache' in your windows host computer, the one that you want to install the agent on. You can do this in the 'Control Panel | internet Options | Content' tab. Click the button marked 'Clear SSL state'.

Then make sure you install your self signed root ca certificate 'ca.crt' (not the 'elastic-stack-ca.pem' version). Install it to the 'local machine' | 'trusted root authorities' store.

You can check all is well by browsing (in Chrome or Edge - not Firefox, it has it's own cert store!) to your Kibana server and you should get a padlock icon to show you have a secure coinnection.

Now, your Agent will install without any issues and certificate related problems should go away.
You should now see your agent appear in Fleet.

I've got my agents installed and running with no errors, and can upgrade them from 7.11.1 to 7.11.2 in Fleet - but they are still not shipping any data - my data streams remain empty.

Some assistance of where to look to figure out what's going on would be appreciated.
This whole process has been incredibly frustrating so far.

#9

Okay, I finally solved this.
So in Fleet, everything is showing healthy, but no log or metrics data is coming in to Elastic.
The logs still have errors indicating that the ssl certificate is untrusted, despite the ca cert being added to the trusted root store. Chrome accesses the elastic/kibana node without probles and is showing a secure connection.

There is a setting in the action_store.yml file that needs to be added, right at the bottom (I copied the ca cert to the agent folder, but it can live anywhere):

ssl.certificate_authorities: ["C:\\Program Files\\Elastic\\Agent\\elasticsearch-ca.pem"]

This line needs to slot in here:

    default:
      api_key: ----
      hosts:
      - https://your-node.your-domain.local:9200
      ssl.certificate_authorities: ["C:\\Program Files\\Elastic\\Agent\\elasticsearch-ca.pem"]
      type: elasticsearch

Without this line, the ssl certificate from the elastic node won't be trusted, regardless of whatever other hoops you jump through.

Sadly, everytime you change your configuration in Fleet, this line gets wiped out.
There's an 'advanced' config option in the Endpoint Security configuration for windows called windows.advanced.elasticsearch.tls.ca_cert but no matter what I put there (it wants a path to the ca.pem file), this seems to have no impact.

So, it's all working now with manual workarounds - I hope the devs fix it so this line can be added automagically - otherwise there's no point in having an automated deployment methodology if you have to race around and manully update it after each change.

#10

Final update: What I thought might be a bug, isn't - see this topic for a fix to the action_store.yml file output section. :slight_smile:

https://discuss.elastic.co/t/possible-bug-with-elastic-agent-ca-certificate-checks/267253

#11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.