Forgive me is this is a silly question but i have installed the agents, under Hosts in security it shows them connected, yet there is no data in the dashboards, everything says "all values returned zero" or "no data to display" on all pages under Security.
If i dig thru what there the only thing that seems to populate is the "Events" section under the hosts but this appears to be housekeeping data for the agent and not actual logs.
Ive read thru the documentation and I'm either missing something or just don't get what its trying to tell me.
I should note this is running on 7.9 and i cannot yet upgrade due to other integrations, long story.
Also i can see traffic inbound by packet capture/tcpdump so the agent is sending something and there appears to be more than the housekeeping events would suggest but that's a guess
Any pointers in the right direction would be greatly appreciated as I'm feeling rather stuck.
It sounds like you are trying to enable Endpoint Security in your system? To enable it in 7.9 go to Ingest Manager -> Fleet -> Configurations. On that page click on the configuration you want to add Endpoint Security to. On the page you're taken to click "Add integration" to search for an add Endpoint Security.
Thanks for the reply, ive already done that though... where I'm stuck is that there appears to be no data being process... all the dashboards are blank and i cannot determine the reason for this.,
Are you by chance using self signed SSL certificates?
Can you confirm if Elastic Agent is in a healthy state or if not share what state it is in? You should see the Agent you installed listed on the Ingest Manager -> Fleet page.
Can you look in Endpoint's logs to see why it can't connect to Kibana or Elasticsearch? Endpoint's logs are found in c:\Program Files\Elastic\Endpoint\state\log (Windows), /Library/Elastic/Endpoint/state/log (macOS), /opt/Elastic/Endpoint/state/log (Linux). Endpoint logs each time it creates an HTTP[S] connection and whether the connection succeeds or fails. Searching for the string "HTTP" should be a good way to narrow down what you're looking at.
I know you said you have restrictions preventing you from upgrading your stack past 7.9. Once you are able, I really recommend upgrading so you can use newer Elastic Agent updates. We've made a lot if improvements in the past year.
I seem to be seeing a lot of these repeating:
{"@timestamp":"2021-08-23T18:16:43.6550600Z","agent":{"id":"4a71e16e-6c87-46ba-b64c-10b8eb92b9b7","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"info","origin":{"file":{"line":1392,"name":"HttpLib.cpp"}}},"message":"HttpLib.cpp:1392 Establishing GET connection to [https://servernameurl:9200/_cluster/health]","process":{"pid":4552,"thread":{"id":9136}}}
{"@timestamp":"2021-08-23T18:16:43.37770700Z","agent":{"id":"4a71e16e-6c87-46ba-b64c-10b8eb92b9b7","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":4552,"thread":{"id":9136}}}
I wonder could this have to do with my not enabling anonymous access? Or have i set the authentication values incorrectly? If i try to access this URL directly i get the 403 error
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/health] is unauthorized for user [anonymous_user]"}],"type":"security_exception","reason":"action [cluster:monitor/health] is unauthorized for user [anonymous_user]"},"status":403}
No, you should not need to allow anonymous access to /_cluster/health. The Endpoint process should be making this request using an Elasticsearch API key to authenticate. The API key is generated by Fleet and it should have the required permissions.
If you are not seeing a message like HTTP Status Code (NNN): <error> in the logs then the connection might be failing at the TLS layer before it gets to the HTTP part. To troubleshoot this, I would try tcpdump or wireshark to capture the traffic to tcp port 9200. This should at least tell us if it's making a successful connection or provide a TLS error. If packet capture isn't an option then the Elasticsearch audit log can show if those _/cluster/health requests are being made and whether they are being rejected (it won't show connection issues).
Hmm, ok, thats a little odd in that once i did this it seemed to connect so something else is wrong I'm guessing.
My basic issue though is that even with the agent connected the dashboards appear blank telling me only "all values returned zero" or similar instead of displaying the results of the traffic.
It would appear not....
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open metrics-endpoint.metadata_current_default xfTPKkHHRSGFmWLKhNp05g 1 0 0 0 208b 208b
This is twice now ive seen this same issue once when trying to get 7.9 to work and now again on and updated 7.14 box... in both cases i can see the agents connected, but no event data ... im clearly missing something and have been unable to determine what
This is twice now ive seen this same issue once when trying to get 7.9 to work and now again on and updated 7.14 box... in both cases i can see the agents connected, but no event data ... im clearly missing something and have been unable to determine what
Do you see the same Endpoint log messages in 7.14 as you saw before? Specifically this one repeating: {"line":65,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:65 Elasticsearch connection is down","process":{"pid":4552,"thread":{"id":9136}}}
If this is the case then the Endpoint is having a problem connecting to ES.
On your 7.14 box, are you also using updated 7.14 Agents/Endpoint as well with Fleet Server? The issue with the Endpoint not being able to establish a connection with ES was most prevalent in 7.9.
No im not seeing those same errors on the 7.14 box.
The basic symptom is that i can see that the hosts are connected with recent activity on the hosts page, but there is no data about them on any dashboard or even the hosts page authentication or dns blocks.
It still seems like your Endpoints are never streaming data to ES. The Agent is successfully connecting based on your observation that you see healthy Agents, but the fact that you're not seeing any data in the Hosts page seems to indicate that the Endpoint specific docs are still not making their way to ES. The Endpoint establishes it's own connection to ES, so it's possible that it's not correctly streaming while the Agent itself is.
Can you check the Endpoint logs again as you did before, but can you check for messages that look like this?
09:43:09.101
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] BulkQueueConsumer.cpp:224 Sent 180 documents to Elasticsearch
This will indicate that the Endpoint is successfully streaming data to ES. If you never see these messages, there may be more errors occurring. Searching the logs for "error" could help point to more problems. Also, if you're okay with sharing your Endpoint log file, we can take a closer look ourselves.
Further, can you check "Stack Management" > "Index Management" > "Data Streams" ? If the Endpoint is successfully streaming, you should see several data streams with "endpoint" in the name.
My apologies for late reply, was involved on another urgent project.
I do not see any logs like your example. Oddly enough i also don't see any errors.
I see a lot of these:
{"log.level":"info","@timestamp":"2021-08-27T12:55:57.999Z","log.origin":{"file.name":"application/periodic.go","file.line":101},"message":"No configuration change","ecs.version":"1.6.0"}, not much else save for service housekeeping.
Also, oddly enough i only see the agent folder, not an endpoint folder, is it possible that it only installed one and not the other?
Im asking as im curious, having done some added trial including the cloud based option and observing that this installed 2 services, not one, yet mine only installed the one. Something seems not right in that.
I have installed beats, but now how to manage data in dashboard? also there are some errors with the eduhelphub.com host page authentication Looking for the answer.
This is going to be a dumb question I'm sure, but how do i check the policy?
Also it appears none of my agent are falling under any policy as they all read 0.
what i find most perplexing though is that whenever i bring up the fleet page it acts as though it isn't setup and is asking me to add a fleet server, even though the integration is installed.
Ive missed something that clear but i truly don't know what.
From Kibana, go to Ingest Manager -> Fleet -> Agent Policies
It should provide you a list of policy currently configured. The summary page also shows you number agents and integrations tied to that policy.
If you clicking on individual policy, it shows you individual integration tied to the policy. Make sure the policy tied to the machine has Endpoint Security integrated. If that's not the case, you can always add that integration by clicking Add Integration.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
