Hello Elastic team,
I am testing elastic agent and we are facing an issue/bug not documented in fleet troubleshooting article, this is why I would like to learn how to Troubleshoot elastic-agent communication with Fleet.
The fleet agent's Status appears as "Offline" even though I am able to receive logs from it. Meaning that when I select the "Logs" tab under Fleet/Agents/[host name] , I am receiving elastic_agent event dataset message in (near) real time, as well as in Kibana Discover panel.
To troubleshoot this issue, I have tried to enable "Debug" logging on Kibana (logging.verbose:true) and on the agent (agent.logging.level: debug ) but couldn't find any error/event log that indicates that the agent is offline.
I have tried to perform that by doing a regex search in kibana log file, for example by looking for request from the agent :
cat kibana.log |grep fleet |grep xxxx_Agent_ID_xxxxxxxxxxxxx |grep POST
and the newest log that I receive is :
{"type":"response","@timestamp":"2021-05-13T10:02:27+00:00","tags":[],"pid":16971,"method":"post","statusCode":200,"req":{"url":"/api/fleet/agents/xxxx_Agent_ID_xxxxxxxxxxxxx/checkin","method":"post","headers":{"host":"xxx_Kibana_IP_xxx:5601","user-agent":"Elastic Agent v7.12.1","content-length":"736","accept":"application/json","content-type":"application/json","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"xxx_Agent_IP_xxx","userAgent":"Elastic Agent v7.12.1"},"res":{"statusCode":200,"responseTime":228201,"contentLength":33},"message":"POST /api/fleet/agents/xxxx_Agent_ID_xxxxxxxxxxxxx/checkin 200 228201ms - 33.0B"}
Which seems to indicate that on 13th May, when i installed it, the agent has sent this request, but not anymore (currently 17th May).
This is why , i guess, from Kibana side, fleet shows the agent status as "Offline".
And, from the Agent side, when I log to the elastic agent host, it seems like everything is ok , it shows only this kind of logs (the same as those that we see in Flee management agent's logs) :
11:49:04.593 elastic_agent [elastic_agent][debug] Adding 1 file to watch
11:49:04.593 elastic_agent [elastic_agent][info] No configuration change
Thanks for your help.