Agent stuck sending documents over and over

Elastic Security Endpoint Security
1

Not sure what this is, as far as I know it’s a first for us, but we have an agent on a LInux client that’s continously - 2 or 3 times per second - logging that it’s sending documents to Elasticsearch, the actual message is

‘BulkQueueConsumer.cpp:345 Sent 400 documents to Elasticsearch’

It has - as one could imagine - a very noticable effect on the client’s performance, according to the user.

The agent also logs that there are unsynced messages, which sounds like the logical explanation to what’s happening - that the agent tries to push the messages it can’t purge or perhaps send to the backend. No other agents log these messages though so it doesn’t seem like the actual reason..

The only other message coming in from that client is that it’s unable to open smaps file for various PIDS, message ProcFile.cpp:701 Unable to open smaps file for PID

Not been able to restart the client to see if changes anything, but would of course like to know what could be causing this should it happen again.

2

Endpoint stores documents to disk first and then sends them to Elasticsearch.

“Unsynced document log store:” means one of the files of documents has not yet been sent to Elasticsearch. It’s mostly a debugging kind of log. Once all the documents in a file are sent you will see logs like “Pruning fully synced document log store:”.

“BulkQueueConsumer.cpp:345 Sent 400 documents to Elasticsearch” means there’s a lot of activity on the box.

That user could use the Endpoint Top command: endpoint/EndpointTopCommand.md at main · elastic/endpoint · GitHub to see what files are causing the most activity. The event filters could be create in kibana based on that data: Event filters | Elastic Docs

Another potential source of data would be an endpoint diagnostic package. Which has a file called metrics.json within it that has top offenders by file name.

1 Like
3

Cheers Nick, and sorry for the delayed response. It appears that the agent lost the ability to prune documents from its own log files, causing it to resend the same documents repeatedly. After fully uninstalling and reinstalling the agent, the issue ceased and has not recurred. Consequently, we won’t investigate further unless the problem reappears.