Elastic-agent osquery through logstash doesn't work

heikis · August 24, 2022, 9:32am

Hello. We are managing Elastic Agents in Fleet. Entire Elastic Stack is on version 8.3.3.

We have problems doing osqueries in Kibana- the query seems to be successful but we get no results and no indexes are created:
"2 agents have responded, no osquery data has been reported".

In Fleet we have configured agent's output to logstash. In logs we see errors:
"Failed to publish events caused by: write tcp x.x.x.x:51787 (agent) -> y.y.y.y:5046 (logstash): wsasend: An existing connection was forcibly closed by the remote host".

Is osquery supposed to work with logstash as output on agents? I can confirm that osquery is working when agent output is set to elasticsearch, but we need Logstash to work.

Any ideas/suggestions/things to try?

Thanks!

BenB196 · August 24, 2022, 11:43am

Looks like this is a bug that should be fixed in the next release (Osquerybeat: Fix osquerybeat is not running with logstash output by aleksmaus · Pull Request #674 · elastic/elastic-agent · GitHub)

aleksmaus · August 24, 2022, 3:19pm

@BenB196 That PR fixes the issue with the agent not running osquerybeat for any other outputs besides elasticsearch. It was merged into 8.3 here [8.3](backport #674) Osquerybeat: Fix osquerybeat is not running with logstash output by mergify[bot] · Pull Request #681 · elastic/elastic-agent · GitHub and was released as a part of 8.3.3, so the agent should run osquerybeat.

There was another issue where the data was not routed/saved properly into the elasticsearch datastream because the events where missing the datastream properties.

github.com/elastic/beats

Osquerybeat: Add missing data_stream to events in order to support logstash configuration better

elastic:main ← aleksmaus:fix/osquery_data_stream

opened 10:36PM - 28 Jul 22 UTC

aleksmaus

+38 -7

## What does this PR do? Adds the ```add_data_stream``` processors that appen…ds the ```data_stream``` fields to the events. This enabled better compatibility with the logstash configuration allowing it to figure out the destination datastream for the results of the osquery. ## Why is it important? Better support osquery with logstash as the output. ## Checklist - [x] My code follows the style guidelines of this project - [x] I have commented my code, particularly in hard-to-understand areas ## How to test this PR locally Here is an example logstash pipeline configuration that was not working before this change: ``` input { elastic_agent { port => 5044 ssl => true ssl_certificate_authorities => ["certs/ca/ca.crt"] ssl_certificate => "certs/logstash.crt" ssl_key => "certs/logstash.pkcs8.key" ssl_verify_mode => "force_peer" } } filter { mutate { rename => ["_host", "host" ] } } output { elasticsearch { hosts => "https://<redacted>.us-west2.gcp.elastic-cloud.com:443" data_stream => true ssl => true user => elastic password => <redacted> } } ``` After the change the event has additional datastream attributes ``` "data_stream" => { "namespace" => "default", "type" => "logs", "dataset" => "osquery_manager.result" }, ``` ## Screenshots Verified the result are now reaching the osquery_manager datastream: <img width="1282" alt="Screen Shot 2022-07-28 at 6 08 15 PM" src="https://user-images.githubusercontent.com/872351/181648483-fa8fea43-4732-4924-9eb7-2e6995ed28f8.png">

This was merged into 8.4 and should be a part of 8.4.0 release.

heikis · August 25, 2022, 10:31am

We got osquery working in our environment.
Upgraded the elasticsearch and logstash nodes to 8.4.0 but that did not resolve the problem.
Then upgraded osquerymanager integration from 1.3.1 to 1.4.1 - still nothing.
Then upgraded Fleet Server agent and all endpoint Elastic-agents to 8.4.0 and osqueries are now working. Thank you for your help!

system · September 22, 2022, 10:31am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.