Osquery Manager does not return results from curl table

hwang · May 18, 2022, 5:44am

Hi Elastic team,

Running a query from the curl table returns an empty result. I've confirmed that other queries work and curl_certificates work as well.

SELECT * FROM curl WHERE url = 'https://www.google.com'

Is there any additional configuration that needs to be enabled for the curl table to work on OSQuery Manager?

Version Info: Elasticstack 8.2, OSQuery Manager 1.2.1

Broken curl:

Working curl_certificate:

aleksmaus · May 18, 2022, 5:52pm

If you run osquery standalone in interractive mode:

➜  elastic ./osqueryd -S
Using a virtual database. Need help, type '.help'
osquery> SELECT * FROM curl WHERE url = 'https://www.google.com';
W0518 13:35:22.938369 228421120 tls.cpp:101] Cannot read TLS server certificate(s): /var/osquery/certs/certs.pem
W0518 13:35:23.049543 228421120 curl.cpp:83] Error making request: certificate verify failed
+------------------------+--------+------------+---------------+-----------------+-------+--------+
| url                    | method | user_agent | response_code | round_trip_time | bytes | result |
+------------------------+--------+------------+---------------+-----------------+-------+--------+
| https://www.google.com | GET    | osquery    |               |                 |       |        |
+------------------------+--------+------------+---------------+-----------------+-------+--------+

You will see that it checks the the specific directory for the certs, the example was on Mac OS.
Here is the list of default paths defined in osquery for the different platforms:

github.com

osquery/osquery/blob/2cd5b42c8f8fa52c6d251c6537595c7c59c90f4c/osquery/utils/config/default_paths.h

/**
 * Copyright (c) 2014-present, The osquery authors
 *
 * This source code is licensed as defined by the LICENSE file found in the
 * root directory of this source tree.
 *
 * SPDX-License-Identifier: (Apache-2.0 OR GPL-2.0-only)
 */

/**
 * @brief A series of platform-specific home folders.
 *
 * There are several platform-specific folders where osquery reads and writes
 * content. Most of the variance is due to legacy support.
 *
 * OSQUERY_HOME: Configuration, flagfile, extensions and module autoload.
 * OSQUERY_DB_HOME: Location of RocksDB persistent storage.
 * OSQUERY_LOG_HOME: Location of log data when the filesystem plugin is used.
 */

This file has been truncated. show original

That error should be in osquerybeat log.

The official osquery bundle contains, the certs.pem file, but we do not install it at the moment. We probably should and possibly override this with --tls_server_certs option to point to our private install location, need to research. There is the osquery ticket that mentions some potential "side-effect" of using this option: Cannot fetch HTTPS using curl table when using the `--tls_server_certs` flag for TLS pinning · Issue #6964 · osquery/osquery · GitHub, and it looks that the original purpose of this flag might have been different.

The short term fix at the moment is to copy the certs.pem from osquery repository osquery/certs.pem at master · osquery/osquery · GitHub or osquery official distribution package into the default path for the given platform /var/osquery/certs/certs.pem for Mac OS, for example.

hwang · May 19, 2022, 3:14am

Hi Aleksandr,

Thank you for the prompt reply and I believe your diagnosis is correct for HTTPS websites. However, I'm also seeing the save issue for HTTP websites that do not require any TLS certificate configuration. See screenshot below:

Best,
Han

aleksmaus · May 19, 2022, 12:21pm

For that particular url you get redirected to HTTPS url

And then you are back to square one with TLS certs error:

osquery> select * from curl where url='http://ifconfig.me';
W0519 08:18:30.609016 371273216 tls.cpp:101] Cannot read TLS server certificate(s): /var/osquery/certs/certs.pem
W0519 08:18:30.714504 371273216 curl.cpp:83] Error making request: certificate verify failed
+--------------------+--------+------------+---------------+-----------------+-------+--------+
| url                | method | user_agent | response_code | round_trip_time | bytes | result |
+--------------------+--------+------------+---------------+-----------------+-------+--------+
| http://ifconfig.me | GET    | osquery    |               |                 |       |        |
+--------------------+--------+------------+---------------+-----------------+-------+--------+

hwang · May 19, 2022, 4:44pm

Thank you for the clarification and this makes sense. Unfortunately, we won't be able to do a local copy of the certificate folder. Would bundling the official certs.pem be possible at a later Osquery Manager release?

aleksmaus · May 19, 2022, 4:56pm

Yes, we are working on bundling the certs file with our solution, hopefully for 8.3 release.

hwang · May 19, 2022, 5:08pm

Perfect, thank you!

aleksmaus · May 19, 2022, 5:28pm

Thank you for letting us know!

system · June 16, 2022, 5:29pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.