OSQuery field types

Elastic Stack Beats
1

I just realized all osquery fields get set to type 'string'.

In osquery fields have appropriate types (for example for a temperature sensor it will have a string name and a double celsius temperature)

But the json osqueryd.results.log that filebeat's osquery module reads from doesn't have any type hints.

{"name":"temperatures","calendarTime":"Wed Jul 17 05:15:37 2019 UTC","unixTime":1563340537,"epoch":0,"counter":10,"columns":{"fahrenheit":"123.1","name":"GPU Die"},"action":"added"}

What's the best practices here? I need to have the right formats so I can do things like plot CPU usage and temperatures and have gauges and whatnot. The field names can change (or new ones appear at least) every time osquery gets updated (for queries that just do select *) or one of the configured queries gets tweaked or a new osq pack enabled... Keeping ahead of those changes on the elastic index side of things seems like a losing battle

2

Hi @DPattee,

In principle there is nothing that forces a field to be stored as string in Elasticsearch for this module, but it is true that all fields are sent as this type, so Elasticsearch automatically chooses this data type.

Something you can try is to add the convert processor to your osquery module configuration for the fields you know that should be stored as numbers.

It'd be something like:

- module: osquery
  processors:
  - convert:
      fields:
        - {from: "osquery.result.fahrenheit", type: "double"}
      ignore_missing: true
      fail_on_error: false

Let me know if this works for you.

Other option would be to add an OSQuery module to metricbeat, so it considers these fields as metrics, but this is not supported yet.

3

Ahh, good tip, that will prevent some number of problems in the future. I'll have to reindex all my current data though.

As for osquery being in metricbeat, that's actually where I looked for it originally. I'd say 75% of the things I'm planning on tracking with osquery actually match the metric style thought pattern vs the log style pattern.

4

Hmm, I've never used the processors/convert option before so I might be configuring it wrong.

2019-07-17T21:20:47.476-0700 ERROR [reload] cfgfile/list.go:96 Error creating runner from config: Fileset osquery/processors is configured but doesn't exist

Here's what I updated my modules.d/osquery.yml to:

- module: osquery
  result:
    enabled: true
  processors:
  - convert:
      fields:
        - {from: "osquery.result.columns.memory_gb", type: "double"}
        - {from: "osquery.result.columns.creation_time", type: "double"}
        - {from: "osquery.result.columns.failed_login_count", type: "integer"}
        - {from: "osquery.result.columns.failed_login_timestamp", type: "double"}
        - {from: "osquery.result.columns.fahrenheit", type: "double"}
        - {from: "osquery.result.columns.actual_rpm", type: "integer"}
        - {from: "osquery.result.columns.max_rpm", type: "integer"}
        - {from: "osquery.result.columns.min_rpm", type: "integer"}
        - {from: "osquery.result.columns.target_rpm", type: "integer"}
        - {from: "osquery.result.columns.executions", type: "integer"}
        - {from: "osquery.result.columns.interval", type: "integer"}
        - {from: "osquery.result.columns.output_size", type: "integer"}
        - {from: "osquery.result.columns.wall_time", type: "integer"}
        - {from: "osquery.result.columns.avg_user_time", type: "integer"}
        - {from: "osquery.result.columns.avg_system_time", type: "integer"}
        - {from: "osquery.result.columns.average_memory", type: "integer"}
        - {from: "osquery.result.columns.last_executed", type: "integer"}
      ignore_missing: true
      fail_on_error: false

Removing the entire processors section gets it back up and running.
I double-checked that ignore & fail were lined up with fields, and processors was lined up with result, and the individual field lines were one indent further in, and all the indents are double spaces... So I think I have the formatting done right at least

5

@DPattee oh sorry, my mistake. Processors configuration must be placed at the top level configuration, in the main configuration file, and then they are applied to all events, or at the input level, so it is applied to an only input. To apply them when you are using a module, you need to override the input, something like this:

- module: osquery
  result:
    enabled: true
    input:
      processors:
      - convert:
          fields:
            - {from: "osquery.result.columns.memory_gb", type: "double"}
           ...

You can read more about processors here.

Feel free to open an enhancement request for that :slightly_smiling_face:

6

Oh that was my fault... I'd "read" that page, and a couple others, several times, but only looked at the examples. Totally missed the line that says exactly what you said.

Similarly, for Filebeat modules, you can define processors under the input section of the module definition.

Thanks for the help!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.