Help filtering JSON, multiple data formats for records

[jpbruckler]

Full disclosure: I'm a complete noob when it comes to the Elastic Stack.

I've hooked up Carbon Black (CB) Response Event Forwarder to logstash/elasticsearch and I'm having some issues with data.

I think it's down to the fact that CB sends different types of data to logstash, and the fields available in that data is not standardized across different event types. I end up getting a bunch of useless data in kibana, mixed in with the information that we want to see.

For example, one type of event, when looking at the data in kibana, has a bunch of fields under a "docs" heading, which doesn't exist in the source JSON. When trying to search for information on a field which, in the raw JSON is called "process_name", the only way it exists to kibana is under docs.process_name.

I attempted to use the mutate plugin to rename the field, with 0 success. I'm realizing that I'm probably going to end up with a lot of filters, but I'm struggling with the syntax, I guess. Example of my filter config is below.

Any kind of pointers would be greatly appreciated.

filter {
    json {
        source => "message"
    }
    mutate {
        rename => { "docs.process_name" => "process_name" }
    }
}

[paz]

The doc field is automatically created by the JSON filter (it's the default target field, where the filter unpacks the data, as per the documentation)

As for nested fields, their reference is different in Logstash, try this way

filter {
    json {
        source => "message"
    }
    mutate {
        rename => { "[docs][process_name]" => "process_name" }
    }
}

[jpbruckler]

Thanks for the response!

I tried this, and I would expect to be able to now perform a search in Discovery like process_name:"chrome.exe", but process_name is not a field that elastic seems to know anything about.

process_name is also not listed as a field when looking at the index from the management screen.

I'm continuing to fiddle with things, with luck I'll get this sorted soon

[paz]

Can you maybe share a sample Logstash event output?

[jpbruckler]

If I could figure out how to get that to work, yes. Currently scratching my head on why bin/logstash -f some-test-config.conf --path.settings /etc/logstash isn't working.

I was hoping to dump to stdout or file.

[paz]

Isn't working as in "won't start" or "won't have the expected behavior"?
Can you paste your full config?

[jpbruckler]

Oh man, --config.test_and_exit - learning a lot today. It's now outputting to a file and stdout. #Progress!

Yesterday we pruned the data that we were sending to logstash because of the sheer amount of useless data we were getting. I think we got rid of the data feed that was sending process_name in a weird format.

I think I know enough to be dangerous now, I at least can see the data, which I wasn't able to see before.