Mutate gsub pattern help for filter

Elastic Stack Logstash
1

Hello, still new using regular expressions I currently am working on this to parse in my "message":

{"groupByActivity":false,"exampleDevices":[{"ip":"1.2.3.4","hostname":"example.hostname.com","sid":123456,"subnet":"some-subnet","identifier":"some-node-example.com","mac":null,"did":1234}],"groupingIds":["1234"],"userTriggered":false,"activityId":"1234","externalTriggered":false,"id":"example-id","pinned":false,"title":"Possible SSL Command and Control","relatedBreaches":[{"timestamp":123456,"threatScore":123,"modelName":"Container / Unusual External Connections (example)","pbid":12345}],"details":[[{"contents":[{"type":"device","values":[{"ip":"1.2.3.4","hostname":"example.hostname.com","sid":1234,"subnet":"some-subnet","identifier":"some-example.com","mac":null,"did":123456}],"key":null}],"header":"Some example"}],[{"contents":[{"type":"string","values":["123b456c"],"key":"example client hash"}],"header":"Suspicious Application"},{"contents":[{"type":"timestampRange","values":[{"start":123456,"end":123456}],"key":"Time"},{"type":"externalHost","values":[{"ip":null,"hostname":"example.com"}],"key":"Endpoint"},{"type":"percentage","values":[1234],"key":"Hostname rarity"},{"type":"timestamp","values":[1234],"key":"Hostname first observed"},{"type":"externalHost","values":[{"ip":"1.2.3.4","hostname":"1.2.3.4"}],"key":"Most recent example"},{"type":"string","values":["example-test"],"key":"Most recent example"},{"type":"integer","values":[1234],"key":"Destination port"},{"type":"integer","values":[123],"key":"Connection count"},{"type":"dataVolume","values":[1234],"key":"Total data in"},{"type":"dataVolume","values":[1234],"key":"Total data out"},{"type":"string","values":["Unknown"],"key":"Validation Status"},{"type":"string","values":["Unknown"],"key":"Issuer"}],"header":"Suspicious example"}],[{"contents":[{"type":"device","values":[{"ip":"1.2.3.4","hostname":"another-example.com","sid":1324,"subnet":null,"identifier":"some-example.com","mac":null,"did":1234}],"key":null},{"type":"device","values":[{"ip":"1.2.3.4","hostname":"one-example.com","sid":1234,"subnet":null,"identifier":"so-many-examples.com","mac":null,"did":1234}],"key":null},{"type":"device","values":[{"ip":"1.2.3.4","hostname":"too-many-examples.com","sid":1234,"subnet":"12-34","identifier":"stop-with-the-examples.com","mac":null,"did":1234}],"key":null},{"type":"device","values":[{"ip":"1.2.3.4","hostname":"wow-another-example.com","sid":1324,"subnet":null,"identifier":"ok.example.com","mac":null,"did":1234}],"key":null},{"type":"device","values":[{"ip":"1.2.3.4","hostname":"examples.i.give.up.com","sid":1234,"subnet":null,"identifier":"mwhy-are-there-so-many-examples.com","mac":null,"did":1234}],"key":null}],"header":"Other Connecting Devices Include"}]],"children":["12-34-ab-cd"],"summariser":"example_summary","acknowledged":false,"summary":"The device some-example...\n\n stuff.","periods":[{"start":1234,"end":1234}],"attackPhases":[123],"exampleScore":23}"

it looks like the json isn't formatted correctly and the ":" is contained within the string, would a gsub replacement work? or am I going down the wrong path to solving this parsing issue?

Sorry for the bad formatting, for some odd reason the usual

doesn't do right trick when providing the example.

my current filter looks like this:

filter{
    json { source => "message" }
    json { source => "message" }
    mutate { gsub => [ "message", "^[^{]+", "" ] }

}

please let me know if I can provide more to aid in help.

2

You have an extra " at the end of your sample data. You can fix it using

mutate { gsub => [ "message", '"$', "" ] }

before the json filter.

3

thanks for the catch, after attempting to do another json { source => "message" }

it seems to not parse after group activity, I'm stuck attempting to parse that portion as well.

I added another json { source => "message" } but nothing happened.

Example of where it is stopping:

{\"groupByActivity\":false,
    \"exampleDevices\":[{\"ip\":\"1.2.3.4\",
        \"hostname\":\"example.hostname.com\",
        \"sid\":123456,\"subnet\":\"some-subnet\",
        \"identifier\":\"some-node-example.com\",
        \"mac\":null,

is it stopping because groupActivity's value is not wrapped in quotes?

4

is it also the backslashes as well?

5

Hard to say, you really need to provide a reproducible example.

6

I worked on it in vs code and what helped was getting rid of the backslashes after the auto formatting. I'm hoping that the json { source => "message" } works in the same fashion after getting rid of the backslashes. would it be something like this? :

mutate { gsub => [ "message", "[\\]", "" ] }

if so, it's not working, why is that?

7

Again, you really need to provide a reproducible example. That mutate is the standard way to remove backslashes.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.