Logstash (elastic stack 5.6) grok regex

Elastic Stack Logstash
1

Hi everyone!
There is a problem I struggle with for a while:
I have data in a json format:

"documentsList": [{
		"commandScn": "108058599",
		"commandCommitScn": "108058600",
		"commandSequence": "0",
		"commandType": "UPDATE",
		"commandTimestamp": "2017-08-22 14:37:53+03:000",
		"objectDBName": "DEV2",
		"objectSchemaName": "YISHAIN",
		"objectId": "CUSTOMERS",
		"changedFieldsList": [{
				"fieldId": "CUSTOMER_ID",
				"fieldType": "NUMBER",
				"fieldValue": "17",
				"fieldChanged": "N"
			}, {
				"fieldId": "CUSTOMER_FIRST_NAME",
				"fieldType": "VARCHAR2",
				"fieldValue": "Daniel",
				"fieldChanged": "N"
			}, {
				"fieldId": "CUSTOMER_LAST_NAME",
				"fieldType": "VARCHAR2",
				"fieldValue": "Washington",
				"fieldChanged": "N"
			}, {
				"fieldId": "CUSTOMER_COUNTRY",
				"fieldType": "VARCHAR2",
				"fieldValue": "France",
				"fieldChanged": "N"
			}, {
				"fieldId": "CUSTOMER_CITY",
				"fieldType": "VARCHAR2",
				"fieldValue": "La Roche-sur-Yon",
				"fieldChanged": "N"
			}
		}]
}]

What I want is to fetch the values of the fields: fieldId, fieldType, fieldValue, fieldChanged etc. I want to aggregate by these terms afterwards in kibana 5.6.

My questions are:

  1. How can I tell grok to fetch everything after a word (because I want the values of the keys, not the leys themselves)? does elasticsearch support lookbehind? if so - can anyone give me an example of how to accomplish this?
  2. If elastic's regex doesn't support it, how can I get this done?

Thanks!

2

I don't understand. The grok filter is used to parse a text using regular expressions and extract fields from the text. What you have isn't a text but an hierarchical data structure. You can certainly apply a grok filter on individual fields in this structure but that doesn't sound like what you're asking for.

3

Hi, I'm new to the elastic stack, so I don't know the ropes yet.
How can I achive my goal using other filters? I tries useing mutate and json filters, to no avail.

4

You can send what you have to Elasticsearch, but it's not clear if what you've shown is the best representation of the data. Perhaps you want one document per item in the changedFieldsList array? It depends on what you want to do with the data.

Sorting out how to store your data is probably better to ask in the Elasticsearch group.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.