How to store variable length JSON into fields?


(Lewis Barclay) #1

Hi there,

I've ran into similar issues where this kind of solution would be handy to know, but I don't know if its possible!

I have several different storage systems I am trying to monitor using their built-in API. So I use the http-poller input plugin and this works perfectly to get me the information in JSON format.

This works well, except I am trying to return some mount points of the systems which gives me a bit more information about their usage.

My problem is that each system will have different amount of mount points and different names. So is there a way to dynamically store fields depending how many there are?

Here is my JSON:

{
"vol_name": "vol1",
"vol_encryptkey": "",
"vol_guid": "15300780873911754009",
"mountpoint": "/mnt/vol1",
"id": 1,
"is_decrypted": true,
"avail": 7820550885376,
"status": "HEALTHY",
"@timestamp": "2018-11-19T15:18:00.371Z",
"children": [
	{
		"used": 3934271398848,
		"children": [
			{
				"used": 3923449014144,
				"children": [
					{
						"used": 3203196256128,
						"children": [
							{
								"used": 720252615168,
								"children": [
									{
										"used": 680577779520,
										"children": [
											{
												"used": 19408311360,
												"mountpoint": "/mnt/vol1/dataset1/backup1",
												"path": "vol1/dataset1/backup1",
												"type": "dataset",
												"id": 114,
												"used_pct": 0,
												"avail": 3546756324416,
												"name": "backup1",
												"status": "-"
											},
											{
												"used": 427900701888,
												"mountpoint": "/mnt/vol1/dataset1/backup2",
												"path": "vol1/dataset1/backup2",
												"type": "dataset",
												"id": 115,
												"used_pct": 10,
												"avail": 3546756324416,
												"name": "backup2",
												"status": "-"
											},
											{
												"used": 233268635328,
												"mountpoint": "/mnt/vol1/dataset1/backup3",
												"path": "vol1/dataset1/backup3",
												"type": "dataset",
												"id": 116,
												"used_pct": 6,
												"avail": 3546756324416,
												"name": "backup3",
												"status": "-"
											}
										],
										"mountpoint": "/mnt/vol1/dataset1",
										"path": "vol1/dataset1",
										"type": "dataset",
										"id": 113,
										"used_pct": 16,
										"avail": 3546756324416,
										"name": "dataset1",
										"status": "-"
									},
									{
										"used": 16759790400,
										"children": [
											{
												"used": 16759659456,
												"mountpoint": "/mnt/vol1/dataset2/backup1",
												"path": "vol1/Backups/dataset2/backup1",
												"type": "dataset",
												"id": 118,
												"used_pct": 0,
												"avail": 3546756324416,
												"name": "backup1",
												"status": "-"
											}
										],
										"mountpoint": "/mnt/vol1/dataset2/backup1",
										"path": "vol1/Backups/dataset2",
										"type": "dataset",
										"id": 117,
										"used_pct": 0,
										"avail": 3546756324416,
										"name": "dataset2",
										"status": "-"
									}
								]
							}
						],
						"mountpoint": "/mnt/vol1/dataset3",
						"path": "vol1/dataset3",
						"type": "dataset",
						"id": 101,
						"used_pct": 52,
						"avail": 3546756324416,
						"name": "dataset3",
						"status": "-"
					}
				],
				"mountpoint": "/mnt/vol1",
				"path": "vol1",
				"type": "dataset",
				"id": 100,
				"used_pct": 52,
				"avail": 3546756324416,
				"name": "vol1",
				"status": "-"
			}
		],
		"vol_encrypt": 0,
		"@version": "1",
		"used_pct": "50%",
		"is_upgraded": true,
		"name": "vol1",
		"used": 8122367717376
	}
]
}

So each system could have a varying amount of children if that makes sense?

So ideally I'm looking for away to create fields like this:

vol1.dataset1.backup1.used_pct:0
vol1.dataset1.backup2.used_pct:10
vol1.dataset1.backup3.used_pct:6
vol1.dataset1.used_pct:16
vol1.dataset2.backup1.used_pct:0
vol1.dataset2.used_pct:0

And so on.

I know I could create these fields manually in logstash but I want to avoid doing that because it isn't very dynamic because for example

System1 might have the above mount points
System 2 might have only dataset1
System 3 might have dataset3

etc etc.

Is there any good way to tackle this?

I hope that makes sense, its kind of hard to explain!

Thanks in advance!

Lewis


(Lewis Barclay) #2

Anyone?