Hi there,
I've ran into similar issues where this kind of solution would be handy to know, but I don't know if its possible!
I have several different storage systems I am trying to monitor using their built-in API. So I use the http-poller input plugin and this works perfectly to get me the information in JSON format.
This works well, except I am trying to return some mount points of the systems which gives me a bit more information about their usage.
My problem is that each system will have different amount of mount points and different names. So is there a way to dynamically store fields depending how many there are?
Here is my JSON:
{
"vol_name": "vol1",
"vol_encryptkey": "",
"vol_guid": "15300780873911754009",
"mountpoint": "/mnt/vol1",
"id": 1,
"is_decrypted": true,
"avail": 7820550885376,
"status": "HEALTHY",
"@timestamp": "2018-11-19T15:18:00.371Z",
"children": [
{
"used": 3934271398848,
"children": [
{
"used": 3923449014144,
"children": [
{
"used": 3203196256128,
"children": [
{
"used": 720252615168,
"children": [
{
"used": 680577779520,
"children": [
{
"used": 19408311360,
"mountpoint": "/mnt/vol1/dataset1/backup1",
"path": "vol1/dataset1/backup1",
"type": "dataset",
"id": 114,
"used_pct": 0,
"avail": 3546756324416,
"name": "backup1",
"status": "-"
},
{
"used": 427900701888,
"mountpoint": "/mnt/vol1/dataset1/backup2",
"path": "vol1/dataset1/backup2",
"type": "dataset",
"id": 115,
"used_pct": 10,
"avail": 3546756324416,
"name": "backup2",
"status": "-"
},
{
"used": 233268635328,
"mountpoint": "/mnt/vol1/dataset1/backup3",
"path": "vol1/dataset1/backup3",
"type": "dataset",
"id": 116,
"used_pct": 6,
"avail": 3546756324416,
"name": "backup3",
"status": "-"
}
],
"mountpoint": "/mnt/vol1/dataset1",
"path": "vol1/dataset1",
"type": "dataset",
"id": 113,
"used_pct": 16,
"avail": 3546756324416,
"name": "dataset1",
"status": "-"
},
{
"used": 16759790400,
"children": [
{
"used": 16759659456,
"mountpoint": "/mnt/vol1/dataset2/backup1",
"path": "vol1/Backups/dataset2/backup1",
"type": "dataset",
"id": 118,
"used_pct": 0,
"avail": 3546756324416,
"name": "backup1",
"status": "-"
}
],
"mountpoint": "/mnt/vol1/dataset2/backup1",
"path": "vol1/Backups/dataset2",
"type": "dataset",
"id": 117,
"used_pct": 0,
"avail": 3546756324416,
"name": "dataset2",
"status": "-"
}
]
}
],
"mountpoint": "/mnt/vol1/dataset3",
"path": "vol1/dataset3",
"type": "dataset",
"id": 101,
"used_pct": 52,
"avail": 3546756324416,
"name": "dataset3",
"status": "-"
}
],
"mountpoint": "/mnt/vol1",
"path": "vol1",
"type": "dataset",
"id": 100,
"used_pct": 52,
"avail": 3546756324416,
"name": "vol1",
"status": "-"
}
],
"vol_encrypt": 0,
"@version": "1",
"used_pct": "50%",
"is_upgraded": true,
"name": "vol1",
"used": 8122367717376
}
]
}
So each system could have a varying amount of children if that makes sense?
So ideally I'm looking for away to create fields like this:
vol1.dataset1.backup1.used_pct:0
vol1.dataset1.backup2.used_pct:10
vol1.dataset1.backup3.used_pct:6
vol1.dataset1.used_pct:16
vol1.dataset2.backup1.used_pct:0
vol1.dataset2.used_pct:0
And so on.
I know I could create these fields manually in logstash but I want to avoid doing that because it isn't very dynamic because for example
System1 might have the above mount points
System 2 might have only dataset1
System 3 might have dataset3
etc etc.
Is there any good way to tackle this?
I hope that makes sense, its kind of hard to explain!
Thanks in advance!
Lewis