Need help Flattening the json array

[karnamonkster (Karan)]

Hi,

I am trying to flatten the following json:

{
  "beans" : [ {
"name" : "Hadoop:service=NameNode,name=NameNodeInfo",
"modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
"Total" : 976194928640,
"UpgradeFinalized" : true,
"ClusterId" : "CID-289afb0c-9775-46d6-8f22-4d147579f2c5",
"Version" : "2.7.3, rbaa91f7c6bc9cb92be5982de4719c1c8af91ccff",
"Used" : 493926221800,
"Free" : 416318701568,
"Safemode" : "",
"NonDfsUsedSpace" : 65950005272,
"PercentUsed" : 50.59709,
"BlockPoolUsedSpace" : 493926221800,
"PercentBlockPoolUsed" : 50.59709,
"PercentRemaining" : 42.647087,
"CacheCapacity" : 0,
"CacheUsed" : 0,
"TotalBlocks" : 1896,
"TotalFiles" : 220,
"NumberOfMissingBlocks" : 0,
"NumberOfMissingBlocksWithReplicationFactorOne" : 0,
"LiveNodes" : "{\"slave2:50010\":{\"infoAddr\":\"10.xxx.xxx.14:50075\",\"infoSecureAddr\":\"10.xxx.xxx.14:0\",\"xferaddr\":\"10.xxx.xxx.14:50010\",\"lastContact\":0,\"usedSpace\":246963131380,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":33589472268,\"capacity\":488097464320,\"numBlocks\":1896,\"version\":\"2.7.3\",\"used\":246963131380,\"remaining\":207544860672,\"blockScheduled\":0,\"blockPoolUsed\":246963131380,\"blockPoolUsedPercent\":50.597095,\"volfails\":0},\"slave1:50010\":{\"infoAddr\":\"10.xxx.xxx.12:50075\",\"infoSecureAddr\":\"10.xxx.xxx.12:0\",\"xferaddr\":\"10.xxx.xxx.12:50010\",\"lastContact\":0,\"usedSpace\":246963090420,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":32360533004,\"capacity\":488097464320,\"numBlocks\":1896,\"version\":\"2.7.3\",\"used\":246963090420,\"remaining\":208773840896,\"blockScheduled\":0,\"blockPoolUsed\":246963090420,\"blockPoolUsedPercent\":50.597088,\"volfails\":0}}",
"DeadNodes" : "{}",
"DecomNodes" : "{}",
"BlockPoolId" : "BP-484138619-10.xxx.xxx.10-1482316479215",
"NameDirStatuses" : "{\"active\":{\"/srv/hadoop/tmp/dfs/name\":\"IMAGE_AND_EDITS\"},\"failed\":{}}",
"NodeUsage" : "{\"nodeUsage\":{\"min\":\"50.60%\",\"median\":\"50.60%\",\"max\":\"50.60%\",\"stdDev\":\"0.00%\"}}",
"NameJournalStatus" : "[{\"manager\":\"FileJournalManager(root=/srv/hadoop/tmp/dfs/name)\",\"stream\":\"EditLogFileOutputStream(/srv/hadoop/tmp/dfs/name/current/edits_inprogress_0000000000012297436)\",\"disabled\":\"false\",\"required\":\"false\"}]",
"JournalTransactionInfo" : "{\"MostRecentCheckpointTxId\":\"12297435\",\"LastAppliedOrWrittenTxId\":\"12299635\"}",
"NNStarted" : "Mon Apr 16 07:05:30 GST 2018",
"CompileInfo" : "2016-08-18T01:41Z by root from branch-2.7.3",
"CorruptFiles" : "[]",
"DistinctVersionCount" : 1,
"DistinctVersions" : [ {
  "key" : "2.7.3",
  "value" : 2
} ],
"SoftwareVersion" : "2.7.3",
"RollingUpgradeStatus" : null,
"Threads" : 34
  } ]
}

Could someone help me with the same.
I tried with the split filter configuration but could only reach till here -

Suggestions to get the complete array parsed.

[magnusbaeck (Magnus Bäck)]

You have a number of fields that contain JSON strings. Parse them with a json filter.

[karnamonkster (Karan)]

Hi Magnus,

Here is the filter configuration

filter {
json {
    source => "message"
    target => "message"
  }
  split {
field => "[beans]"
target => "beans"
   }
}

How do i apply the filter for the fields that are not parsed?

[magnusbaeck (Magnus Bäck)]

You just use the json filter in the same way as above but choose different field names.

[karnamonkster (Karan)]

Perfect!! Thanks so much Magnus, guess i hesitated to go one step further