Best approach to visualize nested objects in Kibana

[topel (Tobias O.)]

I know, because I searched for it here on discuss, that using nested data/objects in Kibana seems to be a recurring topic but unfortunately I still have not found the answer I was looking for.

I am querying a Jenkins-API and get answers that look like the following code, that I would like to insert into Elasticsearch:

{
        "_class": "org.jenkinsci.plugins.workflow.job.WorkflowJob",
        "actions": [
            {},
            {},
            {},
            {},
            {
                "_class": "hudson.plugins.jobConfigHistory.JobConfigHistoryProjectAction"
            },
            {},
            {},
            {},
            {},
            {},
            {
                "_class": "com.cloudbees.plugins.credentials.ViewCredentialsAction"
            }
        ],
        "description": "",
        "displayName": "XYZ",
        "displayNameOrNull": null,
        "fullDisplayName": "XYZ",
        "fullName": "XYZ",
        "name": "XYZ",
        "url": "https://url.com/job/XYZ/",
        "buildable": true,
        "builds": [
            {
                "_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
                "number": 53,
                "url": "https://url.com/job/XYZ/53/"
            },
            {
                "_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
                "number": 52,
                "url": "https://https://url.com/job/XYZ/52/"
            }, /*
            {
               !!! left out for better readability !!!
            }, */
            {
                "_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
                "number": 1,
                "url": "https://url.com/job/XYZ/1/"
            }
        ],
        "color": "red",
        "firstBuild": {
            "_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
            "number": 1,
            "url": "https://https://url.com/job/XYZ//1/"
        },
       "healthReport": [
            {
                "description": "Build stability: 1 out of the last 5 builds failed.",
                "iconClassName": "icon-health-60to79",
                "iconUrl": "health-60to79.png",
                "score": 80
            },
            {
                "description": "Test Result: 0 tests failing out of a total of 1,693 tests.",
                "iconClassName": "icon-health-80plus",
                "iconUrl": "health-80plus.png",
                "score": 100
            }
        ],
        "inQueue": false,
        "keepDependencies": false,
        "lastBuild": {
            "_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
            "number": 53,
            "url": "https://https://url.com/job/XYZ/53/"
        },
        "lastCompletedBuild": {
            "_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
            "number": 53,
            "url": "https://https://url.com/job/XYZ/53/"
        },
        "lastFailedBuild": {
            "_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
            "number": 53,
            "url": "https://https://url.com/job/XYZ/53/"
        },
        "lastStableBuild": {
            "_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
            "number": 51,
            "url": "https:/https://url.com/job/XYZ/51/"
        },
        "lastSuccessfulBuild": {
            "_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
            "number": 51,
            "url": "https://https://url.com/job/XYZ/51/"
        },
        "lastUnstableBuild": null,
        "lastUnsuccessfulBuild": {
            "_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
            "number": 53,
            "url": "https://https://url.com/job/XYZ/53/"
        },
        "nextBuildNumber": 54,
        "property": [
            {
                "_class": "hudson.plugins.buildblocker.BuildBlockerProperty"
            },
            {
                "_class": "org.datadog.jenkins.plugins.datadog.DatadogJobProperty"
            },
            {
                "_class": "hudson.model.ParametersDefinitionProperty",
                "parameterDefinitions": [
                    {
                        "_class": "hudson.model.StringParameterDefinition",
                        "defaultParameterValue": {
                            "_class": "hudson.model.StringParameterValue",
                            "name": "SCM_REVISION",
                            "value": "default"
                        },
                        "description": "",
                        "name": "SCM_REVISION",
                        "type": "StringParameterDefinition"
                    }
                ]
            },
            {
                "_class": "org.jenkinsci.plugins.workflow.job.properties.DisableConcurrentBuildsJobProperty"
            },
            {
                "_class": "com.synopsys.arc.jenkins.plugins.ownership.jobs.JobOwnerJobProperty"
            },
            {
                "_class": "com.sonyericsson.rebuild.RebuildSettings"
            },
            {
                "_class": "com.synopsys.arc.jenkinsci.plugins.jobrestrictions.jobs.JobRestrictionProperty"
            },
            {
                "_class": "org.jenkinsci.plugins.workflow.job.properties.PipelineTriggersJobProperty"
            }
        ],
        "queueItem": null,
        "concurrentBuild": false,
        "resumeBlocked": false
    }

Now what is my best approach to be able to go over all my Jenkins jobs and visualize the healthReport scores for build stability in a histogram for example? I have the ES query to get the buckets but since it uses the nested query syntax the field isn't available in the "visualize" menu.

I have almost full control over the data, I can modify it freely before sending it to logstash and then ES.

What shall I do? Flatten and denormalize my data to that extent that I don't have nested objects?
How do I keep the semantic connection alive then? Include the information that links the two in both, so for example concerning the builds I would give every build the name of its pipeline and the pipeline just has an array of build names then (that you could also drop because it doesn't say anything but that's another story) and so on? Or is there any value in using parent-child relationships?

Thanks for the reply

Tobias

[lukas (Lukas Olson)]

Sounds like you want to visualize "builds". (i.e., build stability, perhaps the most recent fail or success, etc.)

I would make a "builds" index. Each build document would have the info currently in "builds" (_class, number, url) and also include any other identifying info you need (displayName, completed, success, color, etc.)

Then you could create a pie chart that shows successes vs. failures, you could create a metric that shows the most recent color for a certain build, etc.

[topel (Tobias O.)]

Thank you Lukas for your fast reply,

However I still would like some further explanation regarding your answer. I will have a builds index nonetheless since, I only get "jobs" and "builds" from the API. So builds was probably a bad example to choose by me. However I want to keep the connection between the pipeline/job and the builds alive. Hence I guess I am fine with denormalizing and flattening my data and thus including the job info in the build-index entries and vice versa but can you touch on the topic of the healthReport[] again? Is it advisable to create a new index there as well or should I just keep it "embedded" if you want to say so and Split the Array in two different objects like healthReport.build.xyz-data and healthReport.test.xyz-data ? The array can only have 0-2 entries. Are we now talking about a matter of preference or is there still a right and a wrong way? Please don't hesitate to state the obvious I am new to Kibana and Elasticsearch so there might be some misconceptions from my side.

Thanks a bunch.

Tobias