Calculate MTBF/MTTF Mean time between/to failure for a visualization

[topel (Tobias O.)]

Hi there,

I am struggling to find the way or rather a way to visualize the MTBF of my Jenkins Pipelines/Jobs.

What I have is the following:

• A job index, where one document holds the information about what builds belong to this job (as an Array of urls)
• This also contains information about what the last successful build and the last unsuccessful build are (urls)
• A build index that has the detailed information for each build with a starting timestamp, duration and of course a result field as well as a reference to what pipeline it belogs to (url)

This is what I have but it is still fully adjustable if you tell me, that there is a better approach.

I tried using timelion but I am stuck at the point where I have to decide where my time difference (the tbf part of mtbf) comes from.

As a side-note: The mtbf for me only makes sense for pipeline so I have to filter the corresponding builds at some point to only get the ones I really need.

I am aware I have only given minimal information at the point, I'll gladly provide any additional information if needed.

@Bargs answered a similar question back in 2017 but I think it is not fully applicable here.

Any advice?

Cheers and thanks for the help,
Tobias

[chrisronline (Chris Roberson)]

Hi @topel,

A couple things:

1. Can you provide me with some sample data and the mappings for your indices?
2. Can you provide a link to the related issue back in 2017?

Thanks,
Chris

[topel (Tobias O.)]

Hi @chrisronline,

I just double checked the issue was this one and I seem to have recollected it wrong - it only vaguely touches the topic.

Sample job data:

ES job index entry

 {
    "_index": "test-jobs-old",
    "_type": "doc",
    "_id": "yL_j1WMBkhQDVC7mqY67",
    "_score": 1,
    "_source": {
      "displayName": "GroovyTester_Pipeline",
      "@timestamp": "2018-06-06T16:17:25.649Z",
      "description": "",
      "lastSuccessfulBuild": {
        "url": "https://bla.com/job/GroovyTester_Pipeline/23/",
        "class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
        "number": 23
      },
      "firstBuild": {
        "url": "https://bla.com/job/GroovyTester_Pipeline/1/",
        "class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
        "number": 1
      },
      "concurrentBuild": true,
      "resumeBlocked": false,
      "keepDependencies": false,
      "beat": {
        "name": "dashboard",
        "hostname": "dashboard",
        "version": "6.2.4"
      },
      "healthReport": {
        "buildStability": {
          "remark": "4 out of the last 5 builds failed.",
          "score": 20
        }
      },
      "fullName": "GroovyTester_Pipeline",
      "nextBuildNumber": 24,
      "inQueue": false,
      "parameterDefinitions": {},
      "url": "https://bla.com/job/GroovyTester_Pipeline/",
      "lastStableBuild": {
        "url": "https://bla.com/job/GroovyTester_Pipeline/23/",
        "class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
        "number": 23
      },
      "fullDisplayName": "GroovyTester_Pipeline",
      "lastUnsuccessfulBuild": {
        "url": "https://bla.com/job/GroovyTester_Pipeline/22/",
        "class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
        "number": 22
      },
      "name": "GroovyTester_Pipeline",
      "lastCompletedBuild": {
        "url": "https://bla.com/job/GroovyTester_Pipeline/23/",
        "class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
        "number": 23
      },
      "host": "dashboard",
      "buildable": true,
      "offset": 1394532,
      "prospector": {
        "type": "log"
      },
      "color": "blue",
      "actions": [
        "hudson.plugins.jobConfigHistory.JobConfigHistoryProjectAction",
        "com.cloudbees.plugins.credentials.ViewCredentialsAction",
        "hudson.plugins.buildblocker.BuildBlockerProperty",
        "com.sonyericsson.rebuild.RebuildSettings",
        "com.synopsys.arc.jenkinsci.plugins.jobrestrictions.jobs.JobRestrictionProperty",
        "org.jenkinsci.plugins.workflow.job.properties.PipelineTriggersJobProperty"
      ],
      "lastFailedBuild": {
        "url": "https://bla.com/job/GroovyTester_Pipeline/22/",
        "class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
        "number": 22
      },
      "tags": [
        "json",
        "jenkins",
        "jobs",
        "init",
        "beats_input_raw_event"
      ],
      "builds": [
        "https://bla.com/job/GroovyTester_Pipeline/23/",
        "https://bla.com/job/GroovyTester_Pipeline/22/",
        "https://bla.com/job/GroovyTester_Pipeline/21/",
        "https://bla.com/job/GroovyTester_Pipeline/20/",
        "https://bla.com/job/GroovyTester_Pipeline/19/",
        "https://bla.com/job/GroovyTester_Pipeline/18/",
        "https://bla.com/job/GroovyTester_Pipeline/17/",
        "https://bla.com/job/GroovyTester_Pipeline/16/",
        "https://bla.com/job/GroovyTester_Pipeline/15/",
        "https://bla.com/job/GroovyTester_Pipeline/14/",
        "https://bla.com/job/GroovyTester_Pipeline/13/",
        "https://bla.com/job/GroovyTester_Pipeline/12/",
        "https://bla.com/job/GroovyTester_Pipeline/11/",
        "https://bla.com/job/GroovyTester_Pipeline/10/",
        "https://bla.com/job/GroovyTester_Pipeline/9/",
        "https://bla.com/job/GroovyTester_Pipeline/8/",
        "https://bla.com/job/GroovyTester_Pipeline/7/",
        "https://bla.com/job/GroovyTester_Pipeline/6/",
        "https://bla.com/job/GroovyTester_Pipeline/5/",
        "https://bla.com/job/GroovyTester_Pipeline/4/",
        "https://bla.com/job/GroovyTester_Pipeline/3/",
        "https://bla.com/job/GroovyTester_Pipeline/2/",
        "https://bla.com/job/GroovyTester_Pipeline/1/"
      ],
      "lastBuild": {
        "url": "https://bla.com/job/GroovyTester_Pipeline/23/",
        "class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
        "number": 23
      },
      "source": "/srv/jenkins_data/init_jobs_log_2018-06-06_163617.json",
      "@version": "1",
      "class": "org.jenkinsci.plugins.workflow.job.WorkflowJob",
      "property": []
    }
  }

[topel (Tobias O.)]

Sample build data:

ES build index entry

{
    "_index": "test-builds-old",
    "_type": "doc",
    "_id": "mL_j1WMBkhQDVC7m4sHz",
    "_score": 1,
    "_source": {
      "displayName": "#31-pipeline-name",
      "@timestamp": "2018-03-16T13:12:32.775Z", 
      "changeSetMercurial": [],
      "result": "SUCCESS",             < ----------------------- 
      "keepLog": false,
      "building": false,
      "nextBuild": {
        "number": 32,
        "url": "https:/bla.com/job/package-promotion-pipeline/32/"
      },
      "causeAction": {
        "rebuildTrigger": {
          "upstreamBuild": 30,
          "upstreamUrl": "job/package-promotion-pipeline/",
          "shortDescription": "Rebuilds build #30",
          "upstreamProject": "package-promotion-pipeline"
        },
        "userTrigger": {
          "userId": "johndoe",
          "shortDescription": "Started by user Doe, John",
          "userName": "Doe, John"
        }
      },
      "beat": {
        "name": "dashboard",
        "hostname": "dashboard",
        "version": "6.2.4"
      },
      "number": 31,
      "id": "31",
      "previousBuild": {
        "number": 30,
        "url": "https://bla.com/job/package-promotion-pipeline/30/"
      },
      "pipelineUrl": "https://bla.com/job/package-promotion-pipeline/",
      "fullDisplayName": "package-promotion-pipeline #31-pipeline-name",
      "url": "https://bla.com/job/package-promotion-pipeline/31/",
      "queueId": 68074,
      "culprits": [],
      "mercurialTags": [
        ...
      ],
      "buildData": [
        ...
      ],
      "duration": 12110,
      "host": "dashboard",
      "estimatedDuration": 16512,
      "offset": 68405728,
      "prospector": {
        "type": "log"
      },
      "actions": [
        "hudson.plugins.git.GitTagAction",
        "org.jenkinsci.plugins.workflow.cps.EnvActionImpl",
        "org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"
      ],
      "changeSetGit": [],
      "tags": [
        "json",
        "jenkins",
        "builds",
        "init",
        "beats_input_raw_event"
      ],
      "@version": "1",
      "source": "/srv/jenkins_data/init_builds_log_2018-06-06_163617.json",
      "class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
      "timestamp": 1521202340665 
    }
  }

I marked the build result with an arrow. What I now want is to filter for one pipeline and then do maths on the time it took between the different results. So for example what the average time is between one build with result failure and the next one.

Hope this clarifies things and you are now able to help me.

Cheers,
Tobias

[topel (Tobias O.)]

@chrisronline bump