Kibana fetch data and visualization

Elastic Stack Kibana
1

Hi, I want to build the data table/charts in Kibana for the data 'Tests run: 1, Failures: 0, Errors: 0, Skipped: 0' from jenkins console log which is being shipped into elasticsearch.

Please help me in this regard. Thanks

2

What do the documents in Elasticsearch look like, and how would you like to visualize this data?

3

I would like to visualize in any graphical format and data looks like in elasticsearch as attached and below data needs to the fetched from the message as as shown in the attachment and construct as a graph(any type of graph).

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

4

So you're just indexing the string "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0"?

That's not very usable. It's not searchable or aggregatable. You'll probably want to document it as something like:

{
  total: 1,
  failures: 0,
  errors: 0,
  skipped: 0,
  passed: 1,
  timestamp: "2018-06-27T16:31:04Z"
}

timestamp is useful here if you'd like to track the values of each of the properties over time. passed is useful because then you can query passed vs total directly, instead of having to do a calculation for that information later. You may also want to add information about the project or other metadata, assuming Jenkins is being used for multiple projects, so you can filter the result by a given project.

So every time Jenkins runs, you create a new document with the result data.

Once your documents look like that, it's pretty easy to, for example, create a graph of the tests run, failures, etc. over time. You'd use a date histogram for the x-axis, then add a y-axis value for the sum of each of the properties. This will show the number of passed, failed, and skipped tests over time.

5

Hi,

Please let me know how to filter it in the filbeat?

I am QA person, not having much technical knowledge but willing to learn. If you help initially how to filter it out, I can develop from that point. I appreciate all your support in this regard.

Please find the attached the .yml file that I am using in my filbeat log shipper

Thanks

Dinesh

6

Also here is my sample log and I would like to get only highlighted part to be exported to elastic search from file beats and I am not using logstash.

Running TestSuite, Jun 26, 2018 7:27:11 AM org.openqa.selenium.remote.ProtocolHandshake createSession, INFO: Detected dialect: OSS, search icon, search page, , 1 Scenarios ([32m1 passed[0m), 5 Steps ([32m5 passed0m), 0m12.297s, , , 0 Scenarios, 0 Steps, 0m0.000s, , Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.601 sec, , Results :, , Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, , [INFO] , [INFO] --- maven-postman-plugin:0.1.6:send-mail (send a mail) @ bdd-cucumber ---, [INFO] add attachment D:\sites[qa-reports.vmldev.com\TestProject1\latest\report.html, [INFO] , [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ bdd-cucumber ---, [INFO] Building jar: D:\apps\jenkins\workspace\TestProject1\target\bdd-cucumber-1.0-SNAPSHOT.jar, [INFO] , [INFO] --- maven-cucumber-reporting:3.8.0:generate (execution) @ bdd-cucumber ---, [WARNING] D:\apps\jenkins\workspace\TestProject1\target\cucumber-reports\json-reports does not exist., [INFO] , [INFO] --- maven-install-plugin:2.4:install (default-install) @ bdd-cucumber ---, [INFO] Installing D:\apps\jenkins\workspace\TestProject1\target\bdd-cucumber-1.0-SNAPSHOT.jar to C:\Users\jenkinsreadwrite-ad.m2\repository\com\ashenlive\bdd-cucumber\1.0-SNAPSHOT\bdd-cucumber-1.0-SNAPSHOT.jar, [INFO] Installing D:\apps\jenkins\workspace\TestProject1\pom.xml to C:\Users\jenkinsreadwrite-ad.m2\repository\com\ashenlive\bdd-cucumber\1.0-SNAPSHOT\bdd-cucumber-1.0-SNAPSHOT.pom, [INFO] ------------------------------------------------------------------------, [INFO] BUILD SUCCESS, [INFO] ------------------------------------------------------------------------, [INFO] Total time: 01:16 min, [INFO] Finished at: 2018-06-26T07:27:30-05:00, [INFO] Final Memory: 37M/320M, [INFO]

7

Below is my filebeat pattern to fetch the log lines starts with 'Tests' and exclude the lines that contains 'elapsed'. Please let me whats wrong .yml config. The expectation is pick up the last line from the sample log.

Filebeat.yml

  • type: log
    enabled: true
    paths:
    • C:\Users\dkrishnaswam\Desktop\log.FILE
      include_lines: ['^Tests']
      exclude_lines: ['.(?:elapsed).']
      multiline.pattern: '^[A-Z]{1}[a-z]{2} {1,2}[0-9]{1,2}, [0-9]{4} {1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}'
      multiline.negate: true
      multiline.match: after

Sample Logs:

Tests run: 11, Failures: 4, Errors: 1, Skipped: 0, Time elapsed: 106.378 sec <<< FAILURE! - in com.fordswx.serenity.HomePage
Tests run: 11, Failures: 4, Errors: 1, Skipped: 0, Time elapsed: 106.378 sec <<< FAILURE! - in com.fordswx.serenity.HomePage
Tests run: 11, Failures: 4, Errors: 1, Skipped: 0, Time elapsed: 106.378 sec <<< FAILURE! - in com.fordswx.serenity.HomePage
Tests run: 357, Failures: 34, Errors: 48, Skipped: 35

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.