# Kibana fetch data and visualization
**URL:** https://discuss.elastic.co/t/kibana-fetch-data-and-visualization/137515
**Category:** Kibana
**Created:** [June 26, 2018, 9:38pm UTC](https://discuss.elastic.co/t/kibana-fetch-data-and-visualization/137515 "2018-06-26T21:38:21Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@krisdinesh](https://discuss.elastic.co/u/krisdinesh)
#### Post date: [June 26, 2018, 9:38pm UTC](https://discuss.elastic.co/t/kibana-fetch-data-and-visualization/137515/1 "2018-06-26T21:38:21Z")
Hi, I want to build the data table/charts in Kibana for the data 'Tests run: 1, Failures: 0, Errors: 0, Skipped: 0' from jenkins console log which is being shipped into elasticsearch.
Please help me in this regard. Thanks
---
### Author:  [@Joe\_Fleming](https://discuss.elastic.co/u/Joe_Fleming)
#### Post date: [June 27, 2018, 9:38pm UTC](https://discuss.elastic.co/t/kibana-fetch-data-and-visualization/137515/2 "2018-06-27T21:38:01Z")
What do the documents in Elasticsearch look like, and how would you like to visualize this data?
---
### Author:  [@krisdinesh](https://discuss.elastic.co/u/krisdinesh)
#### Post date: [June 27, 2018, 10:03pm UTC](https://discuss.elastic.co/t/kibana-fetch-data-and-visualization/137515/3 "2018-06-27T22:03:03Z")
I would like to visualize in any graphical format and data looks like in elasticsearch as attached and below data needs to the fetched from the message as as shown in the attachment and construct as a graph(any type of graph).
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
---
### Author:  [@Joe\_Fleming](https://discuss.elastic.co/u/Joe_Fleming)
#### Post date: [June 27, 2018, 11:01pm UTC](https://discuss.elastic.co/t/kibana-fetch-data-and-visualization/137515/4 "2018-06-27T23:01:29Z")
So you're just indexing the string "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0"?
That's not very usable. It's not searchable or aggregatable. You'll probably want to document it as something like:
```json
{
total: 1,
failures: 0,
errors: 0,
skipped: 0,
passed: 1,
timestamp: "2018-06-27T16:31:04Z"
}
```
`timestamp` is useful here if you'd like to track the values of each of the properties over time. `passed` is useful because then you can query passed vs total directly, instead of having to do a calculation for that information later. You may also want to add information about the `project` or other metadata, assuming Jenkins is being used for multiple projects, so you can filter the result by a given project.
So every time Jenkins runs, you create a new document with the result data.
Once your documents look like that, it's pretty easy to, for example, create a graph of the tests run, failures, etc. over time. You'd use a date histogram for the x-axis, then add a y-axis value for the sum of each of the properties. This will show the number of passed, failed, and skipped tests over time.
---
### Author:  [@krisdinesh](https://discuss.elastic.co/u/krisdinesh)
#### Post date: [June 27, 2018, 11:23pm UTC](https://discuss.elastic.co/t/kibana-fetch-data-and-visualization/137515/5 "2018-06-27T23:23:32Z")
Hi,
Please let me know how to filter it in the filbeat?
I am QA person, not having much technical knowledge but willing to learn. If you help initially how to filter it out, I can develop from that point. I appreciate all your support in this regard.
Please find the attached the .yml file that I am using in my filbeat log shipper
Thanks
Dinesh
---
### Author:  [@krisdinesh](https://discuss.elastic.co/u/krisdinesh)
#### Post date: [June 27, 2018, 11:28pm UTC](https://discuss.elastic.co/t/kibana-fetch-data-and-visualization/137515/6 "2018-06-27T23:28:14Z")
Also here is my sample log and I would like to get only highlighted part to be exported to elastic search from file beats and I am not using logstash.
Running TestSuite, Jun 26, 2018 7:27:11 AM org.openqa.selenium.remote.ProtocolHandshake createSession, INFO: Detected dialect: OSS, search icon, search page, , 1 Scenarios ([32m1 passed[0m), 5 Steps ([32m5 passed[0m), 0m12.297s, , , 0 Scenarios, 0 Steps, 0m0.000s, , Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.601 sec, , **Results :, , Tests run: 1, Failures: 0, Errors: 0, Skipped: 0** , , [INFO] , [INFO] --- maven-postman-plugin:0.1.6:send-mail (send a mail) @ bdd-cucumber ---, [INFO] add attachment D:\sites[qa-reports.vmldev.com](http://qa-reports.vmldev.com)\TestProject1\latest\report.html, [INFO] , [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ bdd-cucumber ---, [INFO] Building jar: D:\apps\jenkins\workspace\TestProject1\target\bdd-cucumber-1.0-SNAPSHOT.jar, [INFO] , [INFO] --- maven-cucumber-reporting:3.8.0:generate (execution) @ bdd-cucumber ---, [WARNING] D:\apps\jenkins\workspace\TestProject1\target\cucumber-reports\json-reports does not exist., [INFO] , [INFO] --- maven-install-plugin:2.4:install (default-install) @ bdd-cucumber ---, [INFO] Installing D:\apps\jenkins\workspace\TestProject1\target\bdd-cucumber-1.0-SNAPSHOT.jar to C:\Users\jenkinsreadwrite-ad.m2\repository\com\ashenlive\bdd-cucumber\1.0-SNAPSHOT\bdd-cucumber-1.0-SNAPSHOT.jar, [INFO] Installing D:\apps\jenkins\workspace\TestProject1\pom.xml to C:\Users\jenkinsreadwrite-ad.m2\repository\com\ashenlive\bdd-cucumber\1.0-SNAPSHOT\bdd-cucumber-1.0-SNAPSHOT.pom, [INFO] ------------------------------------------------------------------------, [INFO] BUILD SUCCESS, [INFO] ------------------------------------------------------------------------, [INFO] Total time: 01:16 min, [INFO] Finished at: 2018-06-26T07:27:30-05:00, [INFO] Final Memory: 37M/320M, [INFO]
---
### Author:  [@krisdinesh](https://discuss.elastic.co/u/krisdinesh)
#### Post date: [July 3, 2018, 4:13pm UTC](https://discuss.elastic.co/t/kibana-fetch-data-and-visualization/137515/7 "2018-07-03T16:13:27Z")
Below is my filebeat pattern to fetch the log lines starts with 'Tests' and exclude the lines that contains 'elapsed'. Please let me whats wrong .yml config. The expectation is pick up the last line from the sample log.
Filebeat.yml
- type: log
enabled: true
paths:
- C:\Users\dkrishnaswam\Desktop\log.FILE
include\_lines: ['^Tests']
exclude\_lines: ['._(?:elapsed)._']
multiline.pattern: '^[A-Z]{1}[a-z]{2} {1,2}[0-9]{1,2}, [0-9]{4} {1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}'
multiline.negate: true
multiline.match: after
**Sample Logs:**
Tests run: 11, Failures: 4, Errors: 1, Skipped: 0, Time elapsed: 106.378 sec \<\<\< FAILURE! - in com.fordswx.serenity.HomePage
Tests run: 11, Failures: 4, Errors: 1, Skipped: 0, Time elapsed: 106.378 sec \<\<\< FAILURE! - in com.fordswx.serenity.HomePage
Tests run: 11, Failures: 4, Errors: 1, Skipped: 0, Time elapsed: 106.378 sec \<\<\< FAILURE! - in com.fordswx.serenity.HomePage
Tests run: 357, Failures: 34, Errors: 48, Skipped: 35
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 31, 2018, 4:13pm UTC](https://discuss.elastic.co/t/kibana-fetch-data-and-visualization/137515/8 "2018-07-31T16:13:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.