How do i index these logs with ELK and get the analytics

[kiran_kumar_sukumar]

Hi,

I have a text document which needs to be indexed and analysed in elasticsearch

The format is as follows
Test case id, Start time, Buld number
Test case id , test flow, test action, step1, step1 status
Test case id , test flow, test action, step2, step2 status
Test case id , test flow, test action, step3, step3 status
Test case id , test flow, test action, step4, step4 status
Test case id , test flow, test action, step5, step5 status
Test case id , test flow, test action, step6, step6 status
Test case id , test flow, test action, step7, step7 status

A test case can have multiple steps and each step has its own status(passed or failed).

A test case is passed if all the steps have passed or failed if one of the steps have failed. As you can see there is no status on the test case line itself(first line) but the status of the test case should be calculated based on the status of the steps

By indexing these logs i want to find out if the test case has passed or failed. I dont want to look at the test step level but at the test case level

Thanks well ahead.

Regards
Kumar

[shaunak (Shaunak Kashyap)]

Hi @kiran_kumar_sukumar,

Here's the general idea of what I'm thinking might work:

1. Create a Logstash pipeline with the appropriate input filter (file, stdin, etc. — depends on how you want to "feed" each of the text documents into Logstash).

2. In this Logstash pipeline, use the grok filter to parse each line into fields.

3. Depending on the presence of the step number field, you will know if you are looking at a test case line or a test step line. You can use conditionals in Logstash to make this decision and take either sub-step 1 or 2 below:

   1. If you are looking at a test case line, create a document in Elasticsearch (using the Elasticsearch output plugin) for the test case. At this time, create a "status" field in this document and set its value to "passed".

   2. If you are looking at a test step line AND if the status of this step is "failed", update the associated test case document in Elasticsearch, setting the value of its "status" field to "failed". For this you will need to first retrieve the Elasticsearch document for the associated test case using the Elasticsearch filter plugin, then do the update, and finally index it back into Elasticsearch using the Elasticsearch output plugin.

4. Once you have the test case documents in Elasticsearch, we can visualize them by status in Kibana.

Hope that helps,

Shaunak

[kiran_kumar_sukumar]

Thanks Shaunak