Creating data table. Table of done jobs

[optimistic (Joe)]

I need to create table in Kibana.
I have such logic of my app:
User-refresh event creates, I have id of this event.
Then all services should make some changes and send me info that they done a job.

So, I have such data (logs) in Elasticsearch:

user-refresh id=21
user-refresh id=21, service1
user-refresh id=21, service2
user-refresh id=21, service3
user-refresh id=22
user-refresh id=22, service1
user-refresh id=22, service2
user-refresh id=23
user-refresh id=23, service1
user-refresh id=23, service3

I need to know which service did a job, and which is not.

If I have massage from a serviceN, it means service did a job - v
if I don't have any massage from service with user-refresh id - service did not do a job - x

How to create such a table?

Thank you very much.

UPD. I can also change the output of logs, my main goal is to make such table.

[lukas (Lukas Olson)]

You won't likely be able to accomplish this using the "data table" feature in Kibana, but you might be able to using Canvas.

Could you actually provide examples of the JSON of the docs inside ES? Thanks!

[optimistic (Joe)]

@lukas thanks for answer.

Yes, now I see that such a table don't possible to build with Data table.

This is a test JSON from user-refresh event:

{
"_index": "logstash-2019.06.18-000001",
"_type": "_doc",
"_id": "DECuc2sBt0eN0q1ZwlYQ",
"_version": 1,
"_score": null,
"_source": {
"code": "user-refresh-test",
"schedule": null,
"@timestamp": "2019-06-20T06:59:21.128Z",
"@version": "1",
"creator": "notification-api",
"ip": null,
"id": 80,
"type": "rabbitmq"
},
"fields": {
"@timestamp": [
"2019-06-20T06:59:21.128Z"
]
},
"sort": [
1561013961128
]
}

we just need event "code": "user-refresh-test" and it's id "id": 80

And this reply from service1 that job is done:

{
"_index": "logstash-2019.06.18-000001",
"_type": "_doc",
"_id": "c0Cvc2sBt0eN0q1ZUFYC",
"_version": 1,
"_score": null,
"_source": {
"code": "user-service1-test",
"schedule": null,
"@timestamp": "2019-06-20T06:59:57.465Z",
"@version": "1",
"creator": "notification-api",
"ip": null,
"id": 80,
"service1": true,
"type": "rabbitmq"
},
"fields": {
"@timestamp": [
"2019-06-20T06:59:57.465Z"
]
},
"sort": [
1561013997465
]
}

Here we need approve that job is done "service1": true and id of a job "id": 80

Is it possible to create such table in Visual builder?
Or I need to take a deeper look into Canvas?

[optimistic (Joe)]

I changed logic of logs aggregation.
I configured my Logstash pipeline to change/update Elasticsearch document if its already have a documents with given id.

output {
elasticsearch {
hosts => "elasticsearch:9200"
user => elastic
password => xx
index => "logstash-%{+YYYY.MM.dd}"
document_id => "%{id}" #this field is main
doc_as_upsert => true
action => "update"

So my first service send event:

'user-refresh-test', id: 115

Than other service send:

'user-refresh-test', id: 115, serviceN1: 1

and it's update a first elasticsearch document.
So now I have everything in one document and it is easy to build any visualization.