Visualizing a JSON array as datatable

[Dominik_Deren]

Hey,

I'm wondering if it is possible to convert a JSON array into a Datatable visualization and it on a dashboard. I'm building a monitoring dashboard for an application, and as one of the elements of the dashboard, I wanted to include current state of objects that the application operates on. Kinda like a peek inside the internal database of the service, to have a quick glance at what is happening right now.

To give more tangible example, this is a JSON array I'd like to display in my dashboard:

    [
      {
        "ownerId": "16a2bceb-e15e-4947-9130-09ac34234c5f",
        "updatedAt": "2018-04-19T00:49:43.595Z",
        "addedAt": "2018-04-19T00:19:50.429Z",
        "status": "IN_PROGRESS",
        "externalReference": "tims_Execution1",
        "id": "71c6a3b8-310e-4dd2-8e84-50ef23f2809b",
        "duration": 510846058
      },
      {
        "ownerId": "16a2bceb-e15e-4947-9130-09ac34234c5f",
        "updatedAt": "2018-04-19T15:52:20.195Z",
        "addedAt": "2018-04-19T15:22:26.185Z",
        "status": "IN_PROGRESS",
        "externalReference": "tims_Execution1",
        "id": "6ca80cca-9329-4758-9c1f-42a9d2905db7",
        "duration": 456690302
      }
    ]

I would like to display on my dashboard a data table, that would contain the id, externalReference & duration information from the above JSON array, and order the results by duration.

I was thinking about two approaches to this, either getting the data directly from an HTTP endpoint, but it seems that Kibana is not able to gather data from external sources other than ElasticSearch, so I don't think I would be able to query my own endpoint?

Another approach, which I went with, is to push that object to my application logs, and in Kibana get it from logs. But in this case, I'm not sure how to convert this object to something that can be usable by the datatable.

It seems as if datatable is only able to display data as aggregations from ES search results, but not to parse a specific JSON object into table. Is there another approach here that might help me do what I'm trying to do?

Thanks in advance for your help.

[Joe_Fleming]

You're correct that Kibana can't work with data that's not in Elasticsearch already.

Another approach, which I went with, is to push that object to my application logs, and in Kibana get it from logs. But in this case, I'm not sure how to convert this object to something that can be usable by the datatable.

I'm assuming that your "application logs" live in Elasticsearch. If I'm right, then you're well on your way. You may want to use a different index for these documents than you do for your logs, or you could add another field to identify the document type. Don't use _type though, that's going away, and in 6.x, you can only have one _type value per index. You can add you own type field though, or whatever you'd like to call it. Just some kind of field to filter on.

Once you have a way to filter to just those documents, getting them in a table vis is pretty simple. If you've already set up in the index in Kibana, you'll want to refresh it (Management > Index Patterns > Refresh) so that all the fields are updated (assuming you added new ones, if you didn't, then you can skip that). If you've gone the new index way, add that index pattern to Kibana from that page.

Once Kibana knows about the index and/or fields, then it's pretty simple. However, you don't want to use a visualization, since visualizations are all driven by aggregations, so instead of working with raw document values, it works with a kind of roll-up of that data, and that's not what you want.

Instead, use Discover, which deals only with raw documents, and create a saved search. From discover, use the field list on the side to add the want to see, and you'll see the results as you add them. You can also move fields around if you'd like. When you're happy with what you see, save the search, and then you can add that saved search to the dashboard, which will embed the table you just created.

[Dominik_Deren]

Hey Joe,

Thanks for a quick response. Yes I mean logs stored in elastic search. I was thinking about the approach with displaying a table of data using the search from Discover tab as well. But this one is a bit problematic.

At the moment, I have this object in my log entry:

The object I would like to visualize as a table is the activeExecutions JSON array that is available in a single log line. I'm wondering if there is any way to display information from that JSON array as a table in my dashboard. I wasn't able to find a way to display a JSON array as table.

I think I would have to split the rows in the activeExecutions array into separate log entries, each one containing a single active execution, and then I could display properties of a single log entry as columns in the table. That's fine, but in such a case, how do I filter out duplicate results?

What I mean by that, is I would have that list of log entries pushed to my log recurrently, for example every minute. Would I be able to filter out my table to only take entries that were produced in the last minute? To make sure that I only display current data, and not some stale information?

Thanks in advance for your help!

[Joe_Fleming]

Oh, I see. Sorry for misunderstanding.

I'm wondering if there is any way to display information from that JSON array as a table in my dashboard. I wasn't able to find a way to display a JSON array as table.

No, there is not. Kibana only deals with field values from Elasticsearch. You'll want to index the activeExecutions value in order to visualize it. Each one of those values should be its own document.

You may need to "denormalize" the document a little, basically duplicating the other fields you have on that document into the individual "active execution" documents. This is quite normal in nosql databases. You want to deal with flat documents, and there are no relationships, so denormalization is how you do that.

[Dominik_Deren]

Hey Joe,

Thanks for your help, I managed to get it to work with the Discover view, where my app pushes all "active executions" as separate log lines every minute, and in my Discover view I created a filter for log messages with specific "message", where timestamp is between now-1m and now.