Connecting documents from different kibana indexes

Elastic Stack Kibana
1

Hello Elastic-Community,
I am wondering if there is a way to connect two datasets from two different indexes with each other to create a visualization.

For example:

From Index_1 I receive general informations of a device, that looks like this:

{
"DeviceName": "machine_1",
"ID": 001,
}

And from Index_2 I get temperature-values from this device, but i don't have the "DeviceName"-field:

{
"ID": 001,
"temperature": "20", 
"timestamp": "2021-04-15T07:34:33.079Z",
}

Now i want to create a visualization, where i can filter or search for the "DeviceName" such as "machine_1" and get the current temperature and a chart with the temperatur over time.

Thanks for any kind of help!

2

@timroes can we have more inputs here? Thank you.

3

Hi,

this is unfortunately not really able that way. There is a join field type, but that's unfortunately not supported in Kibana. In general when working with search engines like Elasticsearch (in contrast to relational databases) it's recommended to denormalize data (instead of normalization what you might be used from a relational database), i.e. in this case including the device name into the second document as well. Given how Elasticsearch (and most search engines) are build technically, this is the preferred version for gaining the relevant speed and still be able to execute the desired queries you want.

Cheers,
Tim

4

I agree to @timroes

There are also inbuilt possibilities to do so.
You can use transforms to do the merge or you use the enrich processor on index time.
I would use the second alternative in your case.

5

@Felix_Roessel
Can you please provide some further information to your suggestion. Where can I find these functionalities and how do I use them correctly?

6

You can find it in Kibana -> Management - > Ingest Pipelines
There you configure a new ingest Pipeline for your data using the enrich processor.

Finally you add this new pipeline as default pipeline to the Meeting index template of your main index.

7

@Felix_Roessel
thank you for the information.

I was able to enrich a document with data from another index... perfect!!

At the time i create test-documents, which should be enriched, with the command:

PUT myindex/_doc/mydocument?pipeline=my_enrich_pipeline

Here it is easy to define that the document should use the specific pipeline.

But how/where do I define, that this pipeline should also be used for documents, which are send from elasticsearch?

8

A pipeline is always triggered on index time. To run elasticsearch documents through it you can use the reindex API.

9

I added my pipeline as "default_pipeline" in de index settings.

My pipeline enriches data with the match field: "ID"

But now i get the following error, when i receive data without the field "ID":

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "field [ID] not present as part of path [ID]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "field [ID] not present as part of path [ID]"
  },
  "status" : 400
}

Can I somehow define that a certain pipeline should only be used, when the field "ID" exists?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.