Different fields in different documents

[Anton91]

Hey,

I have the following question and I want to visualize two fields with each other that are not connected to each other. Both are stored in different documents but in the same index.
For example
I have document
A: B:
Values A: Values B:
C, D, E E,F,G

I want to display the values C and F now. Because the values are not stored in a document I cannot visualize them. I have already thought about Scripted Field but there is surely a more elegant solution.

Thanks a lot in advance

[Luca_Belluccini]

Hello @Anton91

Elasticsearch cannot perform joins and with scripted fields you cannot access other documents.

The only way to impose relationships between documents are join type and nested type.

Still, using them and showing them on Kibana has limitations.

If there's a common field or a common value, you might perform a query to filter them and show up all the documents having those common properties.

Can you share a more practical example? Can the possibility of changing the document model be considered?

[Anton91]

Hello @Luca_Belluccini,

thank you for the quick response.

Actually this is a mapping. I have a doucment :AppId: Ndjs that matches the field AppName: Node.js

Here is an example Document :

{
  "_index": "test",
  "_type": "test",
  "_id": "test."
  "_version": 1,
  "_score": zero,
  "_source": {
    "host": "WIN-1234"
    "time": "1592857645226"
    "Sourcetype": "AppNameIdMapping"
    "AppName": "Node.js"
    "AppId": "Ndjs"
  },

Now the other doucment from which I need the data:

{
  "_index": "test",
  "_type": "test",
  "_id": "Zl273XIB9FpepAxds7Fk"
  "_version": 1,
  "_score": zero,
  "_source": {
    "host": "WIN-1234"
    "time": "1592857964609"
    "Sourcetype": "Process:ProcessDetail"
    "ProcName": "node.exe"
    "ProcCPUTimeMs": "31",
    "ProcCPUPercent": "0",
    "ProcIOPSRead": "0",
    "ProcIOPSWrite": "0",
    "ProcIOReadCount": "0"
    "ProcIOWriteCount": "0"
    "ProcIOReadMB": "0",
    "ProcIOWriteMB": "0",
    "ProcWorkingSetMB": "213.5"
    "ProcNetKBPS": "0",
    "ProcUser": "sys."
    "AppId": "Ndjs",
    "AppVersion": "10.15.2"
    "ProcID": "4188"
  }

Here you can also see that the field AppID is stored there but unfortunately not Appname. I want the app name to be displayed with e.g. the summed ProcCPUTimeMs.

Here an example for the visualization:

Theoretically it is exactly that the solution I need. I have created a scripted field for it and I get the right solution. As soon as I merge 3-4 visualizations into one dashboard I get an error : [esaggs] > Request to Elasticsearch failed: {"error":{}}. There must be a more elegant solution for this ?

[Luca_Belluccini]

From the visualization example I cannot see anywhere the information coming from AppId.
What is the scripted field you're using?
It is possible to create an hardcoded mapping between AppId and AppName using a scripted field in Painless, but it must be maintained manually.

Enriching the data

The correct solution to your problem would be using an Enrich processor at indexing time.
The AppNameIdMapping documents should be stored in a separate lookup index and it would be possible to enrich the document Process:ProcessDetail with the data from the AppNameIdMapping documents using an Ingest Pipeline with the Enrich policy.

PUT /_enrich/policy/appname_lookup
{
    "match": {
        "indices": "appnameidmappingindex",
        "match_field": "AppId",
        "enrich_fields": ["AppName"]
    }
}

POST /_enrich/policy/appname_lookup/_execute

PUT /_ingest/pipeline/appname_lookup_pipeline
{
  "description" : "Enriching",
  "processors" : [
    {
      "enrich" : {
        "policy_name": "appname_lookup",
        "field" : "AppId",
        "target_field": "lookup",
        "max_matches": "1"
      }
    }
  ]
}

If you index the documents Process:ProcessDetail using this ingest pipeline, you will find a field lookup.AppName.

---

"As soon as I merge 3-4 visualizations into one dashboard I get an error"...

Would it be possible to share the HAR file of the network requests (see procedure here) while you have the error [esaggs] > Request to Elasticsearch failed: {"error":{}}?

---

Still, I feel like you're trying to adapt UberAgent data to the Elastic Stack, but we also offer Metricbeat, Packetbeat & APM which might cover your needs.