Inventory data being aggregated with periodic data

[mtuska (Montana Tuska)]

Hi,
I am working on creating an application that will do a nightly inventory of CPE equipment, meanwhile also do every minute and every 5 minute statistics. I then would like to be able to display this data via Kibana dashboards. I imagine this being 2 indices, for example, cpe-inventory and cpe-statistics.
For an example of a manipulation I'd like to make for display, each of the CPE's are attached to certain physical nodes. So I would like to then do pivots to have an index of data displaying aggregations of statistics for that complete node.
This isn't a use-case I've really run into yet with Elastic, so not sure of the best way to handle this. My current version of ELK is 7.17(at some point I'll have some more time to work on upgrading), and the application is being built in Python 3.10. It will be exporting data directly to the Elasticsearch.

Here an my document template for inventory

cmts_cm_doc = {
    "@timestamp": datetime.utcnow().isoformat()+'Z',
    "entity": "Cmts_CableModem",
    "cmts": {
        "id": "",
    },
    "cm": {
        "id": "",
        "mac": "",
        "mgmt_ipv4": "",
        "mgmt_ipv6": "",
    },
    "fiber_node": {
        "id": "",
        "name": "",
        "alias": "",
        "up_sg_id": "",
    },
}

Then for the statistics, it'd be similar to

cmts_cm_doc = {
    "@timestamp": "",
    "entity": "Cmts_CableModem",
    "cmts": {
        "id": "",
    },
    "interfaces": [
        {
            "id": "",
            "snr": 0,
        },
    ],
    "cm": {
        "id": "",
        "mac": "",
        "mgmt_ipv4": "",
        "mgmt_ipv6": "",
    },
    "status": "",
    "unerroreds": 0,
    "correcteds": 0,
    "uncorrectables": 0,
}

Though I do plan for there to be more values in both document in the future, this is just my current setup. To compare the two docs, it'd be cmts.id is equal, and cm.id is equal.

[bhavyarm]

Hi,

Either lens or TSVB should be able to solve this problem for you.
Kibana Lens | Elastic or TSVB | Kibana Guide [7.17] | Elastic

I recommend upgrading to our latest version though(8.3.1 is our latest). Lens is becoming super powerful and you can use two layers - two data views and we have annotations etc

Thanks,
Bhavya