A way to visualize changes in a document from week to week

[Mauro_Julian_Fernand (Mauro Julián Fernández)]

I'm currently developing a script which ingests documents to a Elasticsearch index which contains data from a network discovery

    {
      "_index": "device_discovery",
      "_type": "_doc",
      "_id": "YAySBnwBWvky_R2ePp1X",
      "_score": 1,
      "_source": {
        "datetime": "2021-08-24 04:13:50",
        "run_id": "20210921",
        "ip_addr": "10.0.0.1",
        "dns": [
          "name.domain.com"
        ],
        "red_addr": 10.0.0.0",
        "red_addr_cidr": "10.0.0.0/8",
        "type": "device_discovery",
        "source_server": "10.1.0.1",
        "segmento_name": "Testing"
      },
      "fields": {
        "datetime": [
          "2021-08-24T04:13:50.000Z"
        ],
        "run_id": [
          "20210921"
        ],
        "source_server": [
          "10.1.0.1"
        ],
        "red_addr": [
          "10.0.0.0"
        ],
        "dns": [
          "name.domain.com"
        ],
        "segmento_name": [
          "Testing"
        ],
        "type": [
          "device_discovery"
        ],
        "ip_addr": [
          "10.0.01"
        ],
        "red_addr_cidr": [
          "10.0.0.0/8"
        ]
      }
    }

That document basically says "This IP is alive", the script will run each week discovering devices and loading them in the index. What I'm trying to achieve is to visualize somehow the difference each week, lets say, today I run the script and the IP 10.0.0.2 doesn't answer my ping and is not loaded in the index, but others will and that timestamp will have records (I plan to run it each Friday). Next week I run the script and said IP answers, so it's loaded in the index, what I want to see is that difference "this new IP wasn't here last week but now it is". That's my primary objective, but other use cases include seeing if their DNS changes from one scan to another.

I'm open to make any changes to the index pattern, include or take out fields to make it work.

[Marco_Liberati]

Hi @Mauro_Julian_Fernand ,

welcome to the Kibana community.

From your description I understand we're talking about some visualization that includes some timing dimension (week over week) and a boolean state (reachable/not reachable) for each IP in your list.
Considering these types of dimension I would start to look at something like a Heatmap:

Another alternative can be a stacked bar chart, quicker to get the overall trend but slower to spot specific drops:

Would they be useful?

[Mauro_Julian_Fernand (Mauro Julián Fernández)]

Hi @Marco_Liberati, thanks for your answer.

Yes, the main problem with your solution (which was one option on how to build the index) is that I currently don't ingest not reachable IPs, just the ones that are reachable, because I have a couple of nasty /22 and /18 networks that will contain a ton of "trash data" so I wanted to figure a solution with just that, but I see that it will be almost impossible to do it with just reachable IPs.

I'll make the changes to how I ingest the data and try your solution, thanks for the insight.