Visualize 2 lines - 2 nodes with 1 parameter per time

[avovana (Vladimir)]

Good morning!

After intensive search It seems to me that it is impossible. Anyway, maybe someone have an idea to implement it(with Kibana, or, maybe Zabbix + prometheus)...

The case is I have a system consists of many nodes. And have an utility that saves state of a system every 5 minutes in txt:

So I can see for every node some diagnostic parameters for every timestamp:
-quantity of errors
-quantity of input messages
-quantity of output messages
...

The goal is to make a graph with some of the nodes visualising one of the parameter, for example
-quantity of input messages per 5 minutes

I did it with Excel well:

1. Parsed this source file into csv
2. Opened with Excel
3. Made graph

How do it with Kibana?

For 1 node it worked perfect:

1. I parsed this source file into json:
   
   
2. Downloaded with filebeat
3. Opened "Visualise" in Kibana
4. Choose this "data.json file & Node = node1"
5. Choose "quantity of input messages"
6. Choose strange "Max"(for example)
7. Got great graph

But if I going to add second Node = node2 I have a trouble.
I think I have tried all of the variants with interface but the desired result was not achieved...

Is it really possible to do it?
Or I should parse source data file(with all the nodes and its data) into separate nodes files(so 1 file has data of only 1 node)? And play with Visualise interface?

The goal is to have a simple graph with lines(one parameter depending of time) of few nodes.

Could you help me, please?

[mattkime (Matthew Kime)]

@avovana

What you're attempting to do is completely reasonable and I suspect you're very close to making it happen. Can you share the query you're attempting that is resulting in disappointment?

As best I understand, all you need to do is take your existing successful query and have it select an additional node value. Do you know what field type 'Node' is?

Thanks,
Matt

[avovana (Vladimir)]

Node is text.
Some details about format:

{"Node": "feed_asts_ccy_securities", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "17852", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "223144"}
{"Node": "feed_forts_instruments", "Status": "offline", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "17848", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "feed_micex_eqt_clr", "Status": "offline", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "21808", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "supervisor", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10613", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "transformer_asts_ccy", "Status": "online", "MsgIn": "1854", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "6705", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "1723"}
{"Node": "transformer_forts", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "6698", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "transformer_micex", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "6702", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "transport_asts_ccy", "Status": "online", "MsgIn": "1723", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10656", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "1723"}
{"Node": "transport_asts_xml2", "Status": "online", "MsgIn": "109718", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10657", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "10718"}
{"Node": "transport_forts", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10654", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "transport_micex", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10655", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "transport_micex_eqt", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10658", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}

The desired result for 1 node:

But what should I do next to have new line of the another Node here?

[mattkime (Matthew Kime)]

You need to aggregate on Node. If you load the ecommerce sample data you'll see a couple examples in the related dashboard.

[avovana (Vladimir)]

Matthew, thank you for a respond!

Unfortunately I didn't succeed...
Kibana interface doesn't give me such opportunity in the "Aggregation" field.

I deleted and Node : "feed_forts_instruments in a "Filters" field to have access to all of the Nodes in this log file.
But even in this case "Node" is absent in the "Aggregation":

[avovana (Vladimir)]

So I downloaded sample data with a clue

To get this sample data, visit your Kibana homepage and click on “Load a data set and a Kibana dashboard.” There, you will see the sample data provided for eCommerce orders and web logs.

Now I see that "Split Series" will lead me to success.
In the sample data series splitted by text field "category"(with strange ".keyword" suffix):

In my case "Node" has "text" format also(now I see why did you ask me to specify the format).
But I can not choose it:

[mattkime (Matthew Kime)]

The terms aggregation works best with a keyword field type - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html

Note this text in the linked document - In order to use it with text you will need to enable fielddata.

[avovana (Vladimir)]

Thank you, Matthew!
I see that I have 3 options:

1. Set fielddata=true on [your_field_name] in order to load fielddata in memory by uninverting the inverted index.

2. You can enable fielddata on an existing text field using the PUT mapping API as follows: PUT my-index-000001/_mapping { "properties": { "my_field": { "type": "text", "fielddata": true } } }

3. Adding "keyword" type

1.After some search I decided to change filebeat.yml and added fielddata=true to the Node field:

Reloaded filebeat.
Reloaded indices.
But don't see the result - "Node" was absent in a "Split Series".

Removed this addition.

2.After it I tried Dev Tools:
Firstly, tried to add "keyword".

Secondly, made PUT with fielddata=true.
Understood that one parameter was missed. Fixed. Got success:

Reload filebeat. Downloaded data. Again "Node" was absent:

Updated indices, check "fielddata". It appeared:

But still "Node" was absent.
Added data. Again "Node" was absent.

Thirdly, tried again with adding a "keyword":

Updated indices. Made sure that result OK:

Added data. Again "Node" was absent.

I suppouse the success is so close but can not get it...

[mattkime (Matthew Kime)]

Can you share the mapping for your index? It certainly looks to me like you did everything right, two different ways.

[avovana (Vladimir)]

Thank you, Matthew!

Unfortunately, *.txt is not allowed to attache...
Can you give me your email via personal message if it possible.

[mattkime (Matthew Kime)]

Use https://gist.github.com/

[avovana (Vladimir)]

Unfortunately I have not access to it...

I got a successful upload here:
http://files.rsdn.org/133345/mapping.txt

A pretty complicated file. There is no "Node" mapping.

Another variant was made:
In a fields.yml added filedata:

• name: Node
  type: text
  fielddata: true

So after reloading filebeat and elasticsearch can see new mapping in Kibana interface:

"Node": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}

But again failure when try to select in a "Split Series":

In a meantime I can select another field with the same mapping!

[mattkime (Matthew Kime)]

Hello @avovana,

Sorry for the delay.

The mapping certainly looks correct.

Is it possible for you to supply me with a few sample rows of data in csv format so I can attempt to reproduce? Obviously strip out any sensitive info.

Thanks,
Matt

[avovana (Vladimir)]

@mattkime, thank you for a respond!
It inspires me to go further)

Node,Status,MsgIn,VER,Warns,Pid,System,Errs,MsgOut
feed_asts_ccy_securities, online, 0, 39e1e, 0, 17852, MC_IFT_SBRF, 0, 223144
feed_micex, online, 0, 39e1e, 0, 17855, MC_IFT_SBRF, 2, 223155

=========================

I tried one more step with a field.
Added field "Node1" with the type "keyword" in filebeat.yml
I understood that I don't need "text" type with all of this difficulties to make it aggregatable. "keyword" is aggregatable by default.

For the first reload filebeat I saw "Node1" in kibana "Index managment", but now don't:

But "Node1" is in the index pattern(and it is "Aggregatable"):

(as a Node.keyword, though)

Added:
echo '{"Node": "descript", "Node1": "feed_micex", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Warns": "0", "Pid": "17854", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "223155"}' > bla.json

Still can not choose:
Node.keword
Node1

P.S.
By the way I noticed strange mapping. For example, "MsgIn" is "long" in a filebeat.yml, but in the "Index managment" in the "Mapping" is "text" as you can see on the picture above.

Investigated that these 6 mapping conflicts related to all of my fields that are "long".

P.P.S
After so many attempts now I want to start from scratch. All that I need is:

But as I read I can not reload "Index pattern"(filebeat-* in my case) because there are many indexed docs already

I have an idea!
Maybe make new?
So, I will need to change my filebeat name:
filebeat-new-7.8.1-2020.12.08-000118 -> "smth1".
Then modify filebeat.yml as I described above.
And learn how to make new "Index pattern" for "smth1" and fill it with mapping.

[avovana (Vladimir)]

Update.
Deleted many previous indexes.

As I understood new one is created when new filebeat is started.
Because I started and kill it a lot there were many indexes.

Made filebeat as I wanted:

Started filebeat, new index was created with good mapping:

Conflicts dissapeared, types are good, Node is aggregatable:

Added to log 3 rows:

Checked, the 1, 3 are good. The 2nd row was added badly somehow:

And you can guess the result in a "Split Series"...

The same sad story...
I can not choose here the perfect suitable aggregatable "Node"

[avovana (Vladimir)]

Maybe this update will help.
Learned how to retrieve information:

1.GET filebeat-7.8.1-2020.12.09-000001/_doc/UoYdSHYBgrhx-iAPPOPp

 {
  "_index" : "filebeat-7.8.1-2020.12.09-000001",
  "_type" : "_doc",
  "_id" : "UoYdSHYBgrhx-iAPPOPp",
  "_version" : 1,
  "_seq_no" : 2,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "@timestamp" : "2020-12-09T15:27:23.364Z",
    "Node" : "some_name",
    "MsgOut" : "223155",
    "Pid" : "17854",
    "Errs" : "0",
    "input" : {
      "type" : "log"
    },
    "System" : "MC_IFT_SBRF",
    "MsgIn" : "0",
    "Warns" : "0",
    "message" : "some_name",
    "VER" : "39e1e",
    "log" : {
      "offset" : 0,
      "file" : {
        "path" : "/home/mcift_pao/test_logs/bla.json"
      }
    },
    "agent" : {
      "hostname" : "algodev04.msk.trd.ru",
      "ephemeral_id" : "2f3c8f63-f2ba-41b6-80f0-1b15c12ab0b2",
      "id" : "4ef6de1d-c457-4179-88d0-d2a25770e29c",
      "name" : "algodev04.msk.trd.ru",
      "type" : "filebeat",
      "version" : "7.8.1"
    },
    "ecs" : {
      "version" : "1.5.0"
    },
    "Status" : "online",
    "host" : {
      "os" : {
        "codename" : "Maipo",
        "platform" : "rhel",
        "version" : "7.8 (Maipo)",
        "family" : "redhat",
        "name" : "Red Hat Enterprise Linux Server",
        "kernel" : "3.10.0-1127.19.1.el7.x86_64"
      },
      "id" : "0979853c5fc248359ccb2b6e9f67bd3e",
      "containerized" : false,
      "ip" : [
        "10.20.3.141",
        "fe80::250:56ff:fe9c:647c",
        "10.160.20.33",
        "fe80::250:56ff:fe9c:743c",
        "172.18.0.1",
        "172.20.0.1",
        "fe80::42:ceff:fe91:75c6",
        "fe80::fcef:f2ff:fe26:16a0"
      ],
      "name" : "algodev04.msk.trd.ru",
      "mac" : [
        "00:50:56:9c:64:7c",
        "00:50:56:9c:74:3c",
        "00:50:56:9c:44:c8",
        "02:42:a4:00:12:15",
        "02:42:ce:91:75:c6",
        "fe:ef:f2:26:16:a0"
      ],
      "hostname" : "algodev04.msk.trd.ru",
      "architecture" : "x86_64"
    }
  }
}

2. Learned how to make put:

POST /filebeat-7.8.1-2020.12.09-000001/_doc/1
{
  "Node": "feed_asts_ccy_securities",
  "Status": "online",
  "MsgIn": "0",
  "VER": "39e1e",
  "Warns": "0",
  "Pid": "17852",
  "System": "MC_IFT_SBRF",
  "Errs": "0",
  "MsgOut": "223144"
}

So I have:
_id = QIbmR3YBgrhx-iAPQ8_t
_id = 9YYJSHYBgrhx-iAPPtt8
_id = UoYdSHYBgrhx-iAPPOPp
_id = 1

3. Tried to understand whether "Node" is aggregatable thanks to this request:

GET filebeat-7.8.1-2020.12.09-000001/_search
{
  "aggs": {
    "filebeat-7.8.1-2020.12.09-000001": {
       "terms" : { "field": "Node"}
    }
  }
}

Got:

Summary

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-7.8.1-2020.12.09-000001",
        "_type" : "_doc",
        "_id" : "QIbmR3YBgrhx-iAPQ8_t",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2020-12-09T14:27:17.658Z",
          "log" : {
            "offset" : 0,
            "file" : {
              "path" : "/home/mcift_pao/test_logs/bla.json"
            }
          },
          "System" : "MC_IFT_SBRF",
          "Status" : "online",
          "MsgOut" : "223155",
          "Errs" : "0",
          "host" : {
            "ip" : [
              "10.20.3.141",
              "fe80::250:56ff:fe9c:647c",
              "10.160.20.33",
              "fe80::250:56ff:fe9c:743c",
              "172.18.0.1",
              "172.20.0.1",
              "fe80::42:ceff:fe91:75c6",
              "fe80::fcef:f2ff:fe26:16a0"
            ],
            "mac" : [
              "00:50:56:9c:64:7c",
              "00:50:56:9c:74:3c",
              "00:50:56:9c:44:c8",
              "02:42:a4:00:12:15",
              "02:42:ce:91:75:c6",
              "fe:ef:f2:26:16:a0"
            ],
            "name" : "algodev04.msk.trd.ru",
            "hostname" : "algodev04.msk.trd.ru",
            "architecture" : "x86_64",
            "os" : {
              "codename" : "Maipo",
              "platform" : "rhel",
              "version" : "7.8 (Maipo)",
              "family" : "redhat",
              "name" : "Red Hat Enterprise Linux Server",
              "kernel" : "3.10.0-1127.19.1.el7.x86_64"
            },
            "id" : "0979853c5fc248359ccb2b6e9f67bd3e",
            "containerized" : false
          },
          "message" : "feed_micex",
          "input" : {
            "type" : "log"
          },
          "Node" : "feed_micex",
          "Pid" : "17854",
          "MsgIn" : "0",
          "Warns" : "0",
          "VER" : "39e1e",
          "ecs" : {
            "version" : "1.5.0"
          },
          "agent" : {
            "ephemeral_id" : "2f3c8f63-f2ba-41b6-80f0-1b15c12ab0b2",
            "id" : "4ef6de1d-c457-4179-88d0-d2a25770e29c",
            "name" : "algodev04.msk.trd.ru",
            "type" : "filebeat",
            "version" : "7.8.1",
            "hostname" : "algodev04.msk.trd.ru"
          }
        }
      },
      {
        "_index" : "filebeat-7.8.1-2020.12.09-000001",
        "_type" : "_doc",
        "_id" : "9YYJSHYBgrhx-iAPPtt8",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2020-12-09T15:05:33.055Z",
          "host" : {
            "id" : "0979853c5fc248359ccb2b6e9f67bd3e",
            "containerized" : false,
            "ip" : [
              "10.20.3.141",
              "fe80::250:56ff:fe9c:647c",
              "10.160.20.33",
              "fe80::250:56ff:fe9c:743c",
              "172.18.0.1",
              "172.20.0.1",
              "fe80::42:ceff:fe91:75c6",
              "fe80::fcef:f2ff:fe26:16a0"
            ],
            "mac" : [
              "00:50:56:9c:64:7c",
              "00:50:56:9c:74:3c",
              "00:50:56:9c:44:c8",
              "02:42:a4:00:12:15",
              "02:42:ce:91:75:c6",
              "fe:ef:f2:26:16:a0"
            ],
            "name" : "algodev04.msk.trd.ru",
            "hostname" : "algodev04.msk.trd.ru",
            "architecture" : "x86_64",
            "os" : {
              "name" : "Red Hat Enterprise Linux Server",
              "kernel" : "3.10.0-1127.19.1.el7.x86_64",
              "codename" : "Maipo",
              "platform" : "rhel",
              "version" : "7.8 (Maipo)",
              "family" : "redhat"
            }
          },
          "agent" : {
            "name" : "algodev04.msk.trd.ru",
            "type" : "filebeat",
            "version" : "7.8.1",
            "hostname" : "algodev04.msk.trd.ru",
            "ephemeral_id" : "2f3c8f63-f2ba-41b6-80f0-1b15c12ab0b2",
            "id" : "4ef6de1d-c457-4179-88d0-d2a25770e29c"
          },
          "ecs" : {
            "version" : "1.5.0"
          },
          "log" : {
            "offset" : 161,
            "file" : {
              "path" : "/home/mcift_pao/test_logs/bla.json"
            }
          },
          "json" : { },
          "message" : "",
          "input" : {
            "type" : "log"
          }
        }
      },
      {
        "_index" : "filebeat-7.8.1-2020.12.09-000001",
        "_type" : "_doc",
        "_id" : "UoYdSHYBgrhx-iAPPOPp",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2020-12-09T15:27:23.364Z",
          "Node" : "some_name",
          "MsgOut" : "223155",
          "Pid" : "17854",
          "Errs" : "0",
          "input" : {
            "type" : "log"
          },
          "System" : "MC_IFT_SBRF",
          "MsgIn" : "0",
          "Warns" : "0",
          "message" : "some_name",
          "VER" : "39e1e",
          "log" : {
            "offset" : 0,
            "file" : {
              "path" : "/home/mcift_pao/test_logs/bla.json"
            }
          },
          "agent" : {
            "hostname" : "algodev04.msk.trd.ru",
            "ephemeral_id" : "2f3c8f63-f2ba-41b6-80f0-1b15c12ab0b2",
            "id" : "4ef6de1d-c457-4179-88d0-d2a25770e29c",
            "name" : "algodev04.msk.trd.ru",
            "type" : "filebeat",
            "version" : "7.8.1"
          },
          "ecs" : {
            "version" : "1.5.0"
          },
          "Status" : "online",
          "host" : {
            "os" : {
              "codename" : "Maipo",
              "platform" : "rhel",
              "version" : "7.8 (Maipo)",
              "family" : "redhat",
              "name" : "Red Hat Enterprise Linux Server",
              "kernel" : "3.10.0-1127.19.1.el7.x86_64"
            },
            "id" : "0979853c5fc248359ccb2b6e9f67bd3e",
            "containerized" : false,
            "ip" : [
              "10.20.3.141",
              "fe80::250:56ff:fe9c:647c",
              "10.160.20.33",
              "fe80::250:56ff:fe9c:743c",
              "172.18.0.1",
              "172.20.0.1",
              "fe80::42:ceff:fe91:75c6",
              "fe80::fcef:f2ff:fe26:16a0"
            ],
            "name" : "algodev04.msk.trd.ru",
            "mac" : [
              "00:50:56:9c:64:7c",
              "00:50:56:9c:74:3c",
              "00:50:56:9c:44:c8",
              "02:42:a4:00:12:15",
              "02:42:ce:91:75:c6",
              "fe:ef:f2:26:16:a0"
            ],
            "hostname" : "algodev04.msk.trd.ru",
            "architecture" : "x86_64"
          }
        }
      },
      {
        "_index" : "filebeat-7.8.1-2020.12.09-000001",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "Node" : "feed_asts_ccy_securities",
          "Status" : "online",
          "MsgIn" : "0",
          "VER" : "39e1e",
          "Warns" : "0",
          "Pid" : "17852",
          "System" : "MC_IFT_SBRF",
          "Errs" : "0",
          "MsgOut" : "223144"
        }
      }
    ]
  },
  "aggregations" : {
    "filebeat-7.8.1-2020.12.09-000001" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "feed_asts_ccy_securities",
          "doc_count" : 1
        },
        {
          "key" : "feed_micex",
          "doc_count" : 1
        },
        {
          "key" : "some_name",
          "doc_count" : 1
        }
      ]
    }
  }
}

If the example is correct, it is!

The question is why I can not use this aggregatable "Node" in "Terms" in "Split Series" in "Visualization"?

[avovana (Vladimir)]

Good news from Graphana:

I can select it. And it works !!!

So, the problem is in the Kibana.