Visualize 2 lines - 2 nodes with 1 parameter per time

Good morning!

After intensive search It seems to me that it is impossible. Anyway, maybe someone have an idea to implement it(with Kibana, or, maybe Zabbix + prometheus)...

The case is I have a system consists of many nodes. And have an utility that saves state of a system every 5 minutes in txt:


So I can see for every node some diagnostic parameters for every timestamp:
-quantity of errors
-quantity of input messages
-quantity of output messages
...

The goal is to make a graph with some of the nodes visualising one of the parameter, for example
-quantity of input messages per 5 minutes

I did it with Excel well:

  1. Parsed this source file into csv
  2. Opened with Excel
  3. Made graph

How do it with Kibana?

For 1 node it worked perfect:

  1. I parsed this source file into json:
  2. Downloaded with filebeat
  3. Opened "Visualise" in Kibana
  4. Choose this "data.json file & Node = node1"
  5. Choose "quantity of input messages"
  6. Choose strange "Max"(for example)
  7. Got great graph

But if I going to add second Node = node2 I have a trouble.
I think I have tried all of the variants with interface but the desired result was not achieved...

Is it really possible to do it?
Or I should parse source data file(with all the nodes and its data) into separate nodes files(so 1 file has data of only 1 node)? And play with Visualise interface?

The goal is to have a simple graph with lines(one parameter depending of time) of few nodes.

Could you help me, please?

@avovana

What you're attempting to do is completely reasonable and I suspect you're very close to making it happen. Can you share the query you're attempting that is resulting in disappointment?

As best I understand, all you need to do is take your existing successful query and have it select an additional node value. Do you know what field type 'Node' is?

Thanks,
Matt

Node is text.
Some details about format:

{"Node": "feed_asts_ccy_securities", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "17852", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "223144"}
{"Node": "feed_forts_instruments", "Status": "offline", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "17848", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "feed_micex_eqt_clr", "Status": "offline", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "21808", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "supervisor", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10613", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "transformer_asts_ccy", "Status": "online", "MsgIn": "1854", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "6705", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "1723"}
{"Node": "transformer_forts", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "6698", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "transformer_micex", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "6702", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "transport_asts_ccy", "Status": "online", "MsgIn": "1723", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10656", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "1723"}
{"Node": "transport_asts_xml2", "Status": "online", "MsgIn": "109718", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10657", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "10718"}
{"Node": "transport_forts", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10654", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "transport_micex", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10655", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}
{"Node": "transport_micex_eqt", "Status": "online", "MsgIn": "0", "VER": "39e1e", "Timestamp": "2020-11-24T09:02:01.458902Z", "Warns": "0", "Pid": "10658", "System": "MC_IFT_SBRF", "Errs": "0", "MsgOut": "0"}

The desired result for 1 node:

But what should I do next to have new line of the another Node here?

You need to aggregate on Node. If you load the ecommerce sample data you'll see a couple examples in the related dashboard.

Matthew, thank you for a respond!

Unfortunately I didn't succeed...
Kibana interface doesn't give me such opportunity in the "Aggregation" field.

I deleted and Node : "feed_forts_instruments in a "Filters" field to have access to all of the Nodes in this log file.
But even in this case "Node" is absent in the "Aggregation":

So I downloaded sample data with a clue

To get this sample data, visit your Kibana homepage and click on “Load a data set and a Kibana dashboard.” There, you will see the sample data provided for eCommerce orders and web logs.

Now I see that "Split Series" will lead me to success.
In the sample data series splitted by text field "category"(with strange ".keyword" suffix):


In my case "Node" has "text" format also(now I see why did you ask me to specify the format).
But I can not choose it:

The terms aggregation works best with a keyword field type - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html

Note this text in the linked document - In order to use it with text you will need to enable fielddata.

Thank you, Matthew!
I see that I have 3 options:

  1. Set fielddata=true on [your_field_name] in order to load fielddata in memory by uninverting the inverted index.

  2. You can enable fielddata on an existing text field using the PUT mapping API as follows: PUT my-index-000001/_mapping { "properties": { "my_field": { "type": "text", "fielddata": true } } }

  3. Adding "keyword" type

1.After some search I decided to change filebeat.yml and added fielddata=true to the Node field:
image

Reloaded filebeat.
Reloaded indices.
But don't see the result - "Node" was absent in a "Split Series".


Removed this addition.

2.After it I tried Dev Tools:
Firstly, tried to add "keyword".

Secondly, made PUT with fielddata=true.
Understood that one parameter was missed. Fixed. Got success:

Reload filebeat. Downloaded data. Again "Node" was absent:
image
Updated indices, check "fielddata". It appeared:
image
But still "Node" was absent.
Added data. Again "Node" was absent.

Thirdly, tried again with adding a "keyword":

Updated indices. Made sure that result OK:
image

Added data. Again "Node" was absent.

I suppouse the success is so close but can not get it...

Can you share the mapping for your index? It certainly looks to me like you did everything right, two different ways.

Thank you, Matthew!

Unfortunately, *.txt is not allowed to attache...
Can you give me your email via personal message if it possible.

Use https://gist.github.com/

Unfortunately I have not access to it...

I got a successful upload here:
http://files.rsdn.org/133345/mapping.txt

A pretty complicated file. There is no "Node" mapping.

Another variant was made:
In a fields.yml added filedata:

  • name: Node
    type: text
    fielddata: true

So after reloading filebeat and elasticsearch can see new mapping in Kibana interface:

"Node": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}


But again failure when try to select in a "Split Series":
image
In a meantime I can select another field with the same mapping!
image