Kibana Dashboard: Create line graph from a list of floats

Hi,

I have a list of floats that I am passing to ES/Kibana through the Python API.

I am able to see the document in Discover, and the list is visible there as expected. Now, I want to create a graph that iterates over this list and creates a dot for each data point (x-axis), between the min and max values (y-axis).

However, I do not see my object (list) when trying to create a graph using the Lens feature. Am I not able to create a graph based off a list?
Looks like this object is of type "dense_vector" as far as ES is concerned, but it is not clear to me what I can do with this data type.

What is the best way to solve this?

Thanks!

Thanks for reaching out, @Pasjonsfrukt. I have a few follow-up questions:

  • Do you have an example of the dataset and the Python script you are using that you can send over?
  • What version of Elastic are you using?

Thanks,

Jessica

Hi! Here is the script (simplified) that I am using:

from elasticsearch import Elasticsearch, helpers

es = Elasticsearch(
    <cloud_url>,
    api_key=<api_key>
)

seconds = [28.22, 26.92, 49.01, 26.1, 26.12, 25.22, 26.27]

es.index(
 index='hast-poc',
 document={
  'Seconds': seconds,
 })

Essentially I want to plot 'seconds' over the x-axis, with lets say the max y-value as 49.01 and min 0 (I could of course pass individual variables for the y-axis if necessary).

I tried passing a simple string as the value to a key in the 'document', and that shows up as a valid value to use in my dashboard. Do I need to pass my list values as individual key-value pairs?

Elasticsearch Python version:

>>> elasticsearch.__version__
(8, 17, 0)
>>> 

Thanks for following up, @Pasjonsfrukt. I think the issue is that you are using dense vectors, commonly used to store high-dimensional vectors, like embeddings for machine learning models, rather than individual data points like floats that you would graph. Using the float data type might be better here. You can reindex or create a new index.

To create a new index with the proper mappings, it would be something like this:

new_index_name = "hast-poc-v2"

# Define new mapping
mapping = {
    "mappings": {
        "properties": {
            "Seconds": {
                "type": "float"  # Use float for numerical data
            }
        }
    }
}

# Create the new index with updated mapping
es.indices.create(index=new_index_name, body=mapping)

Here is how you would adjust your code for this purpose, using the new index:

seconds = [28.22, 26.92, 49.01, 26.1, 26.12, 25.22, 26.27]

es.index(
    index='hast-poc-v2', 
    document={
        'Seconds': seconds,
    }
)

Thanks for your response!

I am able to read the data into the dashboard/lens now. However, I cannot for the life of me understand how I can create a line graph from this.

Given my example data above (the seconds list/array), would you be able to guide me in how I can create a graph looking like this expertly designed graph?

1 Like

Thanks, @Pasjonsfrukt. I'm glad the first issue is resloved :chart_with_upwards_trend:.

First, double-check that you have selected the correct index pattern in your data view.

What are you looking to plot? Which field are you looking for on the X-Axis, and which on the Y? Is your X-Axis a timestamp field?

Our documentation on this subject may be helpful here, too:

To be a bit more specific, I want my x-axis to be "Attempts" (just starting at 1 and counting upwards), and the y-axis to be "Seconds". I created a quick line graph using matplotlib to show what I mean:

I tried passing a 2nd list to Kibana ("Attempts"), but I just cannot understand how it's possible to plot my y-values ("Seconds") over each attempt. I seem to be unable to extract the individual values from my list:

This is a very trivial thing to do in matplotlib, so I am hoping this can be easily achieved and I am just missing something?

Thanks, @Pasjonsfrukt for your follow up. What field type is attempts in your index?

I made Attempts an integer data type in ES, and I passed a list of ints.

@Pasjonsfrukt if I understood correctly each document contains a list of floats as seconds.
Unfortunately it is not possible to "unpack" a document field and iterate thru that to build a visualization: the "unpack" step should be explicitly done, perhaps with a runtime field.

Here's a short guide to build a runtime field for your use case:

I'll share here the scripted field template I've used:

for (item in doc['<my_field>']) {
    emit(item)
}

This emit within a for loop let unpack a single document into multiple documents, each with a single value of the list.

At this point the new field can be used within the visualization as a regular field.

Thanks, but unfortunately this does not work for me. Tried against my DenseVector (this produced errors) & my "Seconds" object defined by the mapping (outlined by jessgarson). In your example the type is set to "Keyword", this produced a java string error, and when I set it to "Double" I didn't get any errors but the new type behaved the same as my "Seconds" object.

How do people usually create line graphs? I feel like the use case I am after is quite basic, and I am starting to go a little crazy here. Am I approaching this incorrectly? :sweat_smile:

Or, to shift the question a little, how would you go about creating this type of line graph in Kibana, given my data input?

This is my simple python script that produced the graph from matplotlib, shown above:

import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5, 6, 7]
y = [28.22, 26.92, 49.01, 26.1, 26.12, 25.22, 26.27]

plt.plot(x, y, color="red", marker="o")
plt.xlabel("Attempts")
plt.ylabel("Seconds")

plt.show()

I am testing this as a potential tool for use in my team and my free trial is coming to an end soon, but I have to say this doesn't feel very intuitive at this point..! I do very much appreciate the great help, though.

FYI, that code is the same as that shown above, not adjusted

This is completely trivial.

If you do a PUT into elasticsearch with the same info in mathplotlib, you will get much the same results.

DELETE /hast-poc

PUT /hast-poc/_doc/1
{ "attempt" : 1 , "Seconds" : 28.22 }

PUT /hast-poc/_doc/2
{ "attempt" : 2 , "Seconds" : 26.92 }

PUT /hast-poc/_doc/3
{ "attempt" : 3 , "Seconds" : 49.01 }

PUT /hast-poc/_doc/4
{ "attempt" : 4 , "Seconds" : 26.1  }

PUT /hast-poc/_doc/5
{ "attempt" : 5 , "Seconds" : 26.12 }

PUT /hast-poc/_doc/6
{ "attempt" : 6 , "Seconds" : 25.22 }

PUT /hast-poc/_doc/7
{ "attempt" : 7 , "Seconds" : 26.27 }

then 2 different line graphs with lens, and with aggregation based, x-axis is attempts (Terms) and y-axis is median (there's only one value per bucket, so median, max, min will all be the same value).

Note elasticsearch would better fit if there were 1 million attempts per day, every day, and you wanted to look at min/median/max Seconds, split per hour or per-minute, but you would need add a timestamp to the ingested document in that case.

And "attempts" to do what? Lets say its a apache log and its a GET/PUT/POST/... request, and Seconds is the response time, then you could also log the actual URL of the HTTP call, and the remote IP, and ... and slice and dice the data a zillion ways.


I do wonder what your team's real world problems are?

You clearly didn't yet really understand some of the most basic concepts, which means once you try to solve a harder problem you will likely need more help. Like everything, it's a learning curve. Watch some introduction videos, play around with data, learn more, try a harder problem, and iterate.

BUT the "free trial" can last for ever, elasticsearch and Kibana are freely downloadable. Just some fancier features require a paid license.

I appreciate the feedback. You are correct in that I don't understand some of the most basic concepts in regards to ES/Kibana, and that is why I am here. I looked into some tutorials but I failed to find something that outlined what I was trying to do. At this moment I do not require any advanced features, I am just trying to understand if we can easily display simple graphs of our simple data. What "attempts" are does not matter in this context.

I actually asked whether this needed to be sent as individual key-value pairs in my 2nd reply, which seems to be what you are suggesting. I will give this a go, thank you.

there's ways without key-value pairs too, you can do different things to "imply" an x-axis, but if you want a line graph of something against something else, better start with 2-dimensional data. Other ways are therefore a bit hacky. Just IMHO.

The classic "test case" examples for learning elasticsearch/kibana are logs, which are essentially time series data with a bit (but often not a lot) of structure. If I were you, I'd start there.

That is very helpful advice, thank you very much!