I have a list of floats that I am passing to ES/Kibana through the Python API.
I am able to see the document in Discover, and the list is visible there as expected. Now, I want to create a graph that iterates over this list and creates a dot for each data point (x-axis), between the min and max values (y-axis).
However, I do not see my object (list) when trying to create a graph using the Lens feature. Am I not able to create a graph based off a list?
Looks like this object is of type "dense_vector" as far as ES is concerned, but it is not clear to me what I can do with this data type.
Essentially I want to plot 'seconds' over the x-axis, with lets say the max y-value as 49.01 and min 0 (I could of course pass individual variables for the y-axis if necessary).
I tried passing a simple string as the value to a key in the 'document', and that shows up as a valid value to use in my dashboard. Do I need to pass my list values as individual key-value pairs?
Thanks for following up, @Pasjonsfrukt. I think the issue is that you are using dense vectors, commonly used to store high-dimensional vectors, like embeddings for machine learning models, rather than individual data points like floats that you would graph. Using the float data type might be better here. You can reindex or create a new index.
To create a new index with the proper mappings, it would be something like this:
new_index_name = "hast-poc-v2"
# Define new mapping
mapping = {
"mappings": {
"properties": {
"Seconds": {
"type": "float" # Use float for numerical data
}
}
}
}
# Create the new index with updated mapping
es.indices.create(index=new_index_name, body=mapping)
Here is how you would adjust your code for this purpose, using the new index:
I am able to read the data into the dashboard/lens now. However, I cannot for the life of me understand how I can create a line graph from this.
Given my example data above (the seconds list/array), would you be able to guide me in how I can create a graph looking like this expertly designed graph?
To be a bit more specific, I want my x-axis to be "Attempts" (just starting at 1 and counting upwards), and the y-axis to be "Seconds". I created a quick line graph using matplotlib to show what I mean:
I tried passing a 2nd list to Kibana ("Attempts"), but I just cannot understand how it's possible to plot my y-values ("Seconds") over each attempt. I seem to be unable to extract the individual values from my list:
@Pasjonsfrukt if I understood correctly each document contains a list of floats as seconds.
Unfortunately it is not possible to "unpack" a document field and iterate thru that to build a visualization: the "unpack" step should be explicitly done, perhaps with a runtime field.
Here's a short guide to build a runtime field for your use case:
Thanks, but unfortunately this does not work for me. Tried against my DenseVector (this produced errors) & my "Seconds" object defined by the mapping (outlined by jessgarson). In your example the type is set to "Keyword", this produced a java string error, and when I set it to "Double" I didn't get any errors but the new type behaved the same as my "Seconds" object.
How do people usually create line graphs? I feel like the use case I am after is quite basic, and I am starting to go a little crazy here. Am I approaching this incorrectly?
Or, to shift the question a little, how would you go about creating this type of line graph in Kibana, given my data input?
This is my simple python script that produced the graph from matplotlib, shown above:
import matplotlib.pyplot as plt
x = [1, 2, 3, 4, 5, 6, 7]
y = [28.22, 26.92, 49.01, 26.1, 26.12, 25.22, 26.27]
plt.plot(x, y, color="red", marker="o")
plt.xlabel("Attempts")
plt.ylabel("Seconds")
plt.show()
I am testing this as a potential tool for use in my team and my free trial is coming to an end soon, but I have to say this doesn't feel very intuitive at this point..! I do very much appreciate the great help, though.
then 2 different line graphs with lens, and with aggregation based, x-axis is attempts (Terms) and y-axis is median (there's only one value per bucket, so median, max, min will all be the same value).
Note elasticsearch would better fit if there were 1 million attempts per day, every day, and you wanted to look at min/median/max Seconds, split per hour or per-minute, but you would need add a timestamp to the ingested document in that case.
And "attempts" to do what? Lets say its a apache log and its a GET/PUT/POST/... request, and Seconds is the response time, then you could also log the actual URL of the HTTP call, and the remote IP, and ... and slice and dice the data a zillion ways.
I do wonder what your team's real world problems are?
You clearly didn't yet really understand some of the most basic concepts, which means once you try to solve a harder problem you will likely need more help. Like everything, it's a learning curve. Watch some introduction videos, play around with data, learn more, try a harder problem, and iterate.
BUT the "free trial" can last for ever, elasticsearch and Kibana are freely downloadable. Just some fancier features require a paid license.
I appreciate the feedback. You are correct in that I don't understand some of the most basic concepts in regards to ES/Kibana, and that is why I am here. I looked into some tutorials but I failed to find something that outlined what I was trying to do. At this moment I do not require any advanced features, I am just trying to understand if we can easily display simple graphs of our simple data. What "attempts" are does not matter in this context.
I actually asked whether this needed to be sent as individual key-value pairs in my 2nd reply, which seems to be what you are suggesting. I will give this a go, thank you.
there's ways without key-value pairs too, you can do different things to "imply" an x-axis, but if you want a line graph of something against something else, better start with 2-dimensional data. Other ways are therefore a bit hacky. Just IMHO.
The classic "test case" examples for learning elasticsearch/kibana are logs, which are essentially time series data with a bit (but often not a lot) of structure. If I were you, I'd start there.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.