Visualize arrays of values over time

[smlbiobot (SML)]

I am trying to visualize the change of values over time with fields that are stored inside another field.

I don’t know how to explain this using text so I will show you a screenshot of my data here and perhaps you can help me better:

Originally I created a single field called roles which has all the role names / ids / counts grouped together but ES complains that objects are not well supported and that’s why I turn them into 3 arrays instead.

That said, I don’t know how I might show the value of a role over time. What I need: the count changes overtime for each role_name. My work around right now is that I dynamically created a ton of fields:

and now I am able to track them overtime / use it in the Metric visualization. But as you can see this gets pretty bad as the number of fields I have just increased exponentially.

Can you help?

[Stacey_Gammon]

I recommend adjusting your data ingestion so it separates this information into individual fields and documents. If you can index each tuple as a separate document, with fields role_name, role_id, role_count and timestamp, this then becomes easy to visualize, and you can filter by role_name:Battle-Bay to narrow down to a single role.

[smlbiobot (SML)]

Interesting… but I don’t really know how to do that. Right now the library I use just send everything via logstash using this config:

input {
    udp {
        port => 5959
        codec => json
    }
}
output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
    stdout {
        codec => rubydebug
    }
}

these then get sent to logstash-* — I don’t actually know this well enough to create documents.

This is the library I am using btw: https://pypi.python.org/pypi/python-logstash — I need to use Python to send data via logstash to ES. Or are you suggesting that I communicate directly to ES? If so, what would you suggest that I do? Please point me to the necessary documentation if possible.

Should I in fact just skip logstash and use this instead: https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html

Much appreciated!

[smlbiobot (SML)]

You know what, never mind… I was wondering why it previously complained and it won‘t let me model things the way I wanted to. But I just did a test run and this seems to be working right now:

Previously when I did this though it complains and said that field data cannot be an object…

[smlbiobot (SML)]

@Stacey_Gammon oops but wait… while it didn’t complain about the object, how can I grab those values in this visualize graph type:

Coz the field is not a number and it doesn’t show up in the dropdown list.

That btw was why I did what I did… just to get it to show up…