Visualizations combinations

[kumar_k]

I want to visualize electric load profile weekly wise. The electric load profile has the electric load readings taken for every hour on each day.

Typical CSV columns are date,day,hour,load.

In a day 24 readings are taken, in a week (7 multiplied by 24)=168 and in a month (168 multiplied by 4)=672 are taken. So there are around 700 rows for a typical month.

I want to have visualizations for weekly wise, day wise, hour range wise etc., As of now I am just using only two columns with serial number and load. And two visualizations I made.

One as x axis serial number from 1 to 700. And y axis as load.

Second as x axis load and y axis as count.

How to have different combinations of visualizations for day wise and hour wise.

I am using python client to create elasticsearch object and using CSV reader to read the file and populate elasticsearch object.

Please guide me. thanks,

[warkolm (Mark Walkom)]

What does your data structure look like?

[kumar_k]

Hi Warkolm,
The data structure looks like below.

Date,day,hour,load in kW
1st April 2019,Monday,1 AM,50
1st April 2019, monday,2 AM,72
.
.
1st April 2019,Monday,9 AM,200
.
.
1st April 2019,Monday,11pm,55

2nd April 2019,Tuesday,1 Am,45
.
.
.

31st April 2019, Wednesday,1 AM,57
.
.
..

[warkolm (Mark Walkom)]

And what does it look like in Elasticsearch?

[kumar_k]

Since I don't know how to show date and day in visualization I made CSV with two columns serial number,load because they are numbers. I know only to create visualization with numbers.

I created one histogram with x axis as serial nbr and y axis as load.

Second visualization I made with count as y axis and x axis as load. So I can see one particular load how many times occurred in a month.

[warkolm (Mark Walkom)]

How are you loading this data into Elasticsearch?

You should really be converting the date, day and hour into a single timestamp field.

[kumar_k]

Ok Warkolm. Then I have to find how to write python client code for creating timestamp and populate that into elasticsearch object. So you mean to say I will have only two columns ? Timestamp, load.
Is my understanding correct ? What is the visualization then for this kind of data structure ?

[warkolm (Mark Walkom)]

Ideally you would have those, yes, plus the source of the data (which I guess is a meter?).

That way you can do changes over time using line/bar graphs and separate by source.

[kumar_k]

Sorry, I didn't understand. So should I have data structure like this ?

Date, day, time, Timestamp, load in kilowatts.

Could you point me to some documentation explaining visualizations having Timestamp and also data having non number data.

[warkolm (Mark Walkom)]

Like this;

{
  "timestamp" : value,
  "reading": value,
  "source": value
}

If this is in CSV, then you can easily ingest it using Beats or Logstash, without having to write your own code from scratch.

[kumar_k]

Condemn my less intelligence in elasticsearch ...

I feel I have to work more on understanding beats and logstash and how to interpret data structure in relation to elasticsearch.

By the bye, what do you mean by "source" ? I understood "reading" which is electrical load. But what is the "source" which you are mentioning

[warkolm (Mark Walkom)]

No worries!

Regarding source; I am guessing that you have load readings from different sources?

[kumar_k]

Thank you warkolm.

The "load" value taken from an electric meter hourly basis manually by from the meter display. This is the single source for me.

[warkolm (Mark Walkom)]

Ok. I was just thinking you may want to be able to differentiate if you end up with multiple different sources, but sounds like it's not relevant.

[kumar_k]

Yeah. Multiple sources of data as of now not there. But good that you have given valuable insight what if there are multiple sources of data. I will have to explain this.

Could you point me to some visualization involving timestamp. I am having little difficulty in understanding how a timestamp actually appears in a typical visualization. Because I always assume visualization means only numbers on x axis and y axis. Eager to know how timestamp appears in a visualization. Thanks

[warkolm (Mark Walkom)]

https://www.elastic.co/guide/en/kibana/current/most-frequent.html for eg.

Also take a look at our blog, there are tonnes of examples from ourselves and our users.

[kumar_k]

Thanks warkolm for your mini tutorial in this post. I will go through the blog and the examples.