Creating visualization with timestamp un y axis and not count

[Anusha_Kusanghi]

Hi All,

We have a index which stores the status of job running in controlm platform , like job name ,id ,job start time end time and so on...
We want to create a visualization: putting date in y axis (date on which date job executed)
and end_time in x-axis.
example of on how the graph should look like:

Here is a sample log message:

  "_source": {
    "due_in": "0558",
    "state": null,
    "max_rerun": 0,
    "due_out": "0559",
    "task_type": "Job",
    "memname": "TD94A143",
    "ind_cyclic": "E",
    "order_lib": "CONTROLM.TD94A1.PROD.SCHEDULE",
    "summary": "TD94A143/20230619173054/GRWL9/0",
    "cyclic": "0",
    "avg_start_time": "241437",
    "lpar": " ",
    "job_id": "35154",
    "description": "UPDATE TIMESTAMP AMC S USED IN LAST RUN RL00000000",
    "elapsed": 1,
    "job_name": "TD94A143",
    "task_class": " ",
    "end_time": "2023-06-19T22:30:58.000Z",
    "max_wait": 10,
    "rerun_mem": " ",
    "application": "WINKSTOC",
    "to_time": ">",
    "prev_odate_rerun_counter": 0,
    "critical": "0",
    "@timestamp": "2023-06-19T15:30:54.000Z",
    "status": "Ended OK",
    "type": "controlmxpslogs",
	"split_ts_by_date": "20-Jun-2023"

  }

We always get count on y-axis but here it not our requirement how can we achieve this.
let me know for more information.

TIA

[Marta_Bondyra]

Hi Anusha,
this seems like a non-trivial case and I am not sure if using Lens you can achieve exactly what you want because you want to work with non-aggregated data. However we can get pretty close. What I would try to do is to create an extra runtime field to convert end_time to number and make sure that the end_time is comparable across the executions. Here's the screenshot of what I did to create runtime field:

I took a @timestamp field, converted it to a number of milliseconds, then used a modulo function to only get the time of the day (86400000 is the number of ms in the day).

Then I set the format duration as in the screenshot. I then can use this value on my chart as a maximum or last value.
The problem is that you are unable to print a pretty hh:mm format, you will instead get a fraction of the hour (so instead of 12:30, 12.5). Currently I don't see a solution for it in Lens.

You could try Vega, but it's not trivial.

[Marta_Bondyra]

Hi Anusha, a small correction, you can actually achieve what you want, I forgot about one feature we could use here. Firstly, we need to slightly change our runtime field to get the number of seconds instead of milliseconds (remember changing te script AND the format):

Then you can use a custom format in your visualization:

Let me know if that's helpful!

[Anusha_Kusanghi]

Hi Marta,

Thanks a lot for the suggestions , This is a potential solution.
I tried to follow the steps given by you but i dont see custom format for 'end_time_converted' fiels

We are using elastic 7.17.5 version, which version your using?

And we also want a job name/id to be shown with different color coding or job name if we hover over the graph.
Regards,
Anusha K

[Marta_Bondyra]

Aaaah, you are right, custom format is added to the upcoming 8.9 version. So till then you cannot use it and the only option is to have a fraction of the hour here.

When it comes to the color coding for the job, you have to create breakdown by dimension. You can either use top values which will take the most used values from your documents or you can specifically list your jobs by using the filters. Examples below.