Scripted field considers the last updated value for calculation

[Ruthu_Ramya]

Consider my log example:

Nov 30,15:02:42.080 Start playback
Nov 30,15:02:42.082 Song 1 played
Nov 30,15:02:42.085 Song 2 played
Nov 30,15:02:42.089 Song 3 played
Nov 30,15:02:42.090 End playback
Nov 30,15:02:43.060 Start playback
Nov 30,15:02:43.064 Song 4 played
Nov 30,15:02:43.068 Song 2 played
Nov 30,15:02:43.072 Song 5 played
Nov 30,15:02:43.090 End playback

After parsing the data into Elasticsearch, I need to plot a graph in Kibana with X-axis : cumulative time from start playback until each song played. So, I create a new 'scripted field' in Kibana as

(doc['SongPlayedTime'].value - doc['StartPlaybackTime'].value)

The problem I have is that value of doc['StartPlaybackTime'].value is always = Nov 30,15:02:43.060.
It has to be corresponding to the cd-run. I have the individual cd-run IDs in elasticsearch as additional fields:

Timestamp           | CD-run           | StartTime    | Song | Other events...
Nov 30,15:02:42.080 | start1_timestamp | 15:02:42.080 |
Nov 30,15:02:42.082 | start1_timestamp |              | 1
.
.
Nov 30,15:02:43.060 | start2_timestamp | 15:02:43.060 |
Nov 30,15:02:43.064 | start2_timestamp |              | 4

Is there some way I can always refer to correct 'StartPlaybackTime' corresponding to the particular run-event using cd-run-ID.

Appeciate suggestions. Thanks.

Regards,
Ruthu

[thomasneirynck]

@Ruthu_Ramya,

I don't think there is. Scripted fields allow you to have a script read field-values from a particular document. However, to achieve your end, you need to do a "sub-query" (for lack of better word), to read-out the value from another document.

Is there a way you can index the data differently?

E.g. when you index, instead of only indexing the CD-run field to every document, also index the corresponding CD-run 'StartPlaybackTime'?

[Ruthu_Ramya]

@thomasneirynck,

Thanks for you suggestion, it seems to get the stuff working.

I was thinking if mapping could be used while indexing/storing and then use it in the scripted field as

[[CD-run].StartPlaybackTime].value

Any suggestions if such a thing can be achieved.

I have been trying something like this currently:

add_field => {"CDRun.StartTime" => "%{'parsedValueFromLog'}"}

leads to a logstash parsing error (will also check on logstash specific forum):

"error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [CDRun] of different type, current_type [text], merged_type [ObjectMapper]"}

Regards,
Ruthu

[thomasneirynck]

hi @Ruthu_Ramya,

hmmmm, not sure about that, I wouldn't think so, but I'm not an expert on Logstash.

I'd suggest moving this question to the Logstash forum: https://discuss.elastic.co/c/logstash. There might be somebody there who can help you with that.

thx!