I am running into an issue where I can correctly convert document timestamps within a watcher payload and store into a list but I am unsure how to correctly map this to the returned payload.
Hopefully some code will help clear my issue up a bit. Here's a transform in my watcher where I am returning a new payload and mapping fields within those documents to some defined strings:
"transform": {
"chain": [
{
"script": {
"source": "ctx.vars.ogStack = ctx.payload",
"lang": "painless"
}
},
{
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"idx-pattern-*"
],
"rest_total_hits_as_int": true,
"body": {
"size": 1000,
"sort": [
"timestamp.keyword"
],
"query": {
"bool": {
"must": [
{
"wildcard": {
"host.name": {
"value": "*af?m*"
}
}
},
{
"query_string": {
"query": "{{#ctx.payload.hits.hits}} {{_source.session_id}} {{/ctx.payload.hits.hits}}"
}
}
],
"filter": {
"range": {
"@timestamp": {
"from": "{{ctx.trigger.scheduled_time}}||-1d",
"to": "{{ctx.trigger.triggered_time}}"
}
}
}
}
}
}
}
}
},
{
"script": {
"source": """
def testl = [];
for (def doc : ctx.payload.hits.hits){
def temp = doc._source.timestamp;
StringBuilder sb = new StringBuilder(temp);
sb.insert(10,'T');
sb.deleteCharAt(11);
sb.append('Z');
ZonedDateTime zdt = ZonedDateTime.parse(sb);
ZonedDateTime ct = zdt.withZoneSameInstant(ZoneId.of("America/Chicago"));
DateTimeFormatter dtf = DateTimeFormatter.ofPattern("EEE, d MMM yyyy h:mm:ss.SSSS a z");
String date_formatted = ct.format(dtf);
testl.add(date_formatted);
}
return [
'new': ctx.payload.hits.hits.stream().map(t -> {return [
'time': t._source.timestamp,
'msg': t._source.message,
'debug': t._source.debug,
'ses_id': t._source.session_id,
'stack': t._source.stack,
'host': t._source.host.name,
'user': t._source.user,
'clusters': t._source.host.system_name,
'test': testl]}).collect(Collectors.toList())
]
""",
"lang": "painless"
}
}
]
}
After the search level, a new payload is returned and loaded into script transform below it. This payload consists of lets say 100 documents. Each document has multiple fields including a timestamp. What I want is for the timestamp to be converted for each document. I do this separately using the for loop. I iterated over each document in the payload, pulled out the timestamp, converted it, and stored in the list testl . I then added this to the java stream (for testing purposes, I know this isn't the way to go) to ensure the dates within the list are correctly converted.
So now I have a list of correctly converted dates but no mappings to their unconverted counterparts in the new payload. It's important I have the converted timestamps in the new payload and not another payload object for output purposes. Can I apply a function to just the timestamp in the java stream? That seems like a logical solution but I'm not sure how or if that's possible.
If I can clarify anything, please let me know. Thanks in advance.