Ruby for loop logstash

Elastic Stack Logstash
1

I am trying to convert the epoch timestamps which comes in an array.

"steps": [
    {
        "number": 1,
        "status": "COMPLETE",
        "doRetry": null,
        "progress": null,
        "startTime": 1710431698,
        "rowCount": null,
        "endTime": 1710431699
    },
    {
        "number": 2,
        "status": null,
        "startTime": 1710431698,
        "doRetry": null,
        "progress": null
    },
    {
        "number": 3,
        "status": "COMPLETE",
        "startTime": 1710431698,
        "doRetry": null,
        "progress": null,
        "endTime": 1710431734,
        "rowCount": null
    },
    {
        "number": 4,
        "status": null,
        "doRetry": null,
        "progress": null
    },
    {
        "number": 5,
        "status": null,
        "doRetry": null,
        "startTime": 1710431698,
        "progress": null
    },
    {
        "number": 6,
        "status": null,
        "doRetry": null,
        "endTime": 1710431734,
        "progress": null
    }
]

```

Here is my ruby and date filter
```
if  [steps][startTime] {ruby{code => "event.set('endTime',event.get('endTime').to_i* 1000)"}}
    if  [steps][endTime] {ruby{code => "event.set('endTime',event.get('endTime').to_i* 1000)"}}
    

date {
      match => [ "[steps][startTime]","UNIX_MS" ]
      target => "[steps][startTime]"
      timezone =>	"America/Toronto"
    }
    date {
      match => [ "[steps][endTime]","UNIX_MS" ]
      target => "[steps][endTime]"
      timezone =>	"America/Toronto"
    }

it didn't work

Note: startTime and endTime doesn't come in each index of array.

2

If you want to add the timestamps under [steps] then you will need to remove_field the array of hashes. Your timestamps are seconds, not milliseconds, so use UNIX.

    ruby {
        code => '
            steps = event.get("steps")
            if steps.is_a? Array
                steps.each { |x|
                    if x["endTime"]  then event.set("[@metadata][endTime]", x["endTime"]) ; end
                    if x["startTime"]  then event.set("[@metadata][startTime]", x["startTime"]) ; end
                }
            end
        '
        remove_field => [ "steps" ]
    }
    date {
        match => [ "[@metadata][startTime]","UNIX" ]
        target => "[steps][startTime]"
        timezone => "America/Toronto"
    }
3

Thanks @Badger for your quick response, instead of removing the field, I would like to rename it to some other name because their are these two fields startTime and endTime coming in the api data which is not included in my question. So there would be just startTime and endTime fields also [steps][startTime] and [steps][endTime]

4

OK, so use mutate+rename.

5

Thanks @Badger

  1. For some reason, date filter UNIX doesn't work. only UNIX_MS works, so i am multiplying with 1000 to get converted into milliseconds.

  2. Thanks for this code, it works but i need to multiply it with 1000 to convert into milliseconds and use UNIX_MS.



ruby {
        code => '
            steps = event.get("steps")
            if steps.is_a? Array
                steps.each { |x|
                    if x["endTime"]  then event.set("[@metadata][endTime]", x["endTime"]) ; end
                    if x["startTime"]  then event.set("[@metadata][startTime]", x["startTime"]) ; end
                }
            end
        '
    }
    mutate {
        rename => { "steps" => "Steps" }
    }
    date {
        match => [ "[@metadata][startTime]","UNIX" ]
        target => "[steps][startTime]"
        timezone => "America/Toronto"
    }
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.