# Unable to perform airthmetic operations in ruby using logstash pipeline

**URL:** <https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587>\
**Category:** Logstash\
**Created:** [April 7, 2023, 4:37pm UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587 "2023-04-07T16:37:03Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sujith\_Nair](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujith_nair/32/119254_2.png) [@Sujith\_Nair](https://discuss.elastic.co/u/Sujith_Nair)\
**Post date:** [April 7, 2023, 4:37pm UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/1 "2023-04-07T16:37:03Z")

</div>

Hi guys,  
I am facing an issue where i am trying to perform an airthmetic operation using ruby but in at the field section i am getting the same value not the subtracted value.

event.set('[d]', (event.get('[b]').to\_f) - (event.get('[a]').to\_f))  
a = 5, b =4

expected output  
d = 1

actual output  
d =5,  
d =4

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 7, 2023, 6:02pm UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/2 "2023-04-07T18:02:44Z")

</div>

You need to provide more information, because

```
input { generator { count => 1 lines => [''] } }

output { stdout { codec => rubydebug { metadata => false } } }
filter {
   mutate { add_field => { "[a]" => "4" "[b]" => "5" } }
    ruby { code => 'event.set("[d]", (event.get("[b]").to_f) - (event.get("[a]").to_f))' }
}

```

produces

```
         "d" => 1.0,

```

---

<div class="post-metadata">

**Author:** ![Sujith\_Nair](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujith_nair/32/119254_2.png) [@Sujith\_Nair](https://discuss.elastic.co/u/Sujith_Nair)\
**Post date:** [April 8, 2023, 3:07am UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/3 "2023-04-08T03:07:10Z")

</div>

I tried this but still i am getting b = 5 and a = -4

Here's my config file

input {  
file {  
path =\> "C:/data/\*.txt"  
start\_position =\> "beginning"  
}  
}

filter {  
grok{  
match =\> {  
"message"=\> ["%{YEAR:year}-%{MONTHNUM:month}-%{INT:day}\s%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}\s%{WORD:event\_type}\s%{DATA:Transaction\_Number}\s%{LOGLEVEL:LOGLEVEL}\s\s+%{JAVACLASS:error\_file}\s%{DATA:hash\_type}TransactionId\s-\s%{NUMBER:TransactionId:INT}+,+\sTransactionReferenceId\s-\s%{NUMBER:TransactionReferenceId:INT}+]+%{GREEDYDATA:msgbody}"]  
}  
}  
mutate {  
add\_field =\> { "date" =\> "%{year}-%{month}-%{day} %{hour}:%{minute}:%{second}"  
}  
}  
mutate{  
convert=\>  
{"[hour]" =\> "integer"  
"[minute]" =\> "integer"  
"[second]" =\> "integer" }  
}  
}  
filter {  
json {  
source =\> "msgbody"  
}  
}  
filter{  
if [messageType] == "CReq" {  
mutate{  
add\_field =\> {  
"MsgType" =\> "%{messageType}"  
"Creq\_sec" =\> "%{second}"  
}  
convert=\>  
{"[Creq\_sec]" =\> "integer"  
}   
}  
}  
else if [messageType] == "CRes" {  
mutate{  
add\_field =\> {  
"MsgType" =\> "%{messageType}"  
"Cres\_sec" =\> "%{second}"  
}  
convert=\>  
{"[Cres\_sec]" =\> "integer"  
}  
}  
}  
else if [messageType] == "AReq" {  
mutate{  
add\_field =\> {  
"MsgType" =\> "%{messageType}"  
"Areq\_sec" =\> "%{second}"  
}   
}  
}  
else if [messageType] == "ARes" {  
mutate{  
add\_field =\> {  
"MsgType" =\> "%{messageType}"  
"Ares\_sec" =\> "%{second}"  
}   
}  
}  
else if [messageType] == "RReq" {  
mutate{  
add\_field =\> {  
"MsgType" =\> "%{messageType}"  
"Rreq\_sec" =\> "%{second}"  
}  
convert=\>  
{"[Rreq\_sec]" =\> "integer"  
}   
}  
}  
else if [messageType] == "RRes" {  
mutate{  
add\_field =\> {  
"MsgType" =\> "%{messageType}"  
"Rres\_sec" =\> "%{second}"  
}  
convert=\>  
{"[Rres\_sec]" =\> "integer"  
}   
}  
}  
else if [messageType] == "WS OTP Request" {  
mutate{  
add\_field =\> {  
"MsgType" =\> "%{messageType}"  
}  
}  
}  
else if [messageType] == "WS OTP Responce" {  
mutate{  
add\_field =\> {  
"MsgType" =\> "%{messageType}"  
}  
}  
}  
}  
filter{  
if [MsgType] == "AReq" {  
mutate{  
add\_field =\> {  
"AReq\_Transaction\_id" =\> "%{TransactionId}"  
}  
}  
}  
if [MsgType] == "ARes" {  
mutate{  
add\_field =\> {  
"ARes\_Transaction\_id" =\> "%{TransactionId}"  
}   
}  
}  
if [MsgType] == "CReq" {  
mutate{  
add\_field =\> {  
"CReq\_Transaction\_id" =\> "%{TransactionId}"  
}   
}  
}  
if [MsgType] == "CRes" {  
mutate{  
add\_field =\> {  
"CRes\_Transaction\_id" =\> "%{TransactionId}"  
}   
}  
}  
if [MsgType] == "RReq" {  
mutate{  
add\_field =\> {  
"RReq\_Transaction\_id" =\> "%{TransactionId}"  
}   
}  
}  
if [MsgType] == "RRes" {  
mutate{  
add\_field =\> {  
"RRes\_Transaction\_id" =\> "%{TransactionId}"  
}  
convert=\>  
{"[RRes\_Transaction\_Id]" =\> "integer"   
}   
}  
}  
}  
filter {  
mutate {  
copy =\> {  
"AReq\_Transaction\_id" =\> "AReq\_Transaction\_Id"  
"ARes\_Transaction\_id" =\> "ARes\_Transaction\_Id"  
"RReq\_Transaction\_id" =\> "RReq\_Transaction\_Id"  
"RRes\_Transaction\_id" =\> "RRes\_Transaction\_Id"  
"Areq\_sec" =\> "Areq\_Sec"  
"Ares\_sec" =\> "Ares\_Sec"  
"Rreq\_sec" =\> "Rreq\_Sec"  
"Rres\_sec" =\> "Rres\_Sec"  
}  
}  
}  
filter{  
mutate{  
convert=\>  
{  
"[AReq\_Transaction\_Id]" =\> "string"  
"[ARes\_Transaction\_Id]" =\> "string"  
"[RReq\_Transaction\_Id]" =\> "string"  
"[RRes\_Transaction\_Id]" =\> "string"  
"[Areq\_Sec]" =\> "integer"  
"[Ares\_Sec]" =\> "integer"  
"[Rreq\_Sec]" =\> "float"  
"[Rres\_Sec]" =\> "float"  
}  
}  
}

filter {  
ruby {  
code =\>'  
a = event.get("[Areq\_Sec]").to\_f  
b = event.get("[Ares\_Sec]").to\_f  
c = b-a;  
event.set("[req\_sec]", c)  
'  
}  
}

filter{  
mutate{  
remove\_field =\> ["message","@version","event","@timestamp","original","host","log","tags","day","month","year","time"]  
}  
}

output {   
stdout {  
codec =\> rubydebug  
{ metadata =\> false } }  
}

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 8, 2023, 4:10am UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/4 "2023-04-08T04:10:45Z")

</div>

Can you format your post to use the Preformatted text option? The `</>` button, it makes it much more easier to read.

Also, can you share a sample of your message so your pipeline could be replicated?

From what you shared, it seems that you have some logic wrong in it.

Your ruby code is doing this:

```auto
filter {
    ruby {
        code =>'
        a = event.get("[Areq_Sec]").to_f
        b = event.get("[Ares_Sec]").to_f
        c = b-a;
        event.set("[req_sec]", c)
        '
    }
}

```

It is subtracting the fields `Areq_Sec` and `Ares_Sec`.

But from your pipeline those fields are not present in the same message:

```auto
else if [messageType] == "AReq" {
    mutate{
        add_field => {
            "MsgType" => "%{messageType}"
            "Areq_sec" => "%{second}"
        }
    }
}
else if [messageType] == "ARes" {
    mutate{
        add_field => {
            "MsgType" => "%{messageType}"
            "Ares_sec" => "%{second}"
        }
    }
}

```

So, when you have a message with `messageType` equals to `AReq`, you will have the field `Areq_Sec`, but you will not have the field `Ares_Sec`, and when the `messageType` is `ARes`, you will have `Ares_Sec`, but not `Areq_Sec`.

Your ruby code will always have one of the terms equals to zero.

Can you provide more context about your message and what you want to achieve?

---

<div class="post-metadata">

**Author:** ![Sujith\_Nair](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujith_nair/32/119254_2.png) [@Sujith\_Nair](https://discuss.elastic.co/u/Sujith_Nair)\
**Post date:** [April 8, 2023, 4:17am UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/5 "2023-04-08T04:17:08Z")

</div>

I am getting the values perfectly for Areq\_Sec and Ares\_Sec. The problem I am facing is while doing the airthmetic operation using the ruby filter. It's not giving me the desired output.

For example :-  
Its giving me the output like  
Areq\_Sec = 4, Ares\_Sec = 5  
c = b - a  
desired output = 1

actual output  
req\_sec = 5 and req\_sec = -4

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 8, 2023, 4:18am UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/6 "2023-04-08T04:18:03Z")

</div>

Your ruby filter seems to be trying to calculate the difference between Areq\_Sec and Ares\_Seq. I guess these are timings or timestamps for requests and responses, and you want to find how long a request took.

There are multiple issues here.

The ruby filter tries to subtract [Areq\_Sec] from [Ares\_Sec] to find the difference between the timing of the request and response. However, those fields are in different events, and unless you configure a filter that combines them, each event is independent. If a field does not exist then the event.get will return **nil** and .to\_f will return zero. This is why your two events get 0-4 (-4) and 5-0 (5).

In your third filter section, for each [messageType] you try to do mutate+convert as well as mutate+add\_field. You cannot convert a field that you add in the same mutate. The order of mutate operations is [documented](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-proc_order), but nobody tells you that event decoration (add\_field/tag, remove\_field/tag), if done, is always done last.

Doing the [Areq\_sec] to [Areq\_Sec] copy and convert has no obvious purpose. You are doing a .to\_f in the ruby filter, so it is not required in logstash (but I understand that the systems you are stashing data in may require it).

If your messages all have a transaction id then it should be fairly simple to do an [aggregate](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html) filter that merges these field for log entries that refer to the same request.

---

<div class="post-metadata">

**Author:** ![Sujith\_Nair](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujith_nair/32/119254_2.png) [@Sujith\_Nair](https://discuss.elastic.co/u/Sujith_Nair)\
**Post date:** [April 8, 2023, 4:23am UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/7 "2023-04-08T04:23:28Z")

</div>

I tried with the aggregate filter but I am unable to find the desired output. There's a problem while using the aggregate filter. A single transaction\_id contains multiple request and response. I want to calculate the request and response for two events like Areq and Ares. In aggregate it will calculate the seconds for the whole request and response for that particular transaction\_id. Same with the elapsed filter

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 8, 2023, 4:26am UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/8 "2023-04-08T04:26:20Z")

</div>

If you use pipeline.workers 1 and pipline.ordered it may be possible to do that, but without seeing the log file or a very detailed specification of it by you we cannot guess how.

---

<div class="post-metadata">

**Author:** ![Sujith\_Nair](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujith_nair/32/119254_2.png) [@Sujith\_Nair](https://discuss.elastic.co/u/Sujith_Nair)\
**Post date:** [April 8, 2023, 5:41am UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/10 "2023-04-08T05:41:07Z")

</div>

2023-02-22 03:58:04 MUT 795521455 INFO AuditLogService ---- [TransactionId - 2305315306634431865, TransactionReferenceId - 2305315306634431865]{"messageType":"AReq"}  
2023-02-22 03:58:05 MUT 795522299 INFO AuditLogService ---- [TransactionId - 2305315306634431865, TransactionReferenceId - 2305315306634431865]{"messageType":"ARes"}  
2023-02-22 04:00:11 MUT 795649016 INFO AuditLogService ---- [TransactionId - 2305315306634431865, TransactionReferenceId - 2305315306634431865]{"messageType":"RReq"}  
2023-02-22 04:00:13 MUT 795650422 INFO AuditLogService ---- [TransactionId - 2305315306634431865, TransactionReferenceId - 2305315306634431865]{"messageType":"RRes"}

This is the log file for the above given configuration. The messageType will be different for that particular transaction id.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 8, 2023, 5:08pm UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/11 "2023-04-08T17:08:26Z")

</div>

You could try this. Note that when doing a numeric conversion in grok the suffix to the field name has to be ":int", not ":INT". You must set pipeline.workers to 1 and pipeline.ordered must evaluate to true in order to use aggregate.

```
    grok{
        pattern_definitions => { "customDate" => "%{YEAR}-%{MONTHNUM}-%{INT}\s%{HOUR}:%{MINUTE}:%{SECOND}\s%{WORD}" }
        match => { "message"=> "%{customDate:[@metadata][ts]}\s%{DATA:Transaction_Number}\s%{LOGLEVEL:LOGLEVEL}\s+%{JAVACLASS:error_file}\s%{DATA:hash_type}TransactionId\s-\s%{NUMBER:TransactionId:int}+,+\sTransactionReferenceId\s-\s%{NUMBER:TransactionReferenceId:int}+]+%{GREEDYDATA:[@metadata][msgbody]}" }
    }
    mutate { gsub => ["[@metadata][ts]", "MUT", "Etc/GMT-4" ] }
    date { match => ["[@metadata][ts]", "YYYY-MM-dd HH:mm:ss ZZZ" ] }
    json { source => "[@metadata][msgbody]" }
    aggregate {
        task_id => "%{TransactionId}"
        code => '
            begin
                type = event.get("messageType")
                # type1 will be A/C/R/WS OPT
                type1 = type.gsub(/Re[qs].*/, "").strip
                # type2 will be an array containg Req or Res
                type2 = type.match(/Re[qs]/)
                if type2[0] == "Req"
                    map[type1] = event.get("@timestamp").to_f
                else
                    event.set("req_sec", event.get("@timestamp").to_f - map[type1])
                end
            rescue
            end
        '
    }

```

---

<div class="post-metadata">

**Author:** ![Sujith\_Nair](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujith_nair/32/119254_2.png) [@Sujith\_Nair](https://discuss.elastic.co/u/Sujith_Nair)\
**Post date:** [April 9, 2023, 1:54pm UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/12 "2023-04-09T13:54:45Z")

</div>

> [@Badger](#):
>
> `Req`

Thank you @Badger! It worked

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2023, 1:55pm UTC](https://discuss.elastic.co/t/unable-to-perform-airthmetic-operations-in-ruby-using-logstash-pipeline/329587/13 "2023-05-07T13:55:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.