Logstash if condition skipped

katara · July 3, 2020, 12:35pm

Hi,
I have a logstash filter as below:

mutate {
convert => {"[xmldata][metric-data][metricValues][metric-value][value]" => "integer" }
}


if [xmldata][metric-data][metricValues][metric-value][value] == "200" #also tried 200
{
mutate { add_field => { "Status" => "Up"} }
}
else
{
mutate { add_field => { "Status" => "Down"} }
}

and here is the field in ES :

Even if the metric has the value 200, all my rows are DOWN.
It is not taking the if and only considers the else.

What am i missing? Please help me.

Katara

Badger · July 3, 2020, 3:08pm

If you are doing the convert to integer you should be comparing it with 200, not with "200".

Take a look at what is produced by

output { stdout { codec => rubydebug } }

or else on the JSON tab when you expand a document on the Discovery tab in kibana. If this is the product of an xml filter I would not be surprised to find that the data is an array and you really need to test (and convert) [xmldata][metric-data][metricValues][metric-value][value][0]

katara · July 3, 2020, 7:27pm

@badger,
Thank you! I tried with a simple 200, and I also tried not converting the value and tried "200" for which both did not work.
Here's how my source data in the api looks.

I basically used a split filter to get each values.

I have set my output to update with a primary key field so even if there are more than one values, it keeps updating the same row value based on the key field value.
> action=>update
> document_id => "%{application}"
> doc_as_upsert =>true

Badger · July 3, 2020, 7:38pm

I do not need to see the source data. I need to see what an event looks like.

katara · July 6, 2020, 6:21am

Hi @Badger,
Here's the data from Kibana discover,

@version:
1
URL:
http://icdupedix:5000/dashboard/
Status:
Down
tags:
multiline, _split_type_failure
http_poller_metadata.request.method:

get
http_poller_metadata.name:

icdua03_dashboard
http_poller_metadata.response_headers.date:
Mon, 06 Jul 2020 06:15:59 GMT
http_poller_metadata.host:
stelk2
@timestamp:
Jul 6, 2020 @ 10:16:00.435
xmldata.metric-data.metricName:
Server|Component:54|Custom Metrics|URL Monitor|icdupedixapp03-dashboard|Response Code
xmldata.metric-data.metricId:
829359
xmldata.metric-data.metricPath:
Application Infrastructure Performance|URL-Tier|Individual Nodes|URL-Tier_strproapd07|Custom Metrics|URL Monitor|icdupedixapp03-dashboard|Response Code

xmldata.metric-data.metricValues.metric-value:

{ "occurrences": "1", "current": "200", "min": "200", "max": "200", "startTimeInMillis": "1594015800000", "useRange": "true", "count": "1", "sum": "200", "value": "200", "standardDeviation": "0" }, { "occurrences": "1", "current": "200", "min": "200", "max": "200", "startTimeInMillis": "1594015860000", "useRange": "true", "count": "1", "sum": "200", "value": "200", "standardDeviation": "0" }, { "occurrences": "1", "current": "200", "min": "200", "max": "200", "startTimeInMillis": "1594015920000", "useRange": "true", "count": "1", "sum": "200", "value": "200", "standardDeviation": "0" }, { "occurrences": "1", "current": "200", "min": "200", "max": "200", "startTimeInMillis": "1594015980000", "useRange": "true", "count": "1", "sum": "200", "value": "200", "standardDeviation": "0" }, { "occurrences": "1", "current": "200", "min": "200", "max": "200", "startTimeInMillis": "1594016040000", "useRange": "true", "count": "1", "sum": "200", "value": "200", "standardDeviation": "0" }
xmldata.metric-data.frequency:

ONE_MIN
_id:

icdupedixapp03_dashboard
_type:
_doc

_index:

appdappstatus
_score:

1

Why does the data in ES show as a single value and how do i verify these values,
I only want to verify if at least one of it has 200.

Please help me out.

katara · July 6, 2020, 12:12pm

@Badger,
Adding to the above, Here's my full logstash configuration,

input {
http_poller {
urls => {
savvas => "https://appd**********************me-range-type=BEFORE_NOW&rollup=false&duration-in-mins=5"
brand_savvas => "https://appd*********************nse%20Code&time-range-type=BEFORE_NOW&rollup=false&duration-in-mins=5"
playbook_savvas => "https://appd***************type=BEFORE_NOW&rollup=false&duration-in-mins=5"
}
#cacert => "/home/sa_logstash/appd-controller.cer"
truststore => "/opt/appd/cacerts.jks"
truststore_password => "******"
request_timeout => 60
user => "readonly@customer1"
password => "***********"
metadata_target => "http_poller_metadata"
schedule => { cron => "* * * * *"}
codec => multiline {
            pattern => "<metric-datas>"
            negate => "true"
            what => "previous"
        }
}
}
filter
{
xml {
                source => "message"
                store_xml => "true"
                target => "xmldata"
                force_array => false
}
split
        {
        field => ["metric-value"]
        }

#if "METRIC DATA NOT FOUND" in [xmldata][metric-data][metricName] { drop{ } }
if [http_poller_metadata][name] == "savvas"{
        mutate { add_field => { "URL" => "https://vas.com"} }
      }
if [http_poller_metadata][name] == "brand_savvas"{
        mutate { add_field => { "URL" => "https://brand.student.com"} }
      }
if [http_poller_metadata][name] == "playbook_savvas"{
        mutate { add_field => { "URL" => "https://book.sas.com"} }
      }


mutate {
convert => {"[xmldata][metric-data][metricValues][metric-value][value]" => "integer" }
}


if [xmldata][metric-data][metricValues][metric-value][value][0] == 200
{
mutate { add_field => { "Status" => "Up"} }
}
else
{
mutate { add_field => { "Status" => "Down"} }
}


mutate {
    remove_field => ["message","[http_poller_metadata][request][url]",
"[http_poller_metadata][response_headers][x-xss-protection]",
"[http_poller_metadata][response_headers][set-cookie]",
"[http_poller_metadata][response_headers][x-content-type-options]",
"[http_poller_metadata][response_headers][transfer-encoding]",
"[http_poller_metadata][response_headers][content-type]",
"[http_poller_metadata][response_headers][x-frame-options]",
"[http_poller_metadata][response_message]",
"[http_poller_metadata][request.method]",
"[http_poller_metadata][code]",
"[http_poller_metadata][times_retried]",
"[http_poller_metadata][runtime_seconds]"]
}

}
output {
  elasticsearch {
    hosts => ["10.1.455.29:9200"]
    index => "appdappstatus"
        user => "elastic"
        password => "*******"
        action=>update
    document_id => "%{[http_poller_metadata][name]}"
    doc_as_upsert =>true
}
        stdout { codec => rubydebug }
}

Badger · July 6, 2020, 4:52pm

katara:

xmldata.metric-data.metricValues.metric-value:

{ "occurrences": "1", "current": "200", "min": "200", "max": "200", "startTimeInMillis": "1594015800000", "useRange": "true", "count": "1", "sum": "200", "value": "200", "standardDeviation": "0" }, { "occurrences": "1", "current": "200", "min": "200", "max": "200", "startTimeInMillis": "1594015860000", "useRange": "true", "count": "1", "sum": "200", "value": "200", "standardDeviation": "0" }, { "occurrences": "1", "current": "200", "min": "200", "max": "200", "startTimeInMillis": "1594015920000", "useRange": "true", "count": "1", "sum": "200", "value": "200", "standardDeviation": "0" }, { "occurrences": "1", "current": "200", "min": "200", "max": "200", "startTimeInMillis": "1594015980000", "useRange": "true", "count": "1", "sum": "200", "value": "200", "standardDeviation": "0" }, { "occurrences": "1", "current": "200", "min": "200", "max": "200", "startTimeInMillis": "1594016040000", "useRange": "true", "count": "1", "sum": "200", "value": "200", "standardDeviation": "0" }

That appears to be an array. Does it work if you convert/test [xmldata][metric-data][metricValues][metric-value][value][0] ?

Do you just want to test the first entry or test whether any are equal to 200 or whether all are equal to 200?

katara · July 7, 2020, 3:49am

@badger, I want to test if there are any value equal to 200.

[xmldata][metric-data][metricValues][metric-value][value][0] did not work, unfortunately.
also tried,
[xmldata][metric-data][metricValues][metric-value][0][value]

and
if "200" in [xmldata][metric-data][metricValues][metric-value][value]
None of the above works.

Badger · July 7, 2020, 3:09pm

That's the one I would have expected to work.

katara · July 7, 2020, 6:27pm

@Badger, tried that out, but no go.

Is there a way to split it again and not have it as an array?

Badger · July 7, 2020, 7:28pm

If you want to make it multiple events you can use a split filter.

katara · July 8, 2020, 3:59am

@Badger,
I'm already splitting the XML data.
Can I use another json parse and split filter?

katara · July 8, 2020, 8:24am

Ok i found a halfway of a solution i guess, @Badger

Here's what works (not always) ,

if "200" in [xmldata][metric-data][metricValues][metric-value][0][value] or
 "200" in [xmldata][metric-data][metricValues][metric-value][1][value] or
 "200" in [xmldata][metric-data][metricValues][metric-value][2][value] or
 "200" in [xmldata][metric-data][metricValues][metric-value][3][value] or
 "200" in [xmldata][metric-data][metricValues][metric-value][4][value]
{
mutate { add_field => { "Status" => "Up"} }
}
else
{
mutate { add_field => { "Status" => "Down"} }
}

And this seems to validate at points and sometime it doesnt,

This is how it looks when it reads the if and shows a status "UP"

And here's when it shows "DOWN" even if value - 200

I dont understand why it acts this way without any reason,
Am i missing something here?

Please help me out!

Badger · July 8, 2020, 2:31pm

You do not need a json filter, just a split.

katara · July 8, 2020, 2:44pm

@badger,
Now that I'm able to validate each of the values in my above answer, any reason it keeps fluctuations to work?
I don't see any reason why at the first minute for the same input it works and it doesn't after a couple of minutes and works again.

Badger · July 8, 2020, 2:58pm

I can see no reason for that either.

katara · July 10, 2020, 10:59am

@badger,
Okay so i tried splitting the rows,
Here's the json on Es on where the status is down:

{

* "_index": "appdappstatus",
* "_type": "_doc",
* "_id": "worktext",
* "_score": 1,
* "_source": {
  * "@timestamp": "2020-07-10T10:54:00.528Z",
  * "@version": "1",
  * "http_poller_metadata": {
    * "host": "strp2.com",
    * "response_headers": {
      * "date": "Fri, 10 Jul 2020 10:54:00 GMT"},
    * "name": "worktext",
    * "request": {
      * "method": "get"}},
  * "Status": "DOWN",
  * "tags": [
    * "_split_type_failure"],
  * "URL": ["https://worktext.com"](https://worktext.com/),
  * "xmldata": {
    * "metric-data": {
      * "metricName": "Server|Component:54|Custom Metrics|URL Monitor|worktext.com|Response Code",
      * "metricId": "606843",
      * "metricPath": "Application Infrastructure Performance|URL-Tier|Individual Nodes|URL-Tier_str07|Custom Metrics|URL Monitor|worktext.com|Response Code",
      * "frequency": "ONE_MIN",
      * "metricValues": {
        * "metric-value": {
          * "occurrences": "1",
          * "current": "200",
          * "min": "200",
          * "max": "200",
          * "useRange": "true",
          * "startTimeInMillis": "1594378320000",
          * "count": "1",
          * "sum": "200",
          * "value": "200",
          * "standardDeviation": "0"}}},
    * "metric-datas": {
      * "metric-data": {
        * "metricName": "METRIC DATA NOT FOUND",
        * "metricId": "733477",
        * "metricPath": "Application Infrastructure Performance|URL-Tier|Individual Nodes|URL-Tier_strp07|Custom Metrics|URL Monitor|Response Code",
        * "frequency": "ONE_MIN"}},
    * "occurrences": "1",
    * "current": "200",
    * "min": "200",
    * "max": "200",
    * "startTimeInMillis": "1594374300000",
    * "useRange": "true",
    * "count": "1",
    * "sum": "200",
    * "value": "200",
    * "standardDeviation": "0"}}

}

And Heres the one with UP status, done right:

{

* "_index": "appdappstatus",
* "_type": "_doc",
* "_id": "ras080",
* "_score": 1,
* "_source": {
  * "@timestamp": "2020-07-10T10:48:00.553Z",
  * "@version": "1",
  * "http_poller_metadata": {
    * "host": "strom",
    * "response_headers": {
      * "date": "Fri, 10 Jul 2020 10:48:00 GMT"},
    * "name": "ras080",
    * "request": {
      * "method": "get"}},
  * "Status": "UP",
  * "tags": [
    * "multiline"
,    * "_split_type_failure"],
  * "xmldata": {
    * "metric-data": {
      * "metricName": "Server|Component:54|Custom Metrics|URL Monitor|rascom-8080|Response Code",
      * "metricPath": "Application Infrastructure Performance|URL-Tier|Individual Nodes|URL-Tier_spd07|Custom Metrics|URL Monitor|ras080|Response Code",
      * "metricId": "829383",
      * "frequency": "ONE_MIN",
      * "metricValues": {
        * "metric-value": {
          * "occurrences": "1",
          * "useRange": "true",
          * "startTimeInMillis": "1594377960000",
          * "max": "200",
          * "min": "200",
          * "count": "1",
          * "value": "200",
          * "current": "200",
          * "sum": "200",
          * "standardDeviation": "0"},
        * "occurrences": "1",
        * "min": [
          * "200"
,          * "200"],
        * "max": "200",
        * "startTimeInMillis": "1594370160000",
        * "useRange": "true",
        * "value": [
          * "200"
,          * "200"]},
      * "sum": "200",
      * "value": "200",
      * "standardDeviation": "0"},
    * "count": "1",
    * "standardDeviation": "0"},
  * "URL": ["http://1080/"](http://10.080/)}

}

I dont know if there are any valid differences to be considered,

filter
{
xml {
                source => "message"
                store_xml => "true"
                target => "xmldata"
                force_array => false
}
split
        {
        field => ["metric-value"]
        }

split
        {
        field => ["[xmldata][metric-data][metricValues][metric-value]"]
        }
if  [xmldata][metric-data][metricValues][metric-value][value] == "200" 
{
 mutate { add_field => { "Status" => "UP"} }
}
else
{
 mutate { add_field => { "Status" => "DOWN"} }
}
}

@Badger, Do you see a potential cause in these?

Badger · July 10, 2020, 5:16pm

No, I do not understand why that would happen.

system · August 7, 2020, 5:16pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filter if condition in Logstash Logstash	8	19412	July 24, 2019
Mutate -> convert filter is not working as expected Logstash	7	1433	September 20, 2018
HowTo: Integer comparison in logstash filter Logstash	3	4015	May 8, 2019
Filter add_field Logstash	2	306	February 18, 2019
Trouble with filter conditional logic Logstash	6	272	January 21, 2023

Logstash if condition skipped

Related topics