Grok expression with if for Elasticseach

Elastic Stack Elasticsearch
1

Hello,

I would like to know if there is a possibility to make the if condition within a grok for Logstash, but for Elasticsearch ? I don't use Logstash but Filebeat

Regards.

2

I am not sure I understand? Do you want to have several grok expressions? If so, you can specify several grok processors and ignore the failures, or use the conditional if field for a grok processor in order to be run.

3

That right. So I could do something like that ? :

"Feb 17 08:42:54 ig1-edge-dc3-01_re0 rpd[1524]: bgp_send: sending 19 bytes to x.x.x.xInternal AS x) blocked (no spooling requested): Resource temporarily unavailable"

if bgp.state = bgp_send :
"%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:host.hostname}_%{DATA:host.re} %{DATA:process.name}(?:\\[%{POSINT:process.pid:long}\\])?: %{DATA:bgp.state}: %{DATA:bgp.sending} %{IP:client.ip} \\(%{DATA:bgp.as}\\) %{DATA:bgp.response}: %{GREEDYMULTILINE:system.syslog.message}"

and

"Feb 7 00:23:34 ig1-edge-dc3-01_re0 rpd[1524]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 158.58.176.35 (External AS 200271) changed state from EstabSync to Established (event RsyncAck) (instance master)"

if bgp.state = RPD_BGP_NEIGHBOR_STATE_CHANGED
"%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:host.hostname}_%{DATA:host.re} %{DATA:process.name}(?:\\[%{POSINT:process.pid:long}\\])?: %{DATA:bgp.state}: %{DATA:bgp.protocol} %{IP:client.ip} \\(%{DATA:bgp.as}\\) %{GREEDYMULTILINE:system.syslog.message}"

I don't know what is the right syntax.

4

See https://www.elastic.co/guide/en/elasticsearch/reference/current/conditionals-with-multiple-pipelines.html

but be aware, that the splitting has not yet happened, as the condition is executed before the processor itself.

5 
  {
      "pipeline": {
        "if": "bgp.state == 'bgp_send'",
        "name": "bgp_send_pipeline"
      }
    },

bgp.state is a field I created. If the condition is made before, then it won't know that it is.

How can I avoid this ?

6

you could just check if the message field includes a certain keyword in that case?

7

Oh, I did't thing about that, I will try.

Thanks.

8

Here is my pipeline :

PUT _ingest/pipeline/logs_pipeline
{
  "description": "A pipeline of pipelines for log files",
  "processors": [
    {
      "pipeline": {
        "if": "message == 'RPD_BGP_NEIGHBOR_STATE_CHANGED'",
        "name": "filebeat-7.6.2-system-syslog-pipeline"
      }
    },
    {
      "pipeline": {
        "if": "message == 'bgp_send'",
        "name": "bgp_send_pipeline"
      }
    }
  ]
}

but it returns :

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "compile error",
        "script_stack": [
          "message == 'RPD_BGP_NEIGH ...",
          "^---- HERE"
        ],
        "script": "message == 'RPD_BGP_NEIGHBOR_STATE_CHANGED'",
        "lang": "painless"
      }
    ],
    "type": "script_exception",
    "reason": "compile error",
    "script_stack": [
      "message == 'RPD_BGP_NEIGH ...",
      "^---- HERE"
    ],
    "script": "message == 'RPD_BGP_NEIGHBOR_STATE_CHANGED'",
    "lang": "painless",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Variable [message] is not defined."
    }
  },
  "status": 400
}

Why is that ? Is there a problem with the syntax, because the message field exists ?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.