Hello elastic - Multiple Pipelines

[jie (jie.xie)]

DELETE index
PUT index/_doc/1
{
  "foo": "bar"
}
GET index/_search
{
  "query": {
    "match": {
      "foo": "bar"
    }
  }
}

[warkolm (Mark Walkom)]

G'Day. What can we help with?

[jie (jie.xie)]

Hello warkolm

Can multiple pipelines be configured for one log message in Filebeat. yml

[stephenb (Stephen Brown)]

No only a single pipeline per input type.

Here are some thoughts on a solution.

How to configure multiple ingest pipelines on filebeat to load data in elastic cloud

Here is a method / suggestion The advantage of doing it this way having a top level common pipeline that then directs and sorts to the correct specific pipeline is that you can control this outside of beat so if you change something you don't need to change the beats config. You can also build modular pipelines so they can be reused. You can also define a pipeline per input ... See here...

[jie (jie.xie)]

hi
How do I write multiple Groks to parse a log file

[stephenb (Stephen Brown)]

You should really read this Page / Doc entirely Here

See the Docs Here

You can put multiple Groks in a single grok processor

Example

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "...",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes:int} %{NUMBER:duration:double}",
            "%{IP:client} - %{WORD:level} - %{NUMBER:bytes:int} %{DATA:message_detail}"
          ]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "55.3.244.1 GET /index.html 15824 0.043"
      }
    },
    {
      "_source": {
        "message": "55.3.244.1 - WARN - 932 - Can not connect to server "
      }
    }
  ]
}

result

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_id" : "_id",
        "_source" : {
          "duration" : 0.043,
          "request" : "/index.html",
          "method" : "GET",
          "bytes" : 15824,
          "client" : "55.3.244.1",
          "message" : "55.3.244.1 GET /index.html 15824 0.043"
        },
        "_ingest" : {
          "timestamp" : "2022-03-02T02:28:54.1600216Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "_index",
        "_id" : "_id",
        "_source" : {
          "level" : "WARN",
          "bytes" : 932,
          "client" : "55.3.244.1",
          "message" : "55.3.244.1 - WARN - 932 - Can not connect to server ",
          "message_detail" : ""
        },
        "_ingest" : {
          "timestamp" : "2022-03-02T02:28:54.1600448Z"
        }
      }
    }
  ]
}

Or you can create something like this if you have a tag or field you can use see here

PUT _ingest/pipeline/log-type1
{
  "description": "...",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes:int} %{NUMBER:duration:double}"
        ]
      }
    }
  ]
}


PUT _ingest/pipeline/log-type2
{
  "description": "...",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{IP:client} - %{WORD:level} - %{NUMBER:bytes:int} %{DATA:message_detail}"
        ]
      }
    }
  ]
}


PUT _ingest/pipeline/top-level-log-pipeline
{
  "description": "...",
  "processors": [
    {
      "pipeline": {
        "description": "If logtype web-log",
        "if": "ctx.type == 'web-log'",
        "name": "log-type1"
      }
    },
    {
      "pipeline": {
        "description": "If logtype app-log",
        "if": "ctx.type == 'app-log'",
        "name": "log-type2"
      }
    }
  ]
}

POST _ingest/pipeline/top-level-log-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "type" : "web-log",
        "message": "55.3.244.1 GET /index.html 15824 0.043"
      }
    },
    {
      "_source": {
        "type" : "app-log",
        "message": "55.3.244.1 - WARN - 932 - Can not connect to server "
      }
    }
  ]
}

results

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_id" : "_id",
        "_source" : {
          "duration" : 0.043,
          "request" : "/index.html",
          "method" : "GET",
          "bytes" : 15824,
          "client" : "55.3.244.1",
          "type" : "web-log",
          "message" : "55.3.244.1 GET /index.html 15824 0.043"
        },
        "_ingest" : {
          "timestamp" : "2022-03-02T02:55:50.6633169Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "_index",
        "_id" : "_id",
        "_source" : {
          "level" : "WARN",
          "bytes" : 932,
          "client" : "55.3.244.1",
          "type" : "app-log",
          "message" : "55.3.244.1 - WARN - 932 - Can not connect to server ",
          "message_detail" : ""
        },
        "_ingest" : {
          "timestamp" : "2022-03-02T02:55:50.6633322Z"
        }
      }
    }
  ]
}

[jie (jie.xie)]

Thank you very much

[jie (jie.xie)]

hi
Sorry to bother you
The problem

[stephenb (Stephen Brown)]

No easy solution you are going to need to write a grok for each pattern...and put them all the grok processor patterns array like in the first example above.

You should define the most common first and then next most common and continue to iterate

[jie (jie.xie)]

Ok, thanks for your help