# Grok with custom pattern works in debugger but not in pipline

**URL:** <https://discuss.elastic.co/t/grok-with-custom-pattern-works-in-debugger-but-not-in-pipline/348957>\
**Category:** Logstash\
**Created:** [December 9, 2023, 6:57am UTC](https://discuss.elastic.co/t/grok-with-custom-pattern-works-in-debugger-but-not-in-pipline/348957 "2023-12-09T06:57:01Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![helldunkel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/helldunkel/32/133097_2.png) [@helldunkel](https://discuss.elastic.co/u/helldunkel)\
**Post date:** [December 9, 2023, 6:57am UTC](https://discuss.elastic.co/t/grok-with-custom-pattern-works-in-debugger-but-not-in-pipline/348957/1 "2023-12-09T06:57:02Z")

</div>

Hi,

I´m have a lot of problems to get a dataset in elastic.

In Debugger it works.

Log

```auto
<30>2023:12:08-12:59:39 fw-swr-2 ulogd[32373]:

```

grock

```auto
.*>%{SOPHOS_TIMESTAMP:_tmp.timestamp} %{TEST:firewall.name}

```

custom pattern

```auto
SOPHOS_TIMESTAMP (?:%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND})
TEST [a-zA-Z0-9._-]+

```

output

```auto
{
  "_tmp": {
    "timestamp": "2023:12:08-12:59:39"
  },
  "firewall": {
    "name": "fw-swr-2"
  }
}

```

In elk Stack gui:

pattern

```auto
.*>%{SOPHOS_TIMESTAMP:_tmp.timestamp} %{TEST:firewall.name}

```

custom pattern:

```auto
{
  "SOPHOS_TIMESTAMP": "(?:%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND})",
  "TEST": "[a-zA-Z0-9._-]+"
}

```

No entry in Database.  
There is no other processor in the pipline.

Is there something wrong?  
Is there a log for the pipline where I can see what´s going wrong?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 9, 2023, 4:11pm UTC](https://discuss.elastic.co/t/grok-with-custom-pattern-works-in-debugger-but-not-in-pipline/348957/2 "2023-12-09T16:11:28Z")

</div>

Hi @helldunkel

Assuming you mean the Grok Debugger in Kibana Dev Tols

> [@helldunkel](#):
>
> ```auto
> {
> "SOPHOS_TIMESTAMP": "(?:%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND})",
> "TEST": "[a-zA-Z0-9._-]+"
> }
> 
> ```

Should just be this

> [@helldunkel](#):
>
> ```auto
> SOPHOS_TIMESTAMP (?:%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND})
> TEST [a-zA-Z0-9._-]+
> 
> ```

 ![Screenshot 2023-12-09 at 7.51.55 AM](https://us1.discourse-cdn.com/elastic/original/3X/a/3/a37b28afdf6aa839027bd7e13f5aa0f062aa2001.png)

> [@helldunkel](#):
>
> No entry in Database.  
> There is no other processor in the pipline.
> 
> Is there something wrong?  
> Is there a log for the pipline where I can see what´s going wrong?

You will have to provide more details

I used this log file

```auto
<30>2023:12:08-12:59:39 fw-swr-2 ulogd[32373]:
<30>2023:12:08-12:59:40 fw-swr-3 ulogd[32380]:
<30>2023:12:08-12:59:42 fw-abc-2 ulogd[32390]:
<30>2023:12:08-12:59:45 fw-xyz-2 ulogd[32300]:
<30>2023:12:08-12:59:57 fw-nnn-2 ulogd[32388]:

```

This logstash conf

```auto
input {
	file {
		path => "/Users/sbrown/workspace/sample-data/discuss/discuss-sophos.log"
		start_position => "beginning"
		sincedb_path => "/dev/null"
	}
}

filter {
	grok {
		match => { "message" => ".*>%{SOPHOS_TIMESTAMP:_tmp.timestamp} %{TEST:firewall.name}"}
		pattern_definitions => {
				"SOPHOS_TIMESTAMP" => "(?:%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND})"
				"TEST" => "[a-zA-Z0-9._-]+"
		}
	}
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
  }
	stdout{}
}

```

And the data loaded fine...

Output from logstash

```auto
{
        "@timestamp" => 2023-12-09T16:09:24.078651Z,
             "event" => {
        "original" => "<30>2023:12:08-12:59:42 fw-abc-2 ulogd[32390]:"
    },
              "host" => {
        "name" => "hyperion"
    },
               "log" => {
        "file" => {
            "path" => "/Users/sbrown/workspace/sample-data/discuss/discuss-sophos.log"
        }
    },
    "_tmp.timestamp" => "2023:12:08-12:59:42",
          "@version" => "1",
           "message" => "<30>2023:12:08-12:59:42 fw-abc-2 ulogd[32390]:",
     "firewall.name" => "fw-abc-2"
}
....

```

In Elastic

```auto
GET logs-*/_search

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-logs-generic-default-2023.12.09-000001",
        "_id": "8rRYT4wBpwIAo0SDQYd0",
        "_score": 1,
        "_ignored": [
          "_tmp.timestamp"
        ],
        "_source": {
          "@timestamp": "2023-12-09T16:10:33.056416Z",
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-sophos.log"
            }
          },
          "firewall.name": "fw-swr-2",
          "data_stream": {
            "namespace": "default",
            "type": "logs",
            "dataset": "generic"
          },
          "host": {
            "name": "hyperion"
          },
          "@version": "1",
          "_tmp.timestamp": "2023:12:08-12:59:39",
          "event": {
            "original": "<30>2023:12:08-12:59:39 fw-swr-2 ulogd[32373]:"
          },
          "message": "<30>2023:12:08-12:59:39 fw-swr-2 ulogd[32373]:"
        }
      },
      {
        "_index": ".ds-logs-generic-default-2023.12.09-000001",
        "_id": "8bRYT4wBpwIAo0SDQYd0",
        "_score": 1,
        "_ignored": [
          "_tmp.timestamp"
        ],
        "_source": {
          "@timestamp": "2023-12-09T16:10:33.057941Z",
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-sophos.log"
            }
          },
          "firewall.name": "fw-nnn-2",
          "data_stream": {
            "namespace": "default",
            "type": "logs",
            "dataset": "generic"
          },
          "host": {
            "name": "hyperion"
          },
          "@version": "1",
          "_tmp.timestamp": "2023:12:08-12:59:57",
          "event": {
            "original": "<30>2023:12:08-12:59:57 fw-nnn-2 ulogd[32388]:"
          },
          "message": "<30>2023:12:08-12:59:57 fw-nnn-2 ulogd[32388]:"
        }
      },
....

```

---

<div class="post-metadata">

**Author:** ![helldunkel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/helldunkel/32/133097_2.png) [@helldunkel](https://discuss.elastic.co/u/helldunkel)\
**Post date:** [December 11, 2023, 10:15am UTC](https://discuss.elastic.co/t/grok-with-custom-pattern-works-in-debugger-but-not-in-pipline/348957/3 "2023-12-11T10:15:06Z")

</div>

Hi,

OK.  
The setup ist a security Onioin distributed setup. Firewall sends syslog to receiver node, this sends it to manager node.

All logs from firewall are send to a custom pipline.

The pipline config in elastic-management-ingest piplines:

```auto
[
  {
    "grok": {
      "field": "message",
      "patterns": [
        ".*>%{SOPHOS_TIMESTAMP:_tmp.timestamp} %{TEST:firewall.name}"
      ],
      "pattern_definitions": {
        "SOPHOS_TIMESTAMP": "(?:%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND})",
        "TEST": "[a-zA-Z0-9._-]+"
      }
    }
  }
]

```

All tests in grock debugger worked.

My question is: is there a way to see what happens in the pipline? Why the pipline not worked.  
I have no way to start the debugging in the running system.

---

<div class="post-metadata">

**Author:** ![helldunkel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/helldunkel/32/133097_2.png) [@helldunkel](https://discuss.elastic.co/u/helldunkel)\
**Post date:** [December 11, 2023, 11:20am UTC](https://discuss.elastic.co/t/grok-with-custom-pattern-works-in-debugger-but-not-in-pipline/348957/4 "2023-12-11T11:20:46Z")

</div>

hi,

OK. it isn´t grok or custom pattern, it is the database entry:

%{TEST:firewall.name} -\> no  
%{TEST:test.test} -\> no  
%{TEST:test} -\> no  
%{TEST:host.name} -\> yes  
%{TEST:host.hostname} -\> yes

But why? Why I can not set a new datafild?

---

<div class="post-metadata">

**Author:** ![helldunkel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/helldunkel/32/133097_2.png) [@helldunkel](https://discuss.elastic.co/u/helldunkel)\
**Post date:** [December 11, 2023, 1:09pm UTC](https://discuss.elastic.co/t/grok-with-custom-pattern-works-in-debugger-but-not-in-pipline/348957/5 "2023-12-11T13:09:16Z")

</div>

And found.

A missconfig deep inside the Index management.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 11, 2023, 3:48pm UTC](https://discuss.elastic.co/t/grok-with-custom-pattern-works-in-debugger-but-not-in-pipline/348957/6 "2023-12-11T15:48:22Z")

</div>

Glad you got it working...

> [@helldunkel](#):
>
> My question is: is there a way to see what happens in the pipline? Why the pipline not worked.  
> I have no way to start the debugging in the running system.

Just for next time... you can use `verbose` to see more details

> **[Simulate pipeline API | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html#simulate-pipeline-api-query-params)**

> ### Query parameters
> 
> `verbose`  
> (Optional, Boolean) If `true`, the response includes output data for each processor in the executed pipeline.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 8, 2024, 3:48pm UTC](https://discuss.elastic.co/t/grok-with-custom-pattern-works-in-debugger-but-not-in-pipline/348957/7 "2024-01-08T15:48:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.