# Logstash pipeline error date

**URL:** <https://discuss.elastic.co/t/logstash-pipeline-error-date/164801>\
**Category:** Logstash\
**Created:** [January 18, 2019, 1:29pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801 "2019-01-18T13:29:55Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jozziej](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@Jozziej](https://discuss.elastic.co/u/Jozziej)\
**Post date:** [January 18, 2019, 1:29pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/1 "2019-01-18T13:29:55Z")

</div>

I'm trying to parse an error log, but my date filter keeps failing

example log entry:

Wed Nov 28 02:16:24.654 [MCSS:7497:0x7F17018E7700]: Warning: invalid cid (-1) in Msg::receive for unknown ip, with msg id (0x2b0300)

The date I want to put in @Timestamp  
and if that works i want the text in the this field [MCSS:7497:0x7F17018E7700]: to be filtered as useless

Below is my configuration

input {  
file {  
path =\> "C:/cygwin64/home/Logstashfiles/IAC/iqs.log"  
type =\> "iqs.log"  
start\_position =\> "beginning"  
}  
}  
filter {  
grok {  
match =\> { "message" =\> "%{TIMESTAMP:timestamp} %{GREEDYDATA:useless} %{GREEDYDATA:message}" }  
}  
date {  
match =\> ["timestamp" , "EEE MMM dd HH:mm:ss.SSS"] }  
}  
output {  
stdout {}  
}

Below is the error from the Logstash logs

[2019-01-18T14:27:56,740][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}  
[2019-01-18T14:27:57,019][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"#\<LogStash::FilterDelegator:0x1db0ddad @metric\_events\_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric\_events\_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric\_events\_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration\_in\_millis value:0, @id="c04f67adb828c4c42d8d6592d98b10597fb883c99f12c642fc64df949a440c61", @klass=LogStash::Filters::Grok, @metric\_events=#LogStash::Instrument::NamespacedMetric:0xbecbfda, @filter=\<LogStash::Filters::Grok match=\>{"message"=\>"%{TIMESTAMP:timestamp} %{GREEDYDATA:useless} %{GREEDYDATA:message}"}, id=\>"c04f67adb828c4c42d8d6592d98b10597fb883c99f12c642fc64df949a440c61", enable\_metric=\>true, periodic\_flush=\>false, patterns\_files\_glob=\>"\*", break\_on\_match=\>true, named\_captures\_only=\>true, keep\_empty\_captures=\>false, tag\_on\_failure=\>["\_grokparsefailure"], timeout\_millis=\>30000, tag\_on\_timeout=\>"\_groktimeout"\>\>", :error=\>"pattern %{TIMESTAMP:timestamp} not defined", :thread=\>"#\<Thread:0x2f64e4d1 run\>"}  
[2019-01-18T14:27:57,029][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<Grok::PatternError: pattern %{TIMESTAMP:timestamp} not defined\>, :backtrace=\>["C:/ELK/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in `block in compile'", "org/jruby/RubyKernel.java:1292:in`loop'", "C:/ELK/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in `compile'", "C:/ELK/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:281:in`block in register'", "org/jruby/RubyArray.java:1734:in `each'", "C:/ELK/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:275:in`block in register'", "org/jruby/RubyHash.java:1343:in `each'", "C:/ELK/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:270:in`register'", "C:/ELK/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:242:in `register_plugin'", "C:/ELK/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:253:in`block in register\_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "C:/ELK/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:253:in`register\_plugins'", "C:/ELK/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:595:in `maybe_setup_out_plugins'", "C:/ELK/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:263:in`start\_workers'", "C:/ELK/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:200:in `run'", "C:/ELK/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:160:in`block in start'"], :thread=\>"#\<Thread:0x2f64e4d1 run\>"}  
[2019-01-18T14:27:57,058][ERROR][logstash.agent] Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: PipelineAction::Create, action\_result: false", :backtrace=\>nil}  
[2019-01-18T14:27:57,519][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 18, 2019, 3:11pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/2 "2019-01-18T15:11:07Z")

</div>

> [@Jozziej](#):
>
> :exception=\>#\<Grok::PatternError: pattern %{TIMESTAMP:timestamp} not defined

There is no pattern called TIMESTAMP unless you define it.

Personally I would use dissect instead of grok

```
dissect { mapping => { "message" => "%{ts} %{+ts} %{+ts} %{+ts} [%{}]: %{msg}" } }

```

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 18, 2019, 3:18pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/3 "2019-01-18T15:18:54Z")

</div>

It is not the date filter that is failing, it is the grok filter.

> [2019-01-18T14:27:57,029][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<Grok::PatternError: pattern %{TIMESTAMP:timestamp} not defined\>

Your date does not match any predefined formats

This one comes close:  
`DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}`  
but it will not work out of the box.

You will have to define you own pattern.

The grok filter has a setting `pattern_definitions`. NOTE in the example below I also changed the first GREEDYDATA to NOTSPACE as you are wanting to skip all until the next space char. It is not advisable to use more that one GREEDYDATA.

```auto
grok {
  pattern_definitions => { "DATESTAMP_IQS" => "%{DAY} %{MONTH} %{MONTHDAY} %{TIME}" }
  match => { "message" => "%{DATESTAMP_IQS:timestamp} %{NOTSPACE:useless} %{GREEDYDATA:message}" }

```

---

<div class="post-metadata">

**Author:** ![Jozziej](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@Jozziej](https://discuss.elastic.co/u/Jozziej)\
**Post date:** [January 18, 2019, 3:43pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/4 "2019-01-18T15:43:55Z")

</div>

Hi guyboertje thanks for your reply

The parsing still doesn't work correctly

this is the logstash-plain.log

[2019-01-18T16:34:37,288][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2019-01-18T16:34:37,312][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.5.4"}  
[2019-01-18T16:34:45,169][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}  
[2019-01-18T16:34:46,060][INFO][logstash.inputs.file] No sincedb\_path set, generating one based on the "path" setting {:sincedb\_path=\>"C:/ELK/logstash-6.5.4/data/plugins/inputs/file/.sincedb\_40647c5112ef4dde3c411f57a7d3d556", :path=\>["C:/cygwin64/home/Logstashfiles/IAC/iqs.log"]}  
[2019-01-18T16:34:46,119][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\<Thread:0x23a06ca6 run\>"}  
[2019-01-18T16:34:46,223][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>}  
[2019-01-18T16:34:46,257][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections  
[2019-01-18T16:34:46,806][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/5/d5cb303afce2be311b8d789bd5b49cb23e3337ee.png)

See the image as result

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 18, 2019, 4:12pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/5 "2019-01-18T16:12:54Z")

</div>

There are two spaces after the date.

```
match => { "message" => "%{DATESTAMP_IQS:timestamp} %{NOTSPACE:useless} %{GREEDYDATA:message}" }

```

This is why it is helpful to use \</\> (in the toolbar above the edit panel) to blockquote things like logs, data, and configurations.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 18, 2019, 4:14pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/6 "2019-01-18T16:14:35Z")

</div>

Try the dissect option that @badger suggested.

Alternatively you can work systematically from left to right by extending the custom pattern.

1. 

```auto
grok {
  # pattern_definitions => { "DATESTAMP_IQS" => "%{DAY} %{MONTH} %{MONTHDAY} %{TIME}" }
  pattern_definitions => { "DATESTAMP_IQS" => "%{DAY}" }
  match => { "message" => "%{DATESTAMP_IQS:timestamp} %{GREEDYDATA:msg}" }
}

```

1. 

```auto
grok {
  # pattern_definitions => { "DATESTAMP_IQS" => "%{DAY} %{MONTH} %{MONTHDAY} %{TIME}" }
  pattern_definitions => { "DATESTAMP_IQS" => "%{DAY} %{MONTH}" }
  match => { "message" => "%{DATESTAMP_IQS:timestamp} %{GREEDYDATA:msg}" }
}

```

and so on.

---

<div class="post-metadata">

**Author:** ![Jozziej](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@Jozziej](https://discuss.elastic.co/u/Jozziej)\
**Post date:** [January 18, 2019, 4:15pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/7 "2019-01-18T16:15:17Z")

</div>

Hi Badger,

I used dissect now

dissect { mapping =\> { "message" =\> "%{timestamp} %{+timestamp} %{+timestamp} %{+timestamp} [%{}]: %{message}" } }

this works correctly for this error log as far as I can see

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 18, 2019, 4:15pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/8 "2019-01-18T16:15:56Z")

</div>

The sample line posted in the original post has only one space 😖

---

<div class="post-metadata">

**Author:** ![Jozziej](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@Jozziej](https://discuss.elastic.co/u/Jozziej)\
**Post date:** [January 18, 2019, 4:18pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/9 "2019-01-18T16:18:42Z")

</div>

Thanks you all!  
It appears the Dissect works good enough:)

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 18, 2019, 4:22pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/10 "2019-01-18T16:22:51Z")

</div>

Either way, whether you use grok or dissect, I encourage you to use the generator input.  
You can add multiple variant lines in the `lines` array and retest changes really easily until your grok/dissect, date and other filters all work for each variant.

e.g.

```auto
input {
  generator {
    lines => [
      'Wed Nov 28 02:16:24.654 [MCSS:7497:0x7F17018E7700]: Warning: invalid cid (-1) in Msg::receive for unknown ip, with msg id (0x2b0300)',
      'Wed Nov 28 02:16:24.654 [MCSS:7497:0x7F17018E7700]: Warning: invalid cid (-1) in Msg::receive for unknown ip, with msg id (0x2b0300)'
    ]
    count => 1
  }
}

filter {
  grok {
    pattern_definitions => { "DATESTAMP_IQS" => "%{DAY} %{MONTH} %{MONTHDAY} %{TIME}" }
    match => {
      "message" => [
        '%{DATESTAMP_IQS:timestamp} +%{NOTSPACE:discard} %{GREEDYDATA:msg}'
      ]
    }
    break_on_match => true
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

```

Gives:

```auto
{
           "msg" => "Warning: invalid cid (-1) in Msg::receive for unknown ip, with msg id (0x2b0300)",
       "discard" => "[MCSS:7497:0x7F17018E7700]:",
      "sequence" => 0,
       "message" => "Wed Nov 28 02:16:24.654 [MCSS:7497:0x7F17018E7700]: Warning: invalid cid (-1) in Msg::receive for unknown ip, with msg id (0x2b0300)",
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
    "@timestamp" => 2019-01-18T16:21:53.645Z,
     "timestamp" => "Wed Nov 28 02:16:24.654"
}
{
           "msg" => "Warning: invalid cid (-1) in Msg::receive for unknown ip, with msg id (0x2b0300)",
       "discard" => "[MCSS:7497:0x7F17018E7700]:",
      "sequence" => 0,
       "message" => "Wed Nov 28 02:16:24.654 [MCSS:7497:0x7F17018E7700]: Warning: invalid cid (-1) in Msg::receive for unknown ip, with msg id (0x2b0300)",
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
    "@timestamp" => 2019-01-18T16:21:53.629Z,
     "timestamp" => "Wed Nov 28 02:16:24.654"
}

```

---

<div class="post-metadata">

**Author:** ![Jozziej](https://avatars.discourse-cdn.com/v4/letter/j/c57346/32.png) [@Jozziej](https://discuss.elastic.co/u/Jozziej)\
**Post date:** [January 18, 2019, 4:42pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/11 "2019-01-18T16:42:24Z")

</div>

Ah great the generator input helps speed up parsing my logs alot!

Do you recommend to put all my log files in the config as one pipeline?  
will be around 15 logfiles which do not all have the same format.

Also some logfiles i tail with .log\* because they store the older logs with .log.1 .log.2 .log.3 and so on

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 15, 2019, 4:42pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801/12 "2019-02-15T16:42:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.