# Logstash: configure multiline configuration for specific type of logs only

**URL:** <https://discuss.elastic.co/t/logstash-configure-multiline-configuration-for-specific-type-of-logs-only/364700>\
**Category:** Logstash\
**Created:** [August 9, 2024, 4:09pm UTC](https://discuss.elastic.co/t/logstash-configure-multiline-configuration-for-specific-type-of-logs-only/364700 "2024-08-09T16:09:47Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![pix9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pix9/32/128388_2.png) [@pix9](https://discuss.elastic.co/u/pix9)\
**Post date:** [August 9, 2024, 4:09pm UTC](https://discuss.elastic.co/t/logstash-configure-multiline-configuration-for-specific-type-of-logs-only/364700/1 "2024-08-09T16:09:48Z")

</div>

Hey Folks,

We've an existing Logstash configuration where we use fields "logs\_type" to separate the different types of logs and parse them with their respective grok patterns into multiple respective indexes.

Now we are trying to add a configuration of new multiline log into this.

so far input configuration had been simple just accepting steam from Filebeat port,

New change which needs to be achieved is  
if "logs\_type" of input steam is let's say "foo" then run multiline plugin and normalise multiple lines into single lines as per pattern and forward to grok for splitting it into proper document with respective fields.

given below is the configuration we've tried with input.

```auto
input {
        beats {
        port => 5044
        if [fields][log_type] == "foo" {

                codec => multiline {
                        pattern => "\#\s+Time:\s+%{BASE10NUM:ts}\s+%{TIME:ts2}"
                        negate => true
                        what => "previous"
                }
        }

        }
}
~

```

with above configuration we are getting following error when we try to start Logstash.

```auto
[2024-08-09T11:43:13,376][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"=>\" at line 4, column 5 (byte 36) after input {\n\tbeats {\n\tport => 5044\n\tif ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:949:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}

```

Can someone please guide us in fixing configuration or approach or both?

stack:  
Elasticsearch ver: 8.15.0  
Logstash ver: 8.15.0  
Filebear ver: 8.15.0

Regards

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 9, 2024, 4:29pm UTC](https://discuss.elastic.co/t/logstash-configure-multiline-configuration-for-specific-type-of-logs-only/364700/2 "2024-08-09T16:29:10Z")

</div>

There are a couple of things that are wrong in your configuration.

First, you cannot have a conditional inside an input as you did, and you also cannot have conditionals in the `input` block.

You can only use conditionals in the `filter` and `output` block.

Another thing is that you are using the beats input, the multiline configuration needs to be done on beats side, not logstash.

This is mentioned [here](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html#_description_190) in the multiline codec input.

So, you will need to move your multline configuration to Filebeat following [this documentation](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html).

---

<div class="post-metadata">

**Author:** ![pix9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pix9/32/128388_2.png) [@pix9](https://discuss.elastic.co/u/pix9)\
**Post date:** [August 9, 2024, 5:35pm UTC](https://discuss.elastic.co/t/logstash-configure-multiline-configuration-for-specific-type-of-logs-only/364700/3 "2024-08-09T17:35:02Z")

</div>

> [@leandrojmp](#):
>
> Another thing is that you are using the beats input, the multiline configuration needs to be done on beats side, not logst

Ok done as suggested above by.  
here is my filebeats.yml

```auto
filebeat.inputs:
- type: filestream
  paths:
    - /var/log/mysql_slow.log
  multiline.type: pattern
  multiline.pattern: \#\s+Time:\s+%{BASE10NUM:ts}\s+%{TIME:ts2}
  multiline.negate: true
  multiline.match: after
  fields:
    log_type: mysql-slowlog
output.logstash:
  hosts: ["localhost:5044"]
  loadbalance: true
  index: mysql-slow

```

and here my input.conf for logstash.

```auto
input {
  beats {
    port => 5044
  }
}

```

However when i go ahead with this configuration elasticsearch receives 42 documents insted of 2

One if my doubt is does filebeat configurations supports GROK patterns for pattern matching? will be trying to change it with regular regex in config while I post this for you.

---

<div class="post-metadata">

**Author:** ![pix9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pix9/32/128388_2.png) [@pix9](https://discuss.elastic.co/u/pix9)\
**Post date:** [August 9, 2024, 5:48pm UTC](https://discuss.elastic.co/t/logstash-configure-multiline-configuration-for-specific-type-of-logs-only/364700/4 "2024-08-09T17:48:03Z")

</div>

tried changing the multiline pattern in filebeat

From

```auto
multiline.pattern: \#\s+Time:\s+%{BASE10NUM:ts}\s+%{TIME:ts2}

```

to

```auto
multiline.pattern: "\\#\\s+Time:\\s+[0-9]{6}\\s+[0-9].?\\:[0-9].?\\:[0-9].?"

```

Still getting 42 documents on ES instead of just 2

Wondering what is going wrong with my configuration.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 9, 2024, 5:48pm UTC](https://discuss.elastic.co/t/logstash-configure-multiline-configuration-for-specific-type-of-logs-only/364700/5 "2024-08-09T17:48:49Z")

</div>

> [@pix9](#):
>
> ```auto
> multiline.type: pattern
> multiline.pattern: \#\s+Time:\s+%{BASE10NUM:ts}\s+%{TIME:ts2}
> multiline.negate: true
> multiline.match: after
> 
> ```

This configuration is wrong, please check the documentation for the right format.

You are using the `filestream` input, your configuration needs to be in this format:

```auto
parsers:
- multiline:
    type: pattern
    pattern: 'YOUR-PATTERN'
    negate: true
    match: after

```

> [@pix9](#):
>
> One if my doubt is does filebeat configurations supports GROK patterns for pattern matching?

No, it does not, you need to have a regex pattern that will match the start of your multiline line, the pattern you are using will not work.

---

<div class="post-metadata">

**Author:** ![pix9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pix9/32/128388_2.png) [@pix9](https://discuss.elastic.co/u/pix9)\
**Post date:** [August 9, 2024, 6:26pm UTC](https://discuss.elastic.co/t/logstash-configure-multiline-configuration-for-specific-type-of-logs-only/364700/6 "2024-08-09T18:26:29Z")

</div>

Here is my new config file for filebeat.

```auto
filebeat.inputs:
- type: filestream
  paths:
    - /var/log/mysql_slow.log
parsers:
- multiline:
   type: pattern
   pattern: '#.*Time:.*[0-9][0-9][0-9][0-9][0-9][0-9].*'
   negate: true
   match: after
  fields:
    log_type: mysql-slowlog

```

Tried multiple regex and escaping sequence for pattern matching, also tried to turn negate flag false, however  
nothing changed still receiving 42 documents instead of 2 on ES.

Any thing which needs to be added on Logstash config to identify such filesteam?

---

<div class="post-metadata">

**Author:** ![pix9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pix9/32/128388_2.png) [@pix9](https://discuss.elastic.co/u/pix9)\
**Post date:** [August 9, 2024, 6:32pm UTC](https://discuss.elastic.co/t/logstash-configure-multiline-configuration-for-specific-type-of-logs-only/364700/7 "2024-08-09T18:32:44Z")

</div>

Finally found a fix.  
after correcting indentation for filebeat file.

```auto
filebeat.inputs:
- type: filestream
  paths:
    - /var/log/mysql_slow.log
  fields:
    log_type: mysql-slowlog

  parsers:
  - multiline:
     type: pattern
     pattern: '#.*Time:.*[0-9][0-9][0-9][0-9][0-9][0-9].*'
     negate: true
     match: after

```

I was expecting filebeat to crash if yaml indentation is incorrect, strangely however when I restarted the service number of times, it went through without throwing any errors.

Thanks for you help @leandrojmp you helped me understanding this

Now I will take my time playing around different regex and other optimisation.