# Moving from Logstash to Filebeat

**URL:** https://discuss.elastic.co/t/moving-from-logstash-to-filebeat/207460
**Category:** Beats
**Tags:** filebeat
**Created:** [November 12, 2019, 8:09am UTC](https://discuss.elastic.co/t/moving-from-logstash-to-filebeat/207460 "2019-11-12T08:09:55Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![yavidor](https://avatars.discourse-cdn.com/v4/letter/y/2acd7d/32.png) [@yavidor](https://discuss.elastic.co/u/yavidor)
#### Post date: [November 12, 2019, 8:09am UTC](https://discuss.elastic.co/t/moving-from-logstash-to-filebeat/207460/1 "2019-11-12T08:09:55Z")

</div>

Hi.  
were Moving from logstash to filebeat as a part of our transition to containers.

our old logstash configuration was:

```
input {
  file {
    path => "/servicename/_logs/servicename.log"
    codec => multiline {
      pattern => "(^[a-zA-Z.]+(?:Error|Exception).+)|(^\s+at .+)|(^\s+... \d+ more)|(^\t+)|(^\s*Caused by:.+)"
      what => "previous"
    }
  }
}

filter {
  if "multiline" not in [tags]{
  json {
    source => "message"
    remove_field => ["[request][body]","[response][body][response][items]"]
  }
}
else {
  grok {
    pattern_definitions => { APPJSON => "{.*}" }
    match => { "message" => "%{APPJSON:appjson} %{GREEDYDATA:stack_trace}"}
    remove_field => ["message"]
  }
  json {
    source => "appjson"
    remove_field => ["appjson"]
  }
}
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logstash-servicename-%{+YYYY.MM.dd}"
    document_type => "logs"
  }
}

```

To overcome the json issue we used the decode json:

```
  - decode_json_fields:
      fields: ["log"]
      target: ""

```

which worked fine.

we now have a problem with the java exceptions that are not digested well with elasticsearch causing many events to not index and getting these arrors in filebeat:

> 2019-11-12T07:57:29.815Z ERROR pipeline/output.go:92 Failed to publish events: temporary bulk send failure
> 
> 2019-11-12T07:57:29.826Z INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.3.2
> 
> 2019-11-12T07:57:29.863Z INFO template/load.go:73 Template already exists and will not be overwritten.
> 
> 2019-11-12T07:57:30.552Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s

and these wierd delays:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/0/100b5622c10b5a325094f9f9bd40e43c961a4832.png)

we need a way to handle these like we did before. we've tried multiline processing:

```
multiline:
          pattern: (^[a-zA-Z.]+(?:Error|Exception).+)|(^\s+at .+)|(^\s+... \d+ more)|(^\t+)|(^\s*Caused by:.+)
          negate: false
          match: after

```

but without success. (is there a need for ingest node here? if so what will be its configuration?)

any help will be much appreciated!

---

<div class="post-metadata">

### Author: ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)
#### Post date: [November 13, 2019, 2:59pm UTC](https://discuss.elastic.co/t/moving-from-logstash-to-filebeat/207460/2 "2019-11-13T14:59:48Z")

</div>

Hi @yavidor,

You cannot do multiline after using `decode_json_fields`. Perhaps you can try using the `json` parameters in the log input? See [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-config-json](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-config-json) for more details.

Best regards

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 11, 2019, 2:59pm UTC](https://discuss.elastic.co/t/moving-from-logstash-to-filebeat/207460/3 "2019-12-11T14:59:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.