# Filbeat tail\_filter Logs 5.4 Version

**URL:** <https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 26, 2017, 1:04pm UTC](https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821 "2017-06-26T13:04:42Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pandiyan\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pandiyan_m/32/19404_2.png) [@Pandiyan\_M](https://discuss.elastic.co/u/Pandiyan_M)\
**Post date:** [June 26, 2017, 1:04pm UTC](https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821/1 "2017-06-26T13:04:42Z")

</div>

Hi,

I have installed filebeat on ubuntu and ELK master is also ubuntu. By following

> **[How to Install Elastic Stack on Ubuntu 16.04](https://www.howtoforge.com/tutorial/how-to-install-elastic-stack-on-ubuntu-16-04/)**
>
> In this tutorial, I will show you how to install and configure Elastic Stack on a single Ubuntu 16.04 server for monitoring server logs and how to ins...

Where in my ELK server

#input.conf

```
input {
  beats {
    host => "0.0.0.0"
    port => 5443
    type => weblog
    #ssl => true
    #ssl_certificate => "/etc/logstash/logstash.crt"
    #ssl_key => "/etc/logstash/logstash.key"
  }
}

```

#filter.conf

```
filter{
  if [type] == "weblog" {
    grok {
      match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} (?<logger>(?:[a-zA-Z0-9]+\.)*[-A-Za-z0-9$]+) %{GREEDYDATA:message}"]
      overwrite => ["message"]
    }

  }
}

```

#output.conf

```
output {
  elasticsearch { hosts => ["10.141.127.145:9200"]
    hosts => "10.141.127.145:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

```

Built Filbeat on ubuntu client machine

Inside Filbeat Machine, my web logs under /var/log/web.log as follows

#web.log

```
2017-06-01 12:15:40 INFO Utils- DETAILS SUCCESSFULLY UPLOADED
2017-06-01 12:17:16 INFO PortalUpload- Upload Structure Data webservice 
2017-06-01 12:15:39 DEBUG PortalUpload- check
2017-06-01 12:15:39 ERROR PortalUpload- investigate details

```

#filbeat.yml

```
filebeat.prospectors:

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
     - /var/log/web.log

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]
  hosts: ["10.141.127.145:5443"]
  bulk_max_size: 2048
  #ssl.certificate_authorities: ["/etc/filebeat/logstash.crt"]
  template.name: "filebeat"
  template.path: "filebeat.template.json"
  template.overwrite: false

```

#Filbeat Logs, tail -f /var/log/filebeat

```
2017-06-26T17:48:18+05:30 ERR Failed to publish events caused by: write tcp 30.30.0.6:37661->10.141.127.145:5443: write: connection reset by peer
2017-06-26T17:48:18+05:30 INFO Error publishing events (retrying): write tcp 30.30.0.6:37661->10.141.127.145:5443: write: connection reset by peer
2017-06-26T17:48:38+05:30 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=24 libbeat.logstash.publish.write_bytes=10749 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=84 libbeat.logstash.published_but_not_acked_events=84 libbeat.publisher.published_events=84 publish.events=85 registrar.states.current=1 registrar.states.update=85 registrar.writes=1
2017-06-26T17:49:08+05:30 INFO No non-zero metrics in the last 30s
2017-06-26T17:49:38+05:30 INFO No non-zero metrics in the last 30s
2017-06-26T17:50:08+05:30 INFO No non-zero metrics in the last 30s
2017-06-26T17:50:38+05:30 INFO No non-zero metrics in the last 30s
2017-06-26T17:50:38+05:30 INFO Harvester started for file: /var/log/web.log
2017-06-26T17:50:38+05:30 ERR Failed to publish events caused by: write tcp 30.30.0.6:37662->10.141.127.145:5443: write: connection reset by peer
2017-06-26T17:50:38+05:30 INFO Error publishing events (retrying): write tcp 30.30.0.6:37662->10.141.127.145:5443: write: connection reset by peer
2017-06-26T17:51:08+05:30 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=450 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=3 libbeat.logstash.published_but_not_acked_events=3 libbeat.publisher.published_events=3 publish.events=5 registrar.states.update=5 registrar.writes=1

```

My questions are

1. How to resolve " `write: connection reset by peer`" error
2. How to send only "ERROR" logs to logstash server by configuring in filebeat.yml

I read in online, filbeat 5.4 version need `"tail\_files = true"

&&  
logging:  
to\_files: true  
files:  
path: /var/log/mybeat  
name: mybeat  
level: info

Is this correct way in 5.4 version to filter "ERROR" info in my web log to send logstash server.

---

<div class="post-metadata">

**Author:** ![Pandiyan\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pandiyan_m/32/19404_2.png) [@Pandiyan\_M](https://discuss.elastic.co/u/Pandiyan_M)\
**Post date:** [June 27, 2017, 4:56am UTC](https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821/2 "2017-06-27T04:56:43Z")

</div>

Any update ?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [June 27, 2017, 8:53am UTC](https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821/3 "2017-06-27T08:53:29Z")

</div>

> 1. How to resolve " write: connection reset by peer" error

which logstash version are you using? Someone, e.g. Logstash, the server/firewall/network device is closing the connection while filebeat is publish log messages to logstash. Filebeat normally reconnects and sends logs again. try with debug logging (`-d '*'`). In logstash you can increate `client_connectivity_timeout`, so Logstash is less likely to close the connection. Updating Logstash to most recent 5.3.x or 5.4 release + increasing `client_connectivity_timeout` should help

1. How to send only "ERROR" logs to logstash server by configuring in filebeat.yml

Use `include_lines` setting in filebeat prospector.

> I read in online, filbeat 5.4 version need `"tail\_files = true"

Huh, why?

> logging:  
> to\_files: true  
> files:  
> path: /var/log/mybeat  
> name: mybeat  
> level: info

Is this correct way in 5.4 version to filter "ERROR" info in my web log to send logstash server.

Sorry? This is the filebeat its internal logging capabilities. You want to send errors from filebeat or from monitored services to Logstash?

---

<div class="post-metadata">

**Author:** ![Pandiyan\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pandiyan_m/32/19404_2.png) [@Pandiyan\_M](https://discuss.elastic.co/u/Pandiyan_M)\
**Post date:** [June 27, 2017, 12:28pm UTC](https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821/4 "2017-06-27T12:28:38Z")

</div>

My Logs are as follows

2017-06-27 19:16:55 INFO PortalUpload- Pandy Sample Logs  
2017-06-27 19:16:55 ERROR PortalUpload- Pandy Sample Logs

I would like to parse logs contains field "ERROR", Include\_lines is to search lines starts with, but my use case is find FIELD contains "ERROR" , I found from [https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html)

```
fields:
    app_id: query_engine_12

```

How to use above in my use case ?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [June 28, 2017, 9:56am UTC](https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821/5 "2017-06-28T09:56:44Z")

</div>

`include_lines` setting accepts a regular expression. If all you want to forward is messages containing errors you can use `include_lines: ['ERROR']`. You can have the regular expression as strict as you want. For example this pattern `'^\d{4}(-\d{2}){2} \d{2}(:\d{2}){2} ERROR'` only includes lines starting with data + time + "ERROR" string.

Plus, instead of the `include_lines`, you can also use the `drop_events` processor like:

```auto
processors:
- drop_event:
    when.not.contains.message: 'ERROR'

```

---

<div class="post-metadata">

**Author:** ![Pandiyan\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pandiyan_m/32/19404_2.png) [@Pandiyan\_M](https://discuss.elastic.co/u/Pandiyan_M)\
**Post date:** [June 28, 2017, 10:16am UTC](https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821/6 "2017-06-28T10:16:35Z")

</div>

> [@steffens](#):
>
> include\_lines: ['ERROR']

If add "include\_lines: ['ERROR']" it is trying to find lines starting with ERROR, but for me ERROR is middle of row.. how to

`when.not.contains.message: 'ERROR'` I Need to send lines contains word "ERROR" see the sample logs below

2017-06-27 19:16:55 INFO PortalUpload- Pandy Sample Logs  
2017-06-27 19:16:55 ERROR PortalUpload- Pandy Sample Logs

Where ERROR is on third column, sometime it can be any column, so its better to use syntax matching "ERROR" word

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [June 28, 2017, 10:24am UTC](https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821/7 "2017-06-28T10:24:54Z")

</div>

please re-read my previous answer again. `include_lines` is a regular expression, it does not look for `ERROR` at the beginning of the line. The `drop_event` + `when.not` will remove all events not having `ERROR`. It's a double-negation -\> send only messages having string `ERROR`.

---

<div class="post-metadata">

**Author:** ![Pandiyan\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pandiyan_m/32/19404_2.png) [@Pandiyan\_M](https://discuss.elastic.co/u/Pandiyan_M)\
**Post date:** [June 28, 2017, 10:28am UTC](https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821/8 "2017-06-28T10:28:06Z")

</div>

Agreed with answer

`when.not.contains.message: 'ERROR'`

But I have tried Include\_lines: [^ERROR], but not working ..

also by using "when.not.contains.message: " can i use two filters like INFO and ERROR ?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [June 29, 2017, 11:45am UTC](https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821/9 "2017-06-29T11:45:36Z")

</div>

> But I have tried Include\_lines: [^ERROR], but not working ..

It's not working, because you're using the `^` (beginning of text) operator. Just use `include_lines: ["INFO", "ERROR"]`.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 27, 2017, 11:45am UTC](https://discuss.elastic.co/t/filbeat-tail-filter-logs-5-4-version/90821/10 "2017-07-27T11:45:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.