# Filebeat cutting events from tiem to time

**URL:** <https://discuss.elastic.co/t/filebeat-cutting-events-from-tiem-to-time/211795>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [December 13, 2019, 1:11pm UTC](https://discuss.elastic.co/t/filebeat-cutting-events-from-tiem-to-time/211795 "2019-12-13T13:11:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mtudisco](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@mtudisco](https://discuss.elastic.co/u/mtudisco)\
**Post date:** [December 13, 2019, 1:11pm UTC](https://discuss.elastic.co/t/filebeat-cutting-events-from-tiem-to-time/211795/1 "2019-12-13T13:11:14Z")

</div>

Hi,  
I have filebeat 7.3.0-1, that reads files from one directory and sends information to logstash, it process it and writes to elasticsearch.

I have a monitoring daemon that from time to time, gets a report from a database and writes a file on the directory logstash reads, the report might have several lines, in my case for the moment just one line, and writes line by line to the file.

`/home | 25358| 30620|*ERROR*`

every field is separated by a "|" and there might be several spaces before or after ther pipe and the value.

Filebeat reads line by line, filebeat.yml config is:

```
- type: log
  enabled: true
  paths:
    - /stage/logs/*chk_df_autoextend.txt
  fields:
    type: oracle
  fields_under_root: true
  harvester_buffer_size: 2097153
  close_inactive: 5m
  close_removed: true
  close_eof: true

```

then with logstash i process the information with the following filter:

```
filter {
 if [type] == "oracle" and [type2] == "df_autoextend" {
       #mount_point|free_mb|incremento|status
       dissect {
                mapping => {
                            "message" => "%{[df_autoextend][mount_point]->}|%{[df_autoextend][free_mb]}|%{[df_autoextend][incremento]}|%{[df_autoextend][status]->}"
                           }
               }
       mutate {
                convert => {
                  "[df_autoextend][mount_point]" => "string"
                  "[df_autoextend][free_mb]" => "integer"
                  "[df_autoextend][incremento]" => "integer"
                  "[df_autoextend][status]" => "string"
                }
              }
 }
}

```

Once in a while (once a day or even once with more than one day between) i get the message cut by logstash, for instance:

```
{
  "_index": "orachecks-2019.12",
  "_type": "_doc",
  "_id": "CSfE-W4BByjEVZh5ZWFm",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-12-12T11:00:17.000Z",
    "type": "oracle",
    "ecs": {
      "version": "1.0.1"
    },
    "agent": {
      "ephemeral_id": "36696bd3-a1c0-4146-8cd2-8c0e5810a5cb",
      "version": "7.3.0",
      "hostname": "elk-lab01.tilsor.com.uy",
      "id": "7502e50a-1256-42de-b007-d0c5336841ce",
      "type": "filebeat"
    },
    "host": {
      "name": "elk-lab01.tilsor.com.uy"
    },
    "log": {
      "offset": 59,
      "file": {
        "path": "/stage/logs/2019-12-12-080017-chk_df_autoextend.txt"
      }
    },
    "type2": "df_autoextend",
    "tags": [],
    "df_autoextend": {
      "incremento": 30620,
      "mount_point": " ",
      "status": "*ERROR* ",
      "free_mb": 25358
    },
    "input": {
      "type": "log"
    },
    "@version": "1"
  },
  "fields": {
    "@timestamp": [
      "2019-12-12T11:00:17.000Z"
    ],
    "tbs_space.tbs_used_space": [
      null
    ],
    "tbs_space.tbs_status": [
      null
    ],
    "detail_diskgroups.dg_used_mb": [
      null
    ]
  },
  "highlight": {
    "type2.keyword": [
      "@kibana-highlighted-field@df_autoextend@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1576148417000
  ]
}

```

The problem is why do i get a message that start at offset 59 when it should read the whole line and send it to logstash??. Its causing logstash to generate incomplete information in elasticsearch. I could add some filter and discard the malformed events, but its not malformed in the original file.

Any clue?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 10, 2020, 1:11pm UTC](https://discuss.elastic.co/t/filebeat-cutting-events-from-tiem-to-time/211795/2 "2020-01-10T13:11:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.