JSON logs --> fail to ship the last line of the file

Elastic Stack Beats
1

Pulled down the latest alpha (filebeat-5.0.0-alpha3-darwin-x64).
Below is the config to ship .json logs

- input_type: log

    # Paths that should be crawled and fetched. Glob based paths.
    paths:
      - ./test/test.json
      #- c:\programdata\elasticsearch\logs\*
    json.message_key: name
    json.keys_under_root: false
    json.overwrite_keys: false
    json.add_error_key: true`

Here is the input file

{"name":"first","count":100}
{"name":"second","count":200}
{"name":"third","count":300}
{"name":"4th","count":400}
{"name":"5th","count":0}
{"name":"6th","count":99}
{"name":"7th","count":992}

I got 7 lines to ship, below is the console log when I run ./filebeat -e

**==> ./filebeat -e**
{
  "@timestamp": "2016-06-12T17:03:49.223Z",
  "beat": {
    "hostname": "Sahass-MacBook-Pro.local",
    "name": "Sahass-MacBook-Pro.local"
  },
  "input_type": "log",
  "json": {
    "count": 100,
    "name": "first"
  },
  "offset": 29,
  "source": "./test/test.json",
  "type": "log"
}
{
  "@timestamp": "2016-06-12T17:03:49.223Z",
  "beat": {
    "hostname": "Sahass-MacBook-Pro.local",
    "name": "Sahass-MacBook-Pro.local"
  },
  "input_type": "log",
  "json": {
    "count": 200,
    "name": "second"
  },
  "offset": 59,
  "source": "./test/test.json",
  "type": "log"
}
{
  "@timestamp": "2016-06-12T17:03:49.223Z",
  "beat": {
    "hostname": "Sahass-MacBook-Pro.local",
    "name": "Sahass-MacBook-Pro.local"
  },
  "input_type": "log",
  "json": {
    "count": 300,
    "name": "third"
  },
  "offset": 88,
  "source": "./test/test.json",
  "type": "log"
}
{
  "@timestamp": "2016-06-12T17:03:49.223Z",
  "beat": {
    "hostname": "Sahass-MacBook-Pro.local",
    "name": "Sahass-MacBook-Pro.local"
  },
  "input_type": "log",
  "json": {
    "count": 400,
    "name": "4th"
  },
  "offset": 115,
  "source": "./test/test.json",
  "type": "log"
}
{
  "@timestamp": "2016-06-12T17:03:49.223Z",
  "beat": {
    "hostname": "Sahass-MacBook-Pro.local",
    "name": "Sahass-MacBook-Pro.local"
  },
  "input_type": "log",
  "json": {
    "count": 0,
    "name": "5th"
  },
  "offset": 140,
  "source": "./test/test.json",
  "type": "log"
}
{
  "@timestamp": "2016-06-12T17:03:49.223Z",
  "beat": {
    "hostname": "Sahass-MacBook-Pro.local",
    "name": "Sahass-MacBook-Pro.local"
  },
  "input_type": "log",
  "json": {
    "count": 99,
    "name": "6th"
  },
  "offset": 166,
  "source": "./test/test.json",
  "type": "log"
}

Its is consistently missing the last line when lines are added incrementally
When I run filbeat with preset input file, it does ship all 7 log entries..

Am I missing something obvious?

Thanks !

2

Do you perhaps a new line character after the last line?

Note: I reformatted your post to make it more readable.

3

@ruflin - thanks, didn't know that newline at the end is necessary.. that was it.

4

@ruflin is having an newline at the end of the input file a running rule for all input files? Or something unique to JSON input files?

5

It is currently a rule for all inputs. It is the way filebeat detects the end of an "event".

1 Like
6

This topic was automatically closed after 21 days. New replies are no longer allowed.