Parse JSON data with filebeat

Elastic Stack Beats
1

I'm trying to parse JSON logs our server application is producing. It's writing to 3 log files in a directory I'm mounting in a Docker container running Filebeat. So far so good, it's reading the log files all right. However, in Kibana, the messages arrive, but the content itself it just shown as a field called "message" and the data in the content field is not accessible via its own fields (like "source_address", etc.).

I'm writing the logs using logrus and I want Beats to pass them straight on to Elasticsearch. I went through all kinds of documentation and found multiple ways of achieving what I want, but none of it's working.

This is my current configuration for filebeat:

filebeat.prospectors:
- paths:
    - /mnt/log/*.log
  input_type: log
  multiline.pattern: '^{'
  multiline.negate: false
  multiline.match:  after

processors:
 - decode_json_fields:
     fields: ['message']
     target: json

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  template.name: filebeat
  template.path: filebeat.template.json

A little nudge in the right direction would be greatly appreciated.

2

Are the JSON objects written as one per line? The multiline configuration seems to indicate that are not. In general, I'd recommend:

  • configuring logrus to output one object per line
  • remove the multiline configuration
  • Use the json decoding at the prospector level, instead of the processor

See, for example, this blog post on the topic.

3

Thank you for taking the time to respond.

I changed the configuration of Beats to this:

filebeat.prospectors:
- paths:
   - /mnt/log/*.log
  input_type: log
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: log

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  template.name: filebeat
  template.path: filebeat.template.json

This is what the log files look like. Every log is on its own line:

{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-25T14:57:15Z"}
{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-25T14:57:18Z"}
{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-27T16:36:49Z"}
{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-27T16:36:52Z"}
{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-27T16:36:55Z"}

Yet still, Kibana only shows a "message" field. I have a strange feeling that I'm using the message_key field in a wrong way. Am I using it right? What do I use as the key?

4

For those logs I would use message_key: msg. But you can also omit that config option because it's optional and you are not doing anything that would required it.

I ran a quick test with your logs and this config and it worked as expected.

filebeat.prospectors:
- paths:
   - test.json
  input_type: log
  json.keys_under_root: true
  json.add_error_key: true

output.elasticsearch:
  hosts: ["http://localhost:9200"]

I was getting data like:

{
  "@timestamp": "2017-03-27T17:11:31.359Z",
  "beat": {
    "hostname": "x",
    "name": "x",
  },
  "error": "dial tcp 172.21.0.2:5432: getsockopt: connection refused",
  "input_type": "log",
  "level": "error",
  "msg": "Could not open database connection",
  "offset": 790,
  "source": "/Users/akroh/go/src/github.com/elastic/beats/filebeat/.test/logrus/test.json",
  "time": "2017-03-27T16:36:55Z",
  "type": "log"
}
1 Like
5

I queried Elasticsearch and got the following:

{
        "_index" : "filebeat-2017.03.27",
        "_type" : "log",
        "_id" : "AVsQ0S06i5178hY1Lr4U",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2017-03-27T17:29:16.356Z",
          "beat" : {
            "hostname" : "15f7ccc3a4c1",
            "name" : "15f7ccc3a4c1",
            "version" : "5.2.2"
          },
          "input_type" : "log",
          "message" : "{\"error\":\"dial tcp 172.21.0.2:5432: getsockopt: connection refused\",\"level\":\"error\",\"msg\":\"Could not open database connection\",\"time\":\"2017-03-27T17:19:35Z\"}",
          "offset" : 158,
          "source" : "/mnt/log/error.log",
          "type" : "log"
        }
      }

Could it somehow be related to Docker? I recreated all containers before starting the server and checking the logs. I suppose it doesn't matter which file endings the log files have (.json or .log)?

6

I just tested changing the configuration to this:

filebeat.prospectors:
- paths:
    - /mnt/log/info.json
  input_type: log
  json.keys_under_root: true
  json.add_error_key: true

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

What happens now is that filebeat still reads my *.log files, but doesn't read the .json file specified in the config.
This is the output I get when querying Elasticsearch:

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 6,
    "successful" : 6,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".kibana",
        "_type" : "config",
        "_id" : "5.2.2",
        "_score" : 1.0,
        "_source" : {
          "buildNum" : 14723
        }
      },
      {
        "_index" : "filebeat-2017.03.27",
        "_type" : "log",
        "_id" : "AVsRLezyYgiGm0kvKUi9",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2017-03-27T19:10:36.110Z",
          "beat" : {
            "hostname" : "8deef5bae0db",
            "name" : "8deef5bae0db",
            "version" : "5.2.2"
          },
          "input_type" : "log",
          "message" : "{\"error\":\"dial tcp 172.21.0.3:5432: getsockopt: connection refused\",\"level\":\"error\",\"msg\":\"Could not open database connection\",\"time\":\"2017-03-27T19:10:35Z\"}",
          "offset" : 158,
          "source" : "/mnt/log/error.log",
          "type" : "log"
        }
      },
      {
        "_index" : "filebeat-2017.03.27",
        "_type" : "log",
        "_id" : "AVsRLezyYgiGm0kvKUi-",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2017-03-27T19:10:39.111Z",
          "beat" : {
            "hostname" : "8deef5bae0db",
            "name" : "8deef5bae0db",
            "version" : "5.2.2"
          },
          "input_type" : "log",
          "message" : "{\"error\":\"dial tcp 172.21.0.3:5432: getsockopt: connection refused\",\"level\":\"error\",\"msg\":\"Could not open database connection\",\"time\":\"2017-03-27T19:10:38Z\"}",
          "offset" : 316,
          "source" : "/mnt/log/error.log",
          "type" : "log"
        }
      }
    ]
  }
}
7

It sounds like a configuration issue with your Filebeat container where it's not using the config file you expected. Could you trying running Filebeat outside of docker and see if you can reproduce the issue.

File extensions don't matter to Filebeat.

8

I'm so sorry for wasting your time. What you just wrote made me check the documentation of the official beats docker image again. I put the configuration file in the wrong directory. It's now working as intended.

{
        "_index" : "filebeat-2017.03.27",
        "_type" : "log",
        "_id" : "AVsRP1CAXkvX0lRuH3-Y",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2017-03-27T19:30:01.146Z",
          "beat" : {
            "hostname" : "1a38663b0526",
            "name" : "1a38663b0526",
            "version" : "5.2.2"
          },
          "input_type" : "log",
          "level" : "info",
          "msg" : "Directory created",
          "offset" : 306,
          "path" : "/toronto/media/",
          "source" : "/mnt/log/info.log",
          "time" : "2017-03-27T19:29:53Z",
          "type" : "log"
        }

Thank you very much for your help! @andrewkroh @tudor

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.