Filebeat parse json

Elastic Stack Beats
1

Filebeat 6.2.1

I don't think filebeat is seeing my json document inside of an array []. This causes json parse failures. Is there something I should be doing different to process this document?

Original document:

[{"traceID":9261530138935300,"messageType":"Default","eventType":"logEvent","signpostID":0,"source":null,"activityIdentifier":0,"subsystem":"com.apple.sharing","category":"AirDrop","threadID":6008981,"senderImageUUID":"34BD8CEF-7770-3DF6-9B02-B32788EA0C35","processImagePath":"/usr/libexec/sharingd","senderImagePath":"/usr/libexec/sharingd","timestamp":"2018-05-07 08:04:36.381104-0400","machTimestamp":1557005637979752,"eventMessage":"startSending, validated airdrop items. properties: {\n    ConvertMediaFormats = 0;\n    Files =     (\n                {\n            ConvertMediaFormats = 0;\n            FileBomPath = \"./lunch.pdf\";\n            FileIsDirectory = 0;\n            FileName = \"lunch.pdf\";\n            FileType = \"com.adobe.pdf\";\n        }\n    );\n    ReceiverComputerName = \"Tim\\U2019s iPhone\";\n    ReceiverID = daffcc6a517d;\n    VerifiableIdentity = 0;\n}","processImageUUID":"34BD8CEF-7770-3DF6-9B02-B32788EA0C35","processID":45882,"senderProgramCounter":1086398,"parentActivityIdentifier":0,"timezoneName":""}]

Filebeat.yml:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /path/to/airdrop.json 
  json.keys_under_root: true
  json.add_error_key: true
  tags: ["airdrop"]

processors:
- drop_fields:
    fields: ["beat.name", "beat.hostname", "beat.version", "beat", "host", "input_type", "source", "prospector.type"]

- decode_json_fields:
    fields: ["message"]
    process_array: true
    max_depth: 8

output.console:
  pretty: true

Output from console:

{
  "@timestamp": "2018-05-07T12:23:51.899Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.1"
  },
  "tags": [
    "airdrop"
  ],
  "prospector": {},
  "offset": 1048,
  "error": {
    "type": "json",
    "message": "Error decoding JSON: json: cannot unmarshal array into Go value of type map[string]interface {}"
  }
}
2

Currently, filebeat expects one json object per-line and not an array. In the future we might overcome this limitation.

If you don't have control over the log format, you still can get rid of JSON settings under the log prospector.

Remove this lines:

  json.keys_under_root: true
  json.add_error_key: true

And have the decode_json_fields prospector parse the log.

This is the output I get:

{
  "@timestamp": "2018-05-07T14:43:41.233Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "offset": 1048,
  "message": [
    {
      "signpostID": "0",
      "subsystem": "com.apple.sharing",
      "category": "AirDrop",
      "threadID": "6008981",
      "activityIdentifier": "0",
      "parentActivityIdentifier": "0",
      "timezoneName": "",
      "traceID": "9261530138935300",
      "senderImageUUID": "34BD8CEF-7770-3DF6-9B02-B32788EA0C35",
      "senderImagePath": "/usr/libexec/sharingd",
      "timestamp": "2018-05-07 08:04:36.381104-0400",
      "machTimestamp": "1557005637979752",
      "eventMessage": "startSending, validated airdrop items. properties: {\n    ConvertMediaFormats = 0;\n    Files =     (\n                {\n            ConvertMediaFormats = 0;\n            FileBomPath = \"./lunch.pdf\";\n            FileIsDirectory = 0;\n            FileName = \"lunch.pdf\";\n            FileType = \"com.adobe.pdf\";\n        }\n    );\n    ReceiverComputerName = \"Tim\\U2019s iPhone\";\n    ReceiverID = daffcc6a517d;\n    VerifiableIdentity = 0;\n}",
      "processID": "45882",
      "senderProgramCounter": "1086398",
      "messageType": "Default",
      "eventType": "logEvent",
      "source": null,
      "processImagePath": "/usr/libexec/sharingd",
      "processImageUUID": "34BD8CEF-7770-3DF6-9B02-B32788EA0C35"
    }
  ],
  "tags": [
    "airdrop"
  ],
  "prospector": {},
  "input": {
    "type": "log"
  }
}
1 Like
3

What is the decode_json_fields.process_array for? I thought it would be to process arrays in json data?

4

There are two separate facilities at work here.

One is the log prospector json support, which does not support arrays.

Another one is the decode_json_fields processor. This one does support arrays if the process_array flag is set.

The main difference in your case is that decode_jon_fields you cannot use the fields_under_root functionality.

5

Thank you. I did as you stated and it worked. so at least I can process the logs. Thanks for the help. I can fix the rest of the issues with \n and = to create valid json. Then all sould be well.

Thank you for the quick response

6

Hi, i am new to filebeat, When i tried with above filbeat.yml, i am having an issue on publishing console output/ to logstash but file beat processing the logs, please find the log below
2018-05-08T22:46:43.923+0200 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.1
2018-05-08T22:46:43.925+0200 INFO instance/beat.go:301 filebeat start running.
2018-05-08T22:46:43.925+0200 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-05-08T22:46:43.925+0200 INFO registrar/registrar.go:108 Loading registrar data from D:\filebeat-6.2.1-windows-x86_64\data\registry
2018-05-08T22:46:43.927+0200 INFO registrar/registrar.go:119 States Loaded from registrar: 1
2018-05-08T22:46:43.928+0200 WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-05-08T22:46:43.928+0200 INFO crawler/crawler.go:48 Loading Prospectors: 1
2018-05-08T22:46:43.928+0200 INFO log/prospector.go:111 Configured paths: [D:\VODMonitoring\post-data\a515734e-b97f-4787-a0d8-5d541e859295.json]
2018-05-08T22:46:43.928+0200 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 1
2018-05-08T22:46:43.929+0200 INFO log/harvester.go:216 Harvester started for file: D:\VODMonitoring\post-data\a515734e-b97f-4787-a0d8-5d541e859295.json
2018-05-08T22:47:14.025+0200 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":140},"total":{"ticks":171,"time":171,"value":171},"user":{"ticks":31,"time":31}},"info":{"ephemeral_id":"f25c49fe-19e3-4095-ae85-02aed7bd5a16","uptime":{"ms":30137}},"memstats":{"gc_next":4194304,"memory_alloc":1557480,"memory_total":3103296,"rss":16474112}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"console"},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":2,"update":1},"writes":1},"system":{"cpu":{"cores":12}}}}}
2018-05-08T22:47:43.927+0200 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":140},"total":{"ticks":171,"time":171,"value":171},"user":{"ticks":31,"time":31}},"info":{"ephemeral_id":"f25c49fe-19e3-4095-ae85-02aed7bd5a16","uptime":{"ms":60039}},"memstats":{"gc_next":4194304,"memory_alloc":1622152,"memory_total":3167968,"rss":2084864}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}}}}}
2018-05-08T22:48:13.927+0200 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":140},"total":{"ticks":171,"time":171,"value":171},"user":{"ticks":31,"time":31}},"info":{"ephemeral_id":"f25c49fe-19e3-4095-ae85-02aed7bd5a16","uptime":{"ms":90039}},"memstats":{"gc_next":4194304,"memory_alloc":1685224,"memory_total":3231040,"rss":32768}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}}}}}
2018-05-08T22:48:43.926+0200 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":140},"total":{"ticks":171,"time":171,"value":171},"user":{"ticks":31,"time":31}},"info":{"ephemeral_id":"f25c49fe-19e3-4095-ae85-02aed7bd5a16","uptime":{"ms":120038}},"memstats":{"gc_next":4194304,"memory_alloc":1743624,"memory_total":3289440,"rss":-8192}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}}}}}
2018-05-08T22:48:46.736+0200 INFO beater/filebeat.go:323 Stopping filebeat
2018-05-08T22:48:46.745+0200 INFO crawler/crawler.go:109 Stopping Crawler
2018-05-08T22:48:46.745+0200 INFO crawler/crawler.go:119 Stopping 1 prospectors
2018-05-08T22:48:46.745+0200 INFO prospector/prospector.go:121 Prospector ticker stopped
2018-05-08T22:48:46.745+0200 INFO prospector/prospector.go:138 Stopping Prospector: 5865738280652087874
2018-05-08T22:48:46.745+0200 INFO log/harvester.go:237 Reader was closed: D:\VODMonitoring\post-data\a515734e-b97f-4787-a0d8-5d541e859295.json. Closing.
2018-05-08T22:48:46.745+0200 INFO crawler/crawler.go:135 Crawler stopped
2018-05-08T22:48:46.745+0200 INFO registrar/registrar.go:210 Stopping Registrar
2018-05-08T22:48:46.745+0200 INFO registrar/registrar.go:165 Ending Registrar
2018-05-08T22:48:46.746+0200 INFO instance/beat.go:308 filebeat stopped.

7

Hi @pbala2018

It's better if you open a separate thread instead of reusing this one.

8

Thanks for update @adrisr. i have created separate thread

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.