How can I parser json log with uncertain fields?

Elastic Stack Beats
1

I am new to filebeat and I trying to find a way to parser json without predefined fields.

For example I have a log file like this:

{"key1":"value1","key2":"value2"}
{"key1":"value1","key3":"value3"}

Some of the keys are fixed so I can put into decode_json_fields, but there are some custom keys that I cannot predict. How should I config my filebeat.yml so that those keys can also be recognized and parsed successfully?

2

Hey @Martin_Ma11 welcome to Elastic discuss.

decode_json_fields works in a way that it will take a field which has a json input and then 'unpacks' the decoded json into your struct.

so in case you have a message field with a json input { "a": "b", "c": 2 }
after decode_json_field configured to parse message field makes his thing you will end up with new content of a message field:

"message": {
   "a": "b",
   "c": 2,
}

there's no need to know the content of the message field

3

Hi @Michal_Pristas,

Thanks for the update.
Sorry I made a mistake in my initial question.
Actually my config is working, but the problem is that it only works for the first time which I believe is due to filebeat remembering file and line info.
As a workaround I am copying the file with different filename each time.

Here is my config file:

filebeat.inputs:
- type: log
  paths:
    - test2.log
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: method

output:
  console:
    pretty: true

What I am trying to do here is testing my config file and see if it works as expected.
To be more specific, I am trying to run filebeat as cmd line with --once.

However I have encountered the following issues:

  1. Filebeat not looking for specified config file, instead it always looks for the default location.
filebeat -c ./filebeat.yml -e --once
2019-07-17T10:06:01.059+0900	INFO	instance/beat.go:606	Home path: [/usr/local/Cellar/filebeat-oss/7.2.0/libexec] Config path: [/usr/local/etc/filebeat] Data path: [/usr/local/var/lib/filebeat] Logs path: [/usr/local/var/log/filebeat]
  1. --once not stopping filebeat after all log lines has been processed in test.log.
2019-07-17T10:06:01.090+0900	INFO	beater/filebeat.go:376	All data collection completed. Shutting down.
2019-07-17T10:06:01.090+0900	INFO	log/harvester.go:253	Harvester started for file: /xxxx/test2.log
2019-07-17T10:06:01.090+0900	INFO	crawler/crawler.go:139	Stopping Crawler
2019-07-17T10:06:01.090+0900	INFO	crawler/crawler.go:149	Stopping 1 inputs
 
<some json output>

2019-07-17T10:06:31.090+0900	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":26,"time":{"ms":26}},"total":{"ticks":61,"time":{"ms":61},"value":61},"user":{"ticks":35,"time":{"ms":35}}},"info":{"ephemeral_id":"159f09a7-6541-470f-8cc6-f9b219ae31e8","uptime":{"ms":30048}},"memstats":{"gc_next":4194304,"memory_alloc":1976464,"memory_total":5629792,"rss":20844544},"runtime":{"goroutines":23}},"filebeat":{"events":{"added":3,"done":3},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":2,"batches":1,"total":2},"type":"console","write":{"bytes":2200}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"published":2,"total":3},"queue":{"acked":2}}},"registrar":{"states":{"current":3,"update":3},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":8},"load":{"1":1.6206,"15":1.9434,"5":1.7983,"norm":{"1":0.2026,"15":0.2429,"5":0.2248}}}}}}

Since I am just testing filebeat and configs, I wish to make it look for config file that I specified and not remember the filename/content, also ideally it would end once all contents of log file has been processed.

Is it possible to achieve the the above?

4

Sorry I forgot to mention that I am using filebeat 7.2.0.

filebeat version 7.2.0 (amd64), libbeat 7.2.0 [9ba65d864ca37cd32c25b980dbb4020975288fc0 built 2019-06-20 14:46:22 +0000 UTC]

My laptop is running MacOS 10.14.5

sw_vers
ProductName: Mac OS X
ProductVersion: 10.14.5
BuildVersion: 18F132

5

I have found the answer to above questions myself.

  1. To specify a custom config file, I must set --path.config as well, in my case I am setting it to local folder --path.config ..
  2. --once actually works, but it will wait for a few minutes before exit, I don't understand why, maybe there is a graceful exit time? Is this expected behavior? Can I configure it?
  3. I still don't know how to make filebeat forget file name, but anyway I can copy the file to workaround it so it's not an urgent issue for me now.

However now I have another issue with the data structure I am trying to parse.

In my json data (actually it's log message), there is a key data.error and it could be one of the following:

  1. "error":Null - when there is no error
  2. "error":"String" - something like "no user found"
  3. "error":{"subkey1":"subvalue1","subkey2":"subvalue2"} - nested object with more details

ES is giving me the following error message:

{"type":"mapper_parsing_exception","reason":"object mapping for [data.error] tried to parse field [error] as object, but found a concrete value"}

And the key that generates this error looks like this:

"data":{"request_guid":"54162ccc-803e-42f3-5954-0118ec9d0f0e::190f13bd-2694-449f-b099-74ac24924749","error":"No running instances found for process guid 974c85ac-5202-4dbd-8446-515e7e3183d5-20aefb63-8cf6-4bfa-a028-710e504481ad"}

So how can I make ES parse this key correctly? That is ES can handle both string and nested key at the same time.

6

@Michal_Pristas

Any chance to have an update regarding this question?

7

hey @Martin_Ma11
sorry for the delay.
if you still have troubles with --once try specifying close_eof link

this will close harvester once it reaches EOF. otherwise it close when inactive which means it will close after specified period of time without events.

filebeat keeps state in registry. by removing this registry you will make filebeat forget the state
it is usually (unless configured otherwise) here:
#filebeat.registry.path: ${path.data}/registry
path.data is dependent on the way you installed filebeat and our OS, but you can get some help here

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.