# How can I parser json log with uncertain fields?

**URL:** <https://discuss.elastic.co/t/how-can-i-parser-json-log-with-uncertain-fields/190680>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 16, 2019, 8:25am UTC](https://discuss.elastic.co/t/how-can-i-parser-json-log-with-uncertain-fields/190680 "2019-07-16T08:25:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Martin\_Ma11](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_ma11/32/46017_2.png) [@Martin\_Ma11](https://discuss.elastic.co/u/Martin_Ma11)\
**Post date:** [July 16, 2019, 8:25am UTC](https://discuss.elastic.co/t/how-can-i-parser-json-log-with-uncertain-fields/190680/1 "2019-07-16T08:25:51Z")

</div>

I am new to filebeat and I trying to find a way to parser json without predefined fields.

For example I have a log file like this:

```auto
{"key1":"value1","key2":"value2"}
{"key1":"value1","key3":"value3"}

```

Some of the keys are fixed so I can put into decode\_json\_fields, but there are some custom keys that I cannot predict. How should I config my filebeat.yml so that those keys can also be recognized and parsed successfully?

---

<div class="post-metadata">

**Author:** ![Michal\_Pristas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michal_pristas/32/46639_2.png) [@Michal\_Pristas](https://discuss.elastic.co/u/Michal_Pristas)\
**Post date:** [July 16, 2019, 9:55am UTC](https://discuss.elastic.co/t/how-can-i-parser-json-log-with-uncertain-fields/190680/2 "2019-07-16T09:55:34Z")

</div>

Hey @Martin_Ma11 welcome to Elastic discuss.

`decode_json_fields` works in a way that it will take a field which has a json input and then 'unpacks' the decoded json into your struct.

so in case you have a `message` field with a json input `{ "a": "b", "c": 2 }`  
after `decode_json_field` configured to parse `message` field makes his thing you will end up with new content of a `message` field:

```auto
"message": {
   "a": "b",
   "c": 2,
}

```

there's no need to know the content of the `message` field

---

<div class="post-metadata">

**Author:** ![Martin\_Ma11](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_ma11/32/46017_2.png) [@Martin\_Ma11](https://discuss.elastic.co/u/Martin_Ma11)\
**Post date:** [July 17, 2019, 1:14am UTC](https://discuss.elastic.co/t/how-can-i-parser-json-log-with-uncertain-fields/190680/3 "2019-07-17T01:14:31Z")

</div>

Hi @Michal_Pristas,

Thanks for the update.  
Sorry I made a mistake in my initial question.  
Actually my config is working, but the problem is that it only works for the first time which I believe is due to filebeat remembering file and line info.  
As a workaround I am copying the file with different filename each time.

Here is my config file:

```auto
filebeat.inputs:
- type: log
  paths:
    - test2.log
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: method

output:
  console:
    pretty: true

```

What I am trying to do here is testing my config file and see if it works as expected.  
To be more specific, I am trying to run filebeat as cmd line with `--once`.

However I have encountered the following issues:

1. Filebeat not looking for specified config file, instead it always looks for the default location.

```auto
filebeat -c ./filebeat.yml -e --once
2019-07-17T10:06:01.059+0900	INFO	instance/beat.go:606	Home path: [/usr/local/Cellar/filebeat-oss/7.2.0/libexec] Config path: [/usr/local/etc/filebeat] Data path: [/usr/local/var/lib/filebeat] Logs path: [/usr/local/var/log/filebeat]

```

1. `--once` not stopping filebeat after all log lines has been processed in `test.log`.

```auto
2019-07-17T10:06:01.090+0900	INFO	beater/filebeat.go:376	All data collection completed. Shutting down.
2019-07-17T10:06:01.090+0900	INFO	log/harvester.go:253	Harvester started for file: /xxxx/test2.log
2019-07-17T10:06:01.090+0900	INFO	crawler/crawler.go:139	Stopping Crawler
2019-07-17T10:06:01.090+0900	INFO	crawler/crawler.go:149	Stopping 1 inputs
 
<some json output>

2019-07-17T10:06:31.090+0900	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":26,"time":{"ms":26}},"total":{"ticks":61,"time":{"ms":61},"value":61},"user":{"ticks":35,"time":{"ms":35}}},"info":{"ephemeral_id":"159f09a7-6541-470f-8cc6-f9b219ae31e8","uptime":{"ms":30048}},"memstats":{"gc_next":4194304,"memory_alloc":1976464,"memory_total":5629792,"rss":20844544},"runtime":{"goroutines":23}},"filebeat":{"events":{"added":3,"done":3},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":2,"batches":1,"total":2},"type":"console","write":{"bytes":2200}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"published":2,"total":3},"queue":{"acked":2}}},"registrar":{"states":{"current":3,"update":3},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":8},"load":{"1":1.6206,"15":1.9434,"5":1.7983,"norm":{"1":0.2026,"15":0.2429,"5":0.2248}}}}}}

```

Since I am just testing filebeat and configs, I wish to make it look for config file that I specified and not remember the filename/content, also ideally it would end once all contents of log file has been processed.

Is it possible to achieve the the above?

---

<div class="post-metadata">

**Author:** ![Martin\_Ma11](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_ma11/32/46017_2.png) [@Martin\_Ma11](https://discuss.elastic.co/u/Martin_Ma11)\
**Post date:** [July 17, 2019, 1:16am UTC](https://discuss.elastic.co/t/how-can-i-parser-json-log-with-uncertain-fields/190680/4 "2019-07-17T01:16:05Z")

</div>

Sorry I forgot to mention that I am using filebeat 7.2.0.

```auto
filebeat version 7.2.0 (amd64), libbeat 7.2.0 [9ba65d864ca37cd32c25b980dbb4020975288fc0 built 2019-06-20 14:46:22 +0000 UTC]

```

My laptop is running MacOS 10.14.5

sw\_vers  
ProductName: Mac OS X  
ProductVersion: 10.14.5  
BuildVersion: 18F132

---

<div class="post-metadata">

**Author:** ![Martin\_Ma11](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_ma11/32/46017_2.png) [@Martin\_Ma11](https://discuss.elastic.co/u/Martin_Ma11)\
**Post date:** [July 17, 2019, 5:57am UTC](https://discuss.elastic.co/t/how-can-i-parser-json-log-with-uncertain-fields/190680/5 "2019-07-17T05:57:21Z")

</div>

I have found the answer to above questions myself.

1. To specify a custom config file, I must set `--path.config` as well, in my case I am setting it to local folder `--path.config .`.
2. `--once` actually works, but it will wait for a few minutes before exit, I don't understand why, maybe there is a graceful exit time? Is this expected behavior? Can I configure it?
3. I still don't know how to make filebeat forget file name, but anyway I can copy the file to workaround it so it's not an urgent issue for me now.

However now I have another issue with the data structure I am trying to parse.

In my json data (actually it's log message), there is a key `data.error` and it could be one of the following:

1. "error":Null - when there is no error
2. "error":"String" - something like "no user found"
3. "error":{"subkey1":"subvalue1","subkey2":"subvalue2"} - nested object with more details

ES is giving me the following error message:

```auto
{"type":"mapper_parsing_exception","reason":"object mapping for [data.error] tried to parse field [error] as object, but found a concrete value"}

```

And the key that generates this error looks like this:

```auto
"data":{"request_guid":"54162ccc-803e-42f3-5954-0118ec9d0f0e::190f13bd-2694-449f-b099-74ac24924749","error":"No running instances found for process guid 974c85ac-5202-4dbd-8446-515e7e3183d5-20aefb63-8cf6-4bfa-a028-710e504481ad"}

```

So how can I make ES parse this key correctly? That is ES can handle both string and nested key at the same time.

---

<div class="post-metadata">

**Author:** ![Martin\_Ma11](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_ma11/32/46017_2.png) [@Martin\_Ma11](https://discuss.elastic.co/u/Martin_Ma11)\
**Post date:** [July 23, 2019, 8:23am UTC](https://discuss.elastic.co/t/how-can-i-parser-json-log-with-uncertain-fields/190680/6 "2019-07-23T08:23:29Z")

</div>

@Michal_Pristas

Any chance to have an update regarding this question?

---

<div class="post-metadata">

**Author:** ![Michal\_Pristas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michal_pristas/32/46639_2.png) [@Michal\_Pristas](https://discuss.elastic.co/u/Michal_Pristas)\
**Post date:** [August 6, 2019, 11:41am UTC](https://discuss.elastic.co/t/how-can-i-parser-json-log-with-uncertain-fields/190680/7 "2019-08-06T11:41:39Z")

</div>

hey @Martin_Ma11  
sorry for the delay.  
if you still have troubles with `--once` try specifying `close_eof` [link](https://www.elastic.co/guide/en/beats/filebeat/master/advanced-settings.html)

this will close harvester once it reaches EOF. otherwise it close when inactive which means it will close after specified period of time without events.

filebeat keeps state in registry. by removing this registry you will make filebeat forget the state  
it is usually (unless configured otherwise) here:  
`#filebeat.registry.path: ${path.data}/registry`  
`path.data` is dependent on the way you installed filebeat and our OS, but you can get some help [here](https://www.elastic.co/guide/en/beats/filebeat/master/directory-layout.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 3, 2019, 11:41am UTC](https://discuss.elastic.co/t/how-can-i-parser-json-log-with-uncertain-fields/190680/8 "2019-09-03T11:41:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.