# Filebeat send a json log but is stored as a string

**URL:** <https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 5, 2016, 2:35pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410 "2016-01-05T14:35:49Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![djvidov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djvidov/32/6135_2.png) [@djvidov](https://discuss.elastic.co/u/djvidov)\
**Post date:** [January 5, 2016, 2:35pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/1 "2016-01-05T14:35:49Z")

</div>

Hello,

I have logs from a file, like that:  
`{"LogLevel":"INFO","StartTime":"15:01:30.625528","ExecutionTime":1,"CallingMethod":"GetHistoryComments"}`  
`{"LogLevel":"INFO","StartTime":"15:01:25.5745528","ExecutionTime":0,"CallingMethod":"InitApp"}`

I use filebeat 1.0 to send these logs to elastiscsearch and instead to have this json object into elasticsearch is stored into "message" field as string, like that:

> {  
> "took": 3,  
> "timed\_out": false,  
> "\_shards": {  
> "total": 5,  
> "successful": 5,  
> "failed": 0  
> },  
> "hits": {  
> "total": 1272,  
> "max\_score": 1,  
> "hits": [  
> {  
> "\_index": "dev-2016.01.05",  
> "\_type": "log",  
> "\_id": "AVISEHLA9KG7wxjkeWpi",  
> "\_score": 1,  
> "\_source": {  
> "@timestamp": "2016-01-05T13:53:29.648Z",  
> "beat": {  
> "hostname": "DEV01",  
> "name": "DEV01"  
> },  
> "count": 1,  
> "fields": null,  
> "input\_type": "log",  
> "message": " {"LogLevel":"INFO","StartTime":"15:53:29.1970996","ExecutionTime":0,"CallingMethod":"InvokeMethod"},  
> "offset": 50444441,  
> "source": "D:\Logs\Pl\_20160105-15.log",  
> "type": "log"  
> }  
> },....

Any ideea how to make it json?

Thank you!  
Ovidiu

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 5, 2016, 2:46pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/2 "2016-01-05T14:46:26Z")

</div>

Set `codec => "json"` for your beats input.

---

<div class="post-metadata">

**Author:** ![djvidov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djvidov/32/6135_2.png) [@djvidov](https://discuss.elastic.co/u/djvidov)\
**Post date:** [January 5, 2016, 3:08pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/3 "2016-01-05T15:08:44Z")

</div>

Thanks for answer.  
where is codec setting?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 5, 2016, 3:10pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/4 "2016-01-05T15:10:02Z")

</div>

```auto
input {
  beats {
    ...
    codec => "json"
  }
}

```

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [January 5, 2016, 8:02pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/5 "2016-01-05T20:02:01Z")

</div>

filebeat is forwarding lines only, not parsing json into any kind of document. This functionality is provided by logstash codec or filters.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [January 5, 2016, 8:04pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/6 "2016-01-05T20:04:05Z")

</div>

there's some discussion [on github](https://github.com/elastic/filebeat/issues/311) about supporting direct indexing of json logs

---

<div class="post-metadata">

**Author:** ![djvidov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djvidov/32/6135_2.png) [@djvidov](https://discuss.elastic.co/u/djvidov)\
**Post date:** [January 6, 2016, 7:29am UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/7 "2016-01-06T07:29:23Z")

</div>

Thank you for answer. I agree with you, filebeat is a forwarding, and I read also discussion from github, but as first expectation was, i have in file json, I will have in elasticsearch json, but in the end a line from a file it will remain a string.

---

<div class="post-metadata">

**Author:** ![djvidov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djvidov/32/6135_2.png) [@djvidov](https://discuss.elastic.co/u/djvidov)\
**Post date:** [January 6, 2016, 10:15am UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/8 "2016-01-06T10:15:21Z")

</div>

Hello,

I start to write json log partser for logstash, which looks like:

> ```
> input {
> stdin { 
> codec => "json"
> }
> beats {
> codec => "json"
> port => 9202
> }
> }
> 
> ```

```
filter { }

output {
	stdout { codec => rubydebug }
	elasticsearch { 
		hosts => ["localhost:9200"] 
		user => "admin1"
		password => "admin1"
	}
}

```

and it really works for a tiny json.

But for a real json, like this:

```
{
	"LogLevel": "INFO",
	"StartTime": "2016-01-05 14:39:40.2012272",
	"ExecutionTime": 0,
	"CallingMethod": "SetPars",
	"Correlation": {
		"CorrelationId": "60e9d53e-e936-4e87-a47f-e009d4e3f3fb",
		"SequenceNo": 1
	},
	"Inputs": ["39b80f66-8547-4a3e-b9e1-e69a2d56329b",
	[{
		"Value": "IBr",
		"Name": "BrName",
		"Status": 0
	},
	{
		"Value": 101,
		"Name": "BrCode",
		"Status": 0
	}]]
}

```

even if this is a vaild json ( I have test it on: [http://jsonlint.com/](http://jsonlint.com/)) it return me this huge error:

```
{
         "LogLevel" => "INFO",
        "StartTime" => "2016-01-05 14:39:40.2012272",
    "ExecutionTime" => 0,
    "CallingMethod" => "SetPars",
      "Correlation" => {
        "CorrelationId" => "60e9d53e-e936-4e87-a47f-e009d4e3f3fb",
           "SequenceNo" => 1
    },
           "Inputs" => [
        [0] "39b80f66-8547-4a3e-b9e1-e69a2d56329b",
        [1] [
            [0] {
                 "Value" => "IBr",
                  "Name" => "BrName",
                "Status" => 0
            },
            [1] {
                 "Value" => 101,
                  "Name" => "BrCode",
                "Status" => 0
            }
        ]
    ],
         "@version" => "1",
       "@timestamp" => "2016-01-06T09:35:15.001Z",
             "host" => "DV01LPT09"
}

←[33mFailed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2016.01.06", :_type=>"logs", :_routing=>nil}, 
#<LogStash::Event:0x1a5364ea @metadata_accessors=#<LogStash::Util::Accessors:0x3fff4430 @store={}, @lut={}>,
 @cancelled=false, @data={"LogLevel"=>"INFO", "StartTime"=>"2016-01-05 14:39:40.2012272", "ExecutionTime"=>0, "CallingMethod"=>"SetPars", "Correlation"=>{"CorrelationId"=>"60e9d53e-e936-4e87-a47f-e009d4e3f3fb", "SequenceNo"=>1}, "Inputs"=>["39b80f66-8547-4a3e-b9e1-e69a2d56329b", [{"Value"=>"IBr", "Name"=>"BrName", "Status"=>0}, {"Value"=>101, "Name"=>"BrCode", "Status"=>0}]], "@version"=>"1", "@timestamp"=>"2016-01-06T09:35:15.001Z", "host"=>"DV01LPT09"}, 
@metadata={}, @accessors=#<LogStash::Util::Accessors:0x6f4b1f0e @store={"LogLevel"=>"INFO", "StartTime"=>"2016-01-05 14:39:40.2012272", "ExecutionTime"=>0, "CallingMethod"=>"SetPars", "Correlation"=>{"CorrelationId"=>"60e9d53e-e936-4e87-a47f-e009d4e3f3fb", "SequenceNo"=>1}, "Inputs"=>["39b80f66-8547-4a3e-b9e1-e69a2d56329b", [{"Value"=>"IBr", "Name"=>"BrName", "Status"=>0}, {"Value"=>101, "Name"=>"BrCode", "Status"=>0}]], "@version"=>"1", "@timestamp"=>"2016-01-06T09:35:15.001Z", "host"=>"DV01LPT09"},
@lut={"host"=>[{"LogLevel"=>"INFO", "StartTime"=>"2016-01-05 14:39:40.2012272","ExecutionTime"=>0, "CallingMethod"=>"SetPars", "Correlation"=>{"CorrelationId"=>"60e9d53e-e936-4e87-a47f-e009d4e3f3fb", "SequenceNo"=>1}, "Inputs"=>["39b80f66-8547-4a3e-b9e1-e69a2d56329b", [{"Value"=>"IBr", "Name"=>"BrName", "Status"=>0}, {"Value"=>101, "Name"=>"BrCode", "Status"=>0}]], "@version"=>"1", "@timestamp"=>"2016-01-06T09:35:15.001Z", "host"=>"DV01LPT09"}, "host"],
 "type"=>[{"LogLevel"=>"INFO", "StartTime"=>"2016-01-05 14:39:40.2012272", "ExecutionTime"=>0, "CallingMethod"=>"SetPars", "Correlation"=>{"CorrelationId"=>"60e9d53e-e936-4e87-a47f-e009d4e3f3fb", "SequenceNo"=>1}, "Inputs"=>["39b80f66-8547-4a3e-b9e1-e69a2d56329b", [{"Value"=>"IBr", "Name"=>"BrName", "Status"=>0}, {"Value"=>101, "Name"=>"BrCode", "Status"=>0}]], "@version"=>"1", "@timestamp"=>"2016-01-06T09:35:15.001Z", "host"=>"DV01LPT09"}, "type"]}>>],
 :response=>{"create"=>{"_index"=>"logstash-2016.01.06", "_type"=>"logs", "_id"=>"AVIWSlGY42d1jACX8Te2", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Merging dynamic updates triggered a conflict: mapper [Inputs] of different type, current_type [string], merged_type [ObjectMapper]"}}}, :level=>:warn}←[0m

```

from what i understand the issue came from Inputs object, which is an generic array:

```
"Inputs": ["39b80f66-8547-4a3e-b9e1-e69a2d56329b",
	[{
		"Value": "IBr",
		"Name": "BrName",
		"Status": 0
	},
	{
		"Value": 101,
		"Name": "BrCode",
		"Status": 0
	}]]

```

But I have no ideea how to fix it.

Do you have any ideea?

Thanks,  
Ovidiu  
P.S. I really like this comments framework, is an open source one? 🙂

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 6, 2016, 11:15am UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/9 "2016-01-06T11:15:54Z")

</div>

Here's the error:

> "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"Merging dynamic updates triggered a conflict: mapper [Inputs] of different type, current\_type [string], merged\_type [ObjectMapper]"}

This means that the `Inputs` field has been mapped as a string in the destination index but you're suddenly trying to turn the field into an object. Once set mappings are fixed for an index. If you're just playing around, perhaps you can just drop the index and start over?

---

<div class="post-metadata">

**Author:** ![djvidov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djvidov/32/6135_2.png) [@djvidov](https://discuss.elastic.co/u/djvidov)\
**Post date:** [January 6, 2016, 11:32am UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/10 "2016-01-06T11:32:45Z")

</div>

Thank you Magnus for answer!

I saw the error, but from my point of view `Inputs` is an object.  
I have delete the index, and when I have try it again with the same message I receive the same error:

> "reason"=\>"Merging dynamic updates triggered a conflict: mapper [Inputs] of different type, current\_type [string], merged\_type [ObjectMapper]"}}}, :level=\>:warn}←[0m

There is a way to mark `Inputs` as an object?

Thanks,  
Ovidiu

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 6, 2016, 11:41am UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/11 "2016-01-06T11:41:56Z")

</div>

You can explicitly set the mappings for the index, but since ES by default sets the mapping of a field based on the the first document containing the field it seems that you're first creating a document where the field is a string (creating a string mapping for the field) and afterwards try to create the document where the field is an object.

---

<div class="post-metadata">

**Author:** ![djvidov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djvidov/32/6135_2.png) [@djvidov](https://discuss.elastic.co/u/djvidov)\
**Post date:** [January 6, 2016, 1:27pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/12 "2016-01-06T13:27:24Z")

</div>

I'm confused.

At this moment the only sure is I didn't create the index with Input field as string because Input field in my code is always an array of params.

1. There is a documentations how to set Inputs field as object instead of string?
2. Why is necessary to map a message if elasticsearch is based on a document database? I cannot save any type of data into an index?

thank you!  
Ovidiu

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [January 6, 2016, 1:36pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/13 "2016-01-06T13:36:29Z")

</div>

When indexing into elasticsearch you have to normalize your data to have a common format. It would be easiest, if the application producing the log has a 'common' scheme for it's output. Alternatively you can try to create a workaround by renaming the 'Inputs' field depending on the type. E.g. if 'Inputs' is a string, rename 'Inputs' to 'message'. You can also try to [json encode the 'Input' field](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json_encode.html), so it's always a string.

---

<div class="post-metadata">

**Author:** ![djvidov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djvidov/32/6135_2.png) [@djvidov](https://discuss.elastic.co/u/djvidov)\
**Post date:** [January 6, 2016, 1:51pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/14 "2016-01-06T13:51:37Z")

</div>

Hi Steffens,

I agree with you about normalization, and at this moment I have all the messages normalized because right now all the messages are saved into a RDBMS, but now I try to put them into elasticsearch in order to improve search. But I don't understand why Inputs is mapped as string even if it is an array object.  
I found mapping for my index, which looks like that:

> ```
> {
> "logstash-2016.01.06": {
> "aliases": {},
> "mappings": {
> "logs": {
> "_all": {
> "enabled": true,
> "omit_norms": true
> },
> "dynamic_templates": [
> {
> "message_field": {
> "mapping": {
> "fielddata": {
> "format": "disabled"
> },
> "index": "analyzed",
> "omit_norms": true,
> "type": "string"
> },
> "match": "message",
> "match_mapping_type": "string"
> }
> },
> {
> "string_fields": {
> "mapping": {
> "fielddata": {
> "format": "disabled"
> },
> "index": "analyzed",
> "omit_norms": true,
> "type": "string",
> "fields": {
> "raw": {
> "ignore_above": 256,
> "index": "not_analyzed",
> "type": "string",
> "doc_values": true
> }
> }
> },
> "match": "*",
> "match_mapping_type": "string"
> }
> },
> 
> ```

And also I found this article, which I hope it will helps me to change `Inputs` from string to object: [Update mapping API | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html)

Regards,  
Ovidiu

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [January 6, 2016, 2:29pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/15 "2016-01-06T14:29:09Z")

</div>

How is the mapping generated?

You might also want to check out [index templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html). Some example using indices templates with mappings in [packetbeat](https://github.com/elastic/beats/blob/master/packetbeat/etc/packetbeat.template.json) and [topbeat](https://github.com/elastic/beats/blob/master/topbeat/etc/topbeat.template.json).

---

<div class="post-metadata">

**Author:** ![djvidov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djvidov/32/6135_2.png) [@djvidov](https://discuss.elastic.co/u/djvidov)\
**Post date:** [January 6, 2016, 2:37pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/16 "2016-01-06T14:37:05Z")

</div>

I don't generate any map, today I discover that I can set this map before I insert some data into index.  
The map is generated at fist insert. 🙂

I'll check your links.

Thank you!  
Ovidiu

---

<div class="post-metadata">

**Author:** ![djvidov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djvidov/32/6135_2.png) [@djvidov](https://discuss.elastic.co/u/djvidov)\
**Post date:** [January 8, 2016, 10:17am UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/17 "2016-01-08T10:17:24Z")

</div>

Hello,

I had finished the task. instead to change the logstash conf I had changed the generated json from log file.

Thanks for your support.  
Ovidiu

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:56pm UTC](https://discuss.elastic.co/t/filebeat-send-a-json-log-but-is-stored-as-a-string/38410/18 "2017-07-05T21:56:55Z")

</div>