# Define upload fields from json file

**URL:** <https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938>\
**Category:** Logstash\
**Created:** [February 28, 2016, 9:40am UTC](https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938 "2016-02-28T09:40:44Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![chenbe2204](https://avatars.discourse-cdn.com/v4/letter/c/c5a1d2/32.png) [@chenbe2204](https://discuss.elastic.co/u/chenbe2204)\
**Post date:** [February 28, 2016, 9:40am UTC](https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938/1 "2016-02-28T09:40:44Z")

</div>

Hi ,

In the conf file , how can I define which fields I want to upload to the ElasticSearch ?  
My Json file contain one row that need to be uploaded to the ElasticSearch , So after the upload I would like to see I row under the index with the fields and values compatibility.

For example , this is my conf file:  
input {  
file{  
path =\> ["/tmp/y4.json"]  
type =\> "json"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
}  
}  
filter{  
grok {  
match =\> ['message', '(?"TestName":.\*"Agent":"[^"]+")' ]  
}  
json {  
source =\> "message"  
}  
}

output {  
stdout {  
codec =\> rubydebug  
}  
elasticsearch {  
host =\> "[XX.XX.XX.XXX](http://XX.XX.XX.XXX)"  
protocol =\> "http"  
index =\> "index\_junit"  
}  
}

I want the following fields from the Json file:  
queueId , timestamp , startTime , result , duration , charset

BR,  
Chen

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 28, 2016, 11:11am UTC](https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938/2 "2016-02-28T11:11:31Z")

</div>

> In the conf file , how can I define which fields I want to upload to the Elasticsearch ?

Elasticsearch gets the whole event (except the `@metadata` field). If you _don't_ want to include particular fields you have to delete them with e.g. the [mutate](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html) or the [prune](https://www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html) filter.

---

<div class="post-metadata">

**Author:** ![chenbe2204](https://avatars.discourse-cdn.com/v4/letter/c/c5a1d2/32.png) [@chenbe2204](https://discuss.elastic.co/u/chenbe2204)\
**Post date:** [February 28, 2016, 11:31am UTC](https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938/3 "2016-02-28T11:31:38Z")

</div>

Hi ,

When I'm uploading a flat Json file (All tags in level 1) I don't have a problem.  
When I'm uploading a complex Json (Tags with more then 1 level) , I have a problem. I don't have the fields I need , only message and in some cases the value of message is "{" or "}",

BR,  
Chen

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 28, 2016, 11:37am UTC](https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938/4 "2016-02-28T11:37:16Z")

</div>

Please show an example so that it's possible to understand what's going on.

Coming back to your original post,

> match =\> ['message', '(?"TestName":.\*"Agent":"[^"]+")' ]

why are you using grok to parse JSON? You're even already using a json filter for that.

---

<div class="post-metadata">

**Author:** ![chenbe2204](https://avatars.discourse-cdn.com/v4/letter/c/c5a1d2/32.png) [@chenbe2204](https://discuss.elastic.co/u/chenbe2204)\
**Post date:** [February 28, 2016, 12:02pm UTC](https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938/5 "2016-02-28T12:02:15Z")

</div>

In Stack Overflow

> <https://stackoverflow.com/questions/25977423/import-json-files-into-logstash-elasticsearch-kibana>

  
It's written that it won't work well with nested JSON structs and I need a simple hash of key/value pairs.  
Is there a way to upload nested JSON structs ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 28, 2016, 2:39pm UTC](https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938/6 "2016-02-28T14:39:51Z")

</div>

Kibana can't deal with arrays of objects, but objects containing other objects are fine.

---

<div class="post-metadata">

**Author:** ![chenbe2204](https://avatars.discourse-cdn.com/v4/letter/c/c5a1d2/32.png) [@chenbe2204](https://discuss.elastic.co/u/chenbe2204)\
**Post date:** [March 2, 2016, 2:30pm UTC](https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938/7 "2016-03-02T14:30:11Z")

</div>

Thanks 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:08am UTC](https://discuss.elastic.co/t/define-upload-fields-from-json-file/42938/8 "2017-07-06T05:08:48Z")

</div>