# Read JSON with Logstash

**URL:** <https://discuss.elastic.co/t/read-json-with-logstash/70931>\
**Category:** Logstash\
**Created:** [January 9, 2017, 9:18am UTC](https://discuss.elastic.co/t/read-json-with-logstash/70931 "2017-01-09T09:18:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![srneodani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srneodani/32/10011_2.png) [@srneodani](https://discuss.elastic.co/u/srneodani)\
**Post date:** [January 9, 2017, 9:18am UTC](https://discuss.elastic.co/t/read-json-with-logstash/70931/1 "2017-01-09T09:18:55Z")

</div>

Hi people!

I want parser a json file and put output in a Elastic every minute, but I don't want add the same json each time.  
I would like if json have 10 element, have only this 10 elements in my index of Elastic but updated each minute, I mean update element if exist and only add if no exist.  
I was investigated about upsert but I don't get it.

Versions I'm using: Logstash 2.4 and Elastic Search 2.2

This is my **input json input**  
[http://pastebin.com/z6KHhgYi](http://pastebin.com/z6KHhgYi)

And this is my **logstash config**  
input {  
http\_poller {  
urls =\> {  
k8s\_namespaces =\> "[http://172.23.0.3:8080/api/v1/namespaces](http://172.23.0.3:8080/api/v1/namespaces)"  
}  
# Maximum amount of time to wait for a request to complete all  
request\_timeout =\> 30  
# How far apart requests should be  
interval =\> 60  
# Decode the results as JSON  
codec =\> "json"  
# Store metadata about the request in this key  
metadata\_target =\> "k8s"  
}  
}

```
filter {
  split {
  field => "items"
  }
}

output {
  elasticsearch{
    action => "update"
    codec => "json"
    hosts => "172.16.230.48"
    document_id => "%{items.metadata.name}"
    index => "test"
    upsert => ' {"name" : "%{items.metadata.name}" }'
    doc_as_upsert => true
  }

```

**And this is the error:**  
Attempted to send a bulk request to Elasticsearch configured at '["[http://172.25.230.48:9200](http://172.25.230.48:9200)"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided? {:error\_message=\>"[400] {"error":{"root\_cause":[{"type":"illegal\_argument\_exception","reason":"Malformed action/metadata line [1], expected a simple value for field [\_upsert] but found [START\_OBJECT]"}],"type":"illegal\_argument\_exception","reason":"Malformed action/metadata line [1], expected a simple value for field [\_upsert] but found [START\_OBJECT]"},"status":400}", :error\_class=\>"Elasticsearch::Transport::Transport::Errors::BadRequest", :backtrace=\>["/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:201:in `__raise_transport_error'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:312:in`perform\_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in`perform\_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in`non\_threadsafe\_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http\_client.rb:38:in `bulk'", "org/jruby/ext/thread/Mutex.java:149:in`synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http\_client.rb:38:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in`safe\_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in`retrying\_submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in`each\_slice'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/output_delegator.rb:130:in`worker\_multi\_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/output\_delegator.rb:114:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:301:in`output\_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:301:in`output\_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:201:in`start\_workers'"], :level=\>:error}

Can you help me?

Thank you

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 9, 2017, 9:52am UTC](https://discuss.elastic.co/t/read-json-with-logstash/70931/2 "2017-01-09T09:52:00Z")

</div>

It might not make a difference to this particular problem, but you need to change all occurrences of `%{items.metadata.name}` to `%{[items][metadata][name]}`.

---

<div class="post-metadata">

**Author:** ![srneodani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srneodani/32/10011_2.png) [@srneodani](https://discuss.elastic.co/u/srneodani)\
**Post date:** [January 9, 2017, 10:13am UTC](https://discuss.elastic.co/t/read-json-with-logstash/70931/3 "2017-01-09T10:13:58Z")

</div>

Thanks, I tried but still not working...

```
  elasticsearch{
    action => "update"
    codec => "json"
    hosts => "172.25.230.48"
    document_id => "%{[items][metadata][name]}"
    index => "test"
    upsert => ' {
    "name" : "%{[items][metadata][name]}"
    }'
    doc_as_upsert => true
  }

```

With the same result:  
ttempted to send a bulk request to Elasticsearch configured at '["[http://172.25.230.48:9200](http://172.25.230.48:9200)"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided? {:error\_message=\>"[400] {"error":{"root\_cause":[{"type":"illegal\_argument\_exception","reason":"Malformed action/metadata line [1], expected a simple value for field [\_upsert] but found [START\_OBJECT]"}],"type":"illegal\_argument\_exception","reason":"Malformed action/metadata line [1], expected a simple value for field [\_upsert] but found [START\_OBJECT]"},"status":400}", :error\_class=\>"Elasticsearch::Transport::Transport::Errors::BadRequest", :backtrace=\>["/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:201:in `__raise_transport_error'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:312:in`perform\_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in`perform\_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in`non\_threadsafe\_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http\_client.rb:38:in `bulk'", "org/jruby/ext/thread/Mutex.java:149:in`synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http\_client.rb:38:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in`safe\_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in`retrying\_submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in`each\_slice'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/output_delegator.rb:130:in`worker\_multi\_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/output\_delegator.rb:114:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:301:in`output\_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:301:in`output\_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:201:in`start\_workers'"], :level=\>:error}

Any other suggestion?

Thank you

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 9, 2017, 10:22am UTC](https://discuss.elastic.co/t/read-json-with-logstash/70931/4 "2017-01-09T10:22:23Z")

</div>

Can you try removing the JSON codec? The Elasticsearch output plugin converts to JSON automatically, so I do not believe this codec should be present.

---

<div class="post-metadata">

**Author:** ![srneodani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srneodani/32/10011_2.png) [@srneodani](https://discuss.elastic.co/u/srneodani)\
**Post date:** [January 9, 2017, 10:41am UTC](https://discuss.elastic.co/t/read-json-with-logstash/70931/5 "2017-01-09T10:41:19Z")

</div>

Thanks, I remove the line, but must be something else, because still show the same error.

```
  elasticsearch{
    action => "update"
# codec => "json"
    hosts => "172.25.230.48"
    document_id => "%{[items][metadata][name]}"
    index => "test"
    upsert => ' {
    "name" : "%{[items][metadata][name]}"
    }'
    doc_as_upsert => true
  }

```

Any suggestion?

Thank you very much

---

<div class="post-metadata">

**Author:** ![srneodani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srneodani/32/10011_2.png) [@srneodani](https://discuss.elastic.co/u/srneodani)\
**Post date:** [January 9, 2017, 11:38am UTC](https://discuss.elastic.co/t/read-json-with-logstash/70931/6 "2017-01-09T11:38:50Z")

</div>

I get it, I have to change doc\_as\_upsert to false.

Now I have a problem, only update the "news changes" ok, but if not exist the item in my json anymore I continue viewing in my elastic index.

How can save in elastic the last update/version of my JSON file?

The only way to do this is to delete the entire index before each update? Its quite radical no? I'm very stuck with this!

Any suggestions?

Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 6, 2017, 11:38am UTC](https://discuss.elastic.co/t/read-json-with-logstash/70931/7 "2017-02-06T11:38:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.