# Upsert with logstash

**URL:** <https://discuss.elastic.co/t/upsert-with-logstash/59116>\
**Category:** Logstash\
**Created:** [August 28, 2016, 2:27pm UTC](https://discuss.elastic.co/t/upsert-with-logstash/59116 "2016-08-28T14:27:12Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![vtangutoori](https://avatars.discourse-cdn.com/v4/letter/v/ee7513/32.png) [@vtangutoori](https://discuss.elastic.co/u/vtangutoori)\
**Post date:** [August 28, 2016, 2:27pm UTC](https://discuss.elastic.co/t/upsert-with-logstash/59116/1 "2016-08-28T14:27:12Z")

</div>

Hi,  
I am looking to upsert a value to a field in existing document in an array, i have the below config file but the system is throwing error with message shown after config file, can anyone please help.

File:  
input {  
file {  
path =\> ["C:\ElasticSearch\logstash-2.3.4\bin\testspecialty.csv"]  
type =\> "csv"  
start\_position =\> "beginning"  
}  
}

filter {  
csv {  
columns =\> ["Provider\_Id","Secondary\_Specialty"]  
separator =\> ","  
}  
}  
output {  
elasticsearch {  
action =\> "update"  
hosts =\> "localhost:9200"  
index =\> "testsearchresults"  
document\_id =\> "%{Provider\_Id}"  
document\_type =\> "testresults"  
upsert =\> {  
"document\_id" : "%{provider\_id}",  
"Secondary\_Specialty\_String" :["%{Secondary\_Specialty}"]  
}  
manage\_template =\> true  
}

stdout { codec =\> "rubydebug" }  
}  
Error:  
fetched an invalid config {:config=\>"\ninput { \n file {\n path =\> ["C:\ElasticSearch\logstash-2.3.4\bin\testspecialty.csv"]\n type =\> "csv"\n start\_position =\> "beginning"\n }\n}\n\nfilter { \n csv {\n columns =\> ["Provider\_Id","Secondary\_Specialty"]\n separator =\> ","\n }\n}\noutput {\nelasticsearch {\n action =\> "update"\n hosts =\> "localhost:9200"\n index =\> "testsearchresults"\n document\_id =\> "%{Provider\_Id}"\n document\_type =\> "testresults"\n upsert =\> {\n "document\_id" : "%{provider\_id}",\n "Secondary\_Specialty\_String" :["%{Secondary\_Specialty}"]\n }\n manage\_template =\> true\n }\n \nstdout { }\n}\n", :reason=\>"Expected one of #, =\> at line 24, column 19 (byte 485) after output {\nelasticsearch {\n action =\> "update"\n hosts =\> "localhost:9200"\n index =\> "testsearchresults"\n document\_id =\> "%{Provider\_Id}"\n document\_type =\> "testresults"\n upsert =\> {\n "document\_id" ", :level=\>:error, :file=\>"/ElasticSearch/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb", :line=\>"430", :method=\>"create\_pipeline"}  
starting agent {:level=\>:info, :file=\>"/ElasticSearch/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb", :line=\>"207", :method=\>"execute"}  
The signal HUP is in use by the JVM and will not work correctly on this platform

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 29, 2016, 5:44am UTC](https://discuss.elastic.co/t/upsert-with-logstash/59116/2 "2016-08-29T05:44:14Z")

</div>

According to the docs the `upsert` option is supposed to be a string, and even if hashes are okay they would have to look like this:

```nohighlight
upsert => {
  "document_id" => "%{provider_id}"
  "Secondary_Specialty_String" => ["%{Secondary_Specialty}"]
}

```

---

<div class="post-metadata">

**Author:** ![vtangutoori](https://avatars.discourse-cdn.com/v4/letter/v/ee7513/32.png) [@vtangutoori](https://discuss.elastic.co/u/vtangutoori)\
**Post date:** [August 29, 2016, 8:31pm UTC](https://discuss.elastic.co/t/upsert-with-logstash/59116/3 "2016-08-29T20:31:18Z")

</div>

Hi Magnus,

I have made the changes as you said and tried the config file again but it gave me the below error. I tried searching a fix for it but couldn't find any, can you please help

Registering file input {:path=\>["C:\ElasticSearch\logstash-2.3.4\bin\testspecialty.csv"], :level=\>:info, :file=\>"/ElasticSearch/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.5/lib/logstash/inputs/file.rb", :line=\>"171", :method=\>"register"}  
No sincedb\_path set, generating one based on the file path {:sincedb\_path=\>"C:\Users\vamsi/.sincedb\_469ac9bb1d0de3017c85ea3629a99535", :path=\>["C:\ElasticSearch\logstash-2.3.4\bin\testspecialty.csv"], :level=\>:info, :file=\>"/ElasticSearch/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.5/lib/logstash/inputs/file.rb", :line=\>"216", :method=\>"register"}  
Invalid setting for elasticsearch output plugin:

output {  
elasticsearch {  
# This setting must be a string  
# Expected string, got {"document\_id"=\>"%{provider\_id}", "Secondary\_Specialty\_String"=\>["%{Secondary\_Specialty}"]}  
upsert =\> {"document\_id"=\>"%{provider\_id}", "Secondary\_Specialty\_String"=\>["%{Secondary\_Specialty}"]}  
...  
}  
} {:level=\>:error, :file=\>"/ElasticSearch/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/config/mixin.rb", :line=\>"374", :method=\>"validate\_check\_parameter\_values"}

---

<div class="post-metadata">

**Author:** ![pramod\_kumar](https://avatars.discourse-cdn.com/v4/letter/p/97f17d/32.png) [@pramod\_kumar](https://discuss.elastic.co/u/pramod_kumar)\
**Post date:** [September 10, 2016, 12:41pm UTC](https://discuss.elastic.co/t/upsert-with-logstash/59116/4 "2016-09-10T12:41:38Z")

</div>

Can you please tell me how did you resolve the above issue ??

---

<div class="post-metadata">

**Author:** ![pramod\_kumar](https://avatars.discourse-cdn.com/v4/letter/p/97f17d/32.png) [@pramod\_kumar](https://discuss.elastic.co/u/pramod_kumar)\
**Post date:** [September 10, 2016, 12:43pm UTC](https://discuss.elastic.co/t/upsert-with-logstash/59116/5 "2016-09-10T12:43:03Z")

</div>

Hi,

Am also facing the same issue with the Upsert.  
Can you please tell me how did you resolve the above issue ??

Thanks  
Pramod

---

<div class="post-metadata">

**Author:** ![boubou191911](https://avatars.discourse-cdn.com/v4/letter/b/51bf81/32.png) [@boubou191911](https://discuss.elastic.co/u/boubou191911)\
**Post date:** [June 20, 2017, 6:29am UTC](https://discuss.elastic.co/t/upsert-with-logstash/59116/6 "2017-06-20T06:29:02Z")

</div>

Hello,  
The only way I found to fill in the upsert field in elasticsearch output is to use the json\_encode filter. This creates a string that I pass in the upsert.

Hope it can help someone...

---

<div class="post-metadata">

**Author:** ![mbertani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mbertani/32/11398_2.png) [@mbertani](https://discuss.elastic.co/u/mbertani)\
**Post date:** [June 26, 2017, 12:58pm UTC](https://discuss.elastic.co/t/upsert-with-logstash/59116/7 "2017-06-26T12:58:53Z")

</div>

Why don't you try

```
elasticsearch {
		hosts => ["localhost:9200"]
		index => "testsearchresults"
		document_id => "%{Provider_Id}"
        doc_as_upsert => true
        action => "update"
        manage_template => true
	}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 4:27am UTC](https://discuss.elastic.co/t/upsert-with-logstash/59116/8 "2022-11-04T04:27:13Z")

</div>