Using:
ES 1.7.1
LS 1.5.4
I am trying the upsert feature and it does not work as I expected (but maybe my expectation is wrong). Here is what I am doing:
I created a mapping file for my elasticsearch index. (Let's say it has 10 fields.). I do a post against the index to create the mapping:
{ "mappings": {
"well": {
"_all": {
"enabled": true
},
"properties": {
... 10 fields detailed here ...
}
}
}
}
I am indexing my documents with logstash (coming in via the jdbc input) using multiple configs to do the import. (Let's say config1 brings in 5 fields, config2 brings in 3 fields, and config3 brings in 2 fields.) I know the document Id and use that across all the config files. I am needing these to be separate updates due to inconsistent updates of information in the different oracle tables.
Basically, Elasticsearch is my consolidated document.
I am using the following elasticsearch output conf for the three configs:
elasticsearch {
host => "localhost"
protocol => "http"
index => "wells"
action => "update"
document_type => "well"
document_id => "%{[master_well_interval_id]}"
doc_as_upsert => true
manage_template => false
}
Now, what happens is, when the document gets updated by (let's use) config1, the mapping gets changed to only having those fields that were specified (i.e. for config1, that is 5 fields). The others get removed from the mapping (i.e. in this example, 5 fields).
If I remove the action => "update"
part from the config, then the mapping stays as it was originally configured, but the document gets overritten each time a config updates that document.
So, is this working as advertised? I would expect my configuration to keep my mapping, but that is not what I am experiencing.
I remember seeing somewhere that update actions don't work against the http protocol, but I thought that was fixed in LS 1.5.4.
Any suggestions?
Thanks in advance!!
Gerry.