Problem with mapping, upserts and multiple configs

(Gerry Riddell) #1

ES 1.7.1
LS 1.5.4

I am trying the upsert feature and it does not work as I expected (but maybe my expectation is wrong). Here is what I am doing:

I created a mapping file for my elasticsearch index. (Let's say it has 10 fields.). I do a post against the index to create the mapping:

{ "mappings": {
        "well": {
            "_all": {
                "enabled": true
            "properties": {
                        ... 10 fields detailed here ...

I am indexing my documents with logstash (coming in via the jdbc input) using multiple configs to do the import. (Let's say config1 brings in 5 fields, config2 brings in 3 fields, and config3 brings in 2 fields.) I know the document Id and use that across all the config files. I am needing these to be separate updates due to inconsistent updates of information in the different oracle tables.

Basically, Elasticsearch is my consolidated document.

I am using the following elasticsearch output conf for the three configs:

elasticsearch {
        host => "localhost"
        protocol => "http"
        index => "wells"
        action => "update"
        document_type => "well"
        document_id =>  "%{[master_well_interval_id]}"
        doc_as_upsert => true
        manage_template => false

Now, what happens is, when the document gets updated by (let's use) config1, the mapping gets changed to only having those fields that were specified (i.e. for config1, that is 5 fields). The others get removed from the mapping (i.e. in this example, 5 fields).

If I remove the action => "update" part from the config, then the mapping stays as it was originally configured, but the document gets overritten each time a config updates that document.

So, is this working as advertised? I would expect my configuration to keep my mapping, but that is not what I am experiencing.

I remember seeing somewhere that update actions don't work against the http protocol, but I thought that was fixed in LS 1.5.4.

Any suggestions?

Thanks in advance!!

(system) #2