Problem with mapping, upserts and multiple configs


(Gerry Riddell) #1

Using:
ES 1.7.1
LS 1.5.4

I am trying the upsert feature and it does not work as I expected (but maybe my expectation is wrong). Here is what I am doing:

I created a mapping file for my elasticsearch index. (Let's say it has 10 fields.). I do a post against the index to create the mapping:

{ "mappings": {
        "well": {
            "_all": {
                "enabled": true
            },
            "properties": {
                        ... 10 fields detailed here ...
                       }
              }
       }
}

I am indexing my documents with logstash (coming in via the jdbc input) using multiple configs to do the import. (Let's say config1 brings in 5 fields, config2 brings in 3 fields, and config3 brings in 2 fields.) I know the document Id and use that across all the config files. I am needing these to be separate updates due to inconsistent updates of information in the different oracle tables.

Basically, Elasticsearch is my consolidated document.

I am using the following elasticsearch output conf for the three configs:

elasticsearch {
        host => "localhost"
        protocol => "http"
        index => "wells"
        action => "update"
        document_type => "well"
        document_id =>  "%{[master_well_interval_id]}"
        doc_as_upsert => true
        manage_template => false
    }

Now, what happens is, when the document gets updated by (let's use) config1, the mapping gets changed to only having those fields that were specified (i.e. for config1, that is 5 fields). The others get removed from the mapping (i.e. in this example, 5 fields).

If I remove the action => "update" part from the config, then the mapping stays as it was originally configured, but the document gets overritten each time a config updates that document.

So, is this working as advertised? I would expect my configuration to keep my mapping, but that is not what I am experiencing.

I remember seeing somewhere that update actions don't work against the http protocol, but I thought that was fixed in LS 1.5.4.

Any suggestions?

Thanks in advance!!
Gerry.


(system) #2