# Logstash doc\_as\_upsert =\> true not updating few fields sometimes **URL:** \ **Category:** Logstash\ **Created:** [May 25, 2018, 9:20am UTC](https://discuss.elastic.co/t/logstash-doc-as-upsert-true-not-updating-few-fields-sometimes/133293 "2018-05-25T09:20:04Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![shoaibjdev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shoaibjdev/32/28124_2.png) [@shoaibjdev](https://discuss.elastic.co/u/shoaibjdev)\ **Post date:** [May 25, 2018, 9:20am UTC](https://discuss.elastic.co/t/logstash-doc-as-upsert-true-not-updating-few-fields-sometimes/133293/1 "2018-05-25T09:20:04Z")
Hi, Have came across a strange issue where in if we are sending the doc\_as\_upsert =\> true for the data to be updated it doesn't update few fields at random. For example if I have 1000 records which after insert are updating a field "record\_status" from PENDING to SUCCESS, it doesn't update for 4-5 records which brings inconsistency and also concern regarding reliability of ELB stack that we planning to use. **Logstash Version:** logstash-6.2.2 **Elasticsearch Version:** elasticsearch-6.2.2 **Logstash Config:** ``` input { beats { port => 5044 } } filter { if "karaf_logs" in [tags] { grok{ match => ['message', "%{TIMESTAMP_ISO8601:tm} \| %{LOGLEVEL:loglevel} \| %{GREEDYDATA:queue_name} \| %{GREEDYDATA:route_name} \| %{GREEDYDATA:thread_info} \| %{GREEDYDATA:content}"] break_on_match => false } } else if "k8s_logs" in [tags] { grok{ match => ['message', "%{GREEDYDATA:raw_date} %{TIME:tm1} %{GREEDYDATA:hostname} journal: %{TIME:tm2} \[.+\] %{LOGLEVEL:loglevel} %{GREEDYDATA:content}"] break_on_match => false } } grok{ match => ["content", '(?(?<=\[).*?(?=\]))' ] match => ["content", '(?({.*$))'] break_on_match => false } json { source => "json_in" remove_field => "json_in" } # --------- Setup ES ID Field (This can only be setup after json filter hence not included in previous check) if ([EXCHANGE_ID]) { mutate { add_field => { "es_id" => "%{EXCHANGE_ID}" } } } else { if [user_reference] { mutate { add_field => { "es_id" => "%{user_reference}" } } } else { mutate { add_field => { "es_id" => "%{exchange_id}" } } } } # ------- Remove unwanted fields mutate { remove_field => "message" remove_field => "content" remove_field => "beat" remove_field => "raw_date" remove_field => "offset" remove_field => "prospector" remove_field => "tm1" remove_field => "tm2" } } output { stdout { codec => rubydebug } elasticsearch { hosts => ["0.0.0.0:9200"] index => "logstash-data-monitor" action => "update" doc_as_upsert => true document_id => "%{[es_id]}" } } ``` **Sample data which updated correctly:** _Insert_ > { > "host" =\> "k8s\_v5\_node1", > "loglevel" =\> "INFO", > "@timestamp" =\> 2018-05-25T07:17:47.629Z, > "type" =\> "KE-az-prc-mno-telco-572202", > "hostname" =\> "k8s\_v5\_node1", > "@version" =\> "1", > "user\_reference" =\> "442E9608443C469B9C5BBD072182F224", > "source" =\> "/var/log/messages", > "tags" =\> [ > [0] "k8s\_logs", > [1] "beats\_input\_codec\_plain\_applied" > ], > "exchange\_start\_time" =\> "Wed May 23 20:47:16 GMT 2018", > "es\_id" =\> "442E9608443C469B9C5BBD072182F224", > "exchange\_id" =\> "ID-az-prc-mno-2567439494-1vq68-36509-1527085265398-0-357", > "record\_status" =\> "INITIATED" > } _Update:_ > { > "telco\_bill\_reference" =\> "0060034001", > "loglevel" =\> "INFO", > "transaction\_amount" =\> "82127.00", > "telco\_reference" =\> "XEN3KG0103", > "source" =\> "/var/log/messages", > "transaction\_type" =\> "CBS\_POSTING", > "host" =\> "k8s\_v5\_node1", > "credit\_account" =\> "5029728001", > "customer\_name" =\> "LOCAL MERCHANT 1", > "@timestamp" =\> 2018-05-25T07:17:47.629Z, > "telco\_transaction\_type" =\> "", > "type" =\> "KE-az-prc-mno-telco-572202", > "telco\_short\_code" =\> "572202", > "hostname" =\> "k8s\_v5\_node1", > "@version" =\> "1", > "user\_reference" =\> "442E9608443C469B9C5BBD072182F224", > "cbs\_reference" =\> "000ADDC181430526", > "tags" =\> [ > [0] "k8s\_logs", > [1] "beats\_input\_codec\_plain\_applied" > ], > "debit\_account" =\> "AS378601", > "es\_id" =\> "442E9608443C469B9C5BBD072182F224", > "record\_status" =\> "SUCCESS" > } **Sample Data which didn't updated correctly:** _Insert:_ > { > "host" =\> "k8s\_v5\_node1", > "loglevel" =\> "INFO", > "@timestamp" =\> 2018-05-25T07:17:48.075Z, > "type" =\> "KE-az-prc-mno-telco-572202", > "hostname" =\> "k8s\_v5\_node1", > "@version" =\> "1", > "user\_reference" =\> "F07C156EB57048A28645955EDB49E27A", > "source" =\> "/var/log/messages", > "tags" =\> [ > [0] "k8s\_logs", > [1] "beats\_input\_codec\_plain\_applied" > ], > "exchange\_start\_time" =\> "Thu May 24 05:52:06 GMT 2018", > "es\_id" =\> "F07C156EB57048A28645955EDB49E27A", > "exchange\_id" =\> "ID-az-prc-mno-2567439494-1vq68-36509-1527085265398-0-722", > "record\_status" =\> "INITIATED" > } _Update:_ > { > "telco\_bill\_reference" =\> "5029728001", > "loglevel" =\> "INFO", > "transaction\_amount" =\> "235033.00", > "telco\_reference" =\> "XEO8KI9009", > "source" =\> "/var/log/messages", > "transaction\_type" =\> "CBS\_POSTING", > "host" =\> "k8s\_v5\_node1", > "credit\_account" =\> "5029728001", > "customer\_name" =\> "MERCHANT 773", > "@timestamp" =\> 2018-05-25T07:17:48.076Z, > "telco\_transaction\_type" =\> "", > "type" =\> "KE-az-prc-mno-telco-572202", > "telco\_short\_code" =\> "572202", > "hostname" =\> "k8s\_v5\_node1", > "@version" =\> "1", > "user\_reference" =\> "F07C156EB57048A28645955EDB49E27A", > "cbs\_reference" =\> "000FTMP181440151", > "tags" =\> [ > [0] "k8s\_logs", > [1] "beats\_input\_codec\_plain\_applied" > ], > "debit\_account" =\> "AS378601", > "es\_id" =\> "F07C156EB57048A28645955EDB49E27A", > "record\_status" =\> "SUCCESS" > } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 22, 2018, 9:20am UTC](https://discuss.elastic.co/t/logstash-doc-as-upsert-true-not-updating-few-fields-sometimes/133293/2 "2018-06-22T09:20:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.