# Update Existing Log Via Logstash

**URL:** <https://discuss.elastic.co/t/update-existing-log-via-logstash/274796>\
**Category:** Logstash\
**Created:** [June 2, 2021, 7:17pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796 "2021-06-02T19:17:01Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![ankitdevnalkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitdevnalkar/32/46158_2.png) [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\
**Post date:** [June 2, 2021, 7:17pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/1 "2021-06-02T19:17:01Z")

</div>

I want to update an existing document in Elasticsearch based on certain conditions. However, I am facing some errors.

**Here is my filter plugin configuration**

```
filter
{
  json { source => "message" }

  if [baseValueUnitAmount] > 968
  {
   mutate 
   { 
    add_field => {"log" => "update"}
    add_field => {"status" => "this is a updated log"}
   }
   elasticsearch {
      hosts => ["es_host:9200"]
      query => "myrefid:%{[myrefid]}"
      fields => { "_id" => "doc_id" }
      index => "replacement-test"
      ssl => true
      user => 'myuser'
      password => 'mypassword'
   }
  }
}
output
{
	stdout { codec => rubydebug }
	          if [log] == "update"
            {
            elasticsearch
            {
            codec => json
            hosts => ["es_host:9200"]
            action => "update"
            document_id => "%{[doc_id]}"
            index => "replacement-test"
            ssl => true
            ssl_certificate_verification => false
            user => 'myuser'
            password => 'mypassword'
            }  
            }
            else
            {
            elasticsearch
            {
            codec => json
            hosts => ["es_host:9200"]
            index => "replacement-test"
            ssl => true
            ssl_certificate_verification => false
            user => 'myuser'
            password => 'mypassword'
            }
            }
}

```

Here is the error I am getting :

`[ERROR] 2021-06-03 00:29:25.847 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}`

Can anyone help ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 2, 2021, 7:32pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/2 "2021-06-02T19:32:47Z")

</div>

> [@ankitdevnalkar](#):
>
> Here is the error I am getting

There should be additional log messages that explain why that happened.

---

<div class="post-metadata">

**Author:** ![ankitdevnalkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitdevnalkar/32/46158_2.png) [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\
**Post date:** [June 3, 2021, 6:05am UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/3 "2021-06-03T06:05:32Z")

</div>

@Badger thanks for responding. I see no other information about the error. adding logs here :

```
root@ip-172-31-2-250:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash -f endpoint.conf --config.reload.automatic
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO] 2021-06-03 11:22:41.645 [main] runner - Starting Logstash {"logstash.version"=>"7.9.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.292-b10 on 1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 +indy +jit [linux-x86_64]"}
[WARN] 2021-06-03 11:22:42.410 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO] 2021-06-03 11:22:47.120 [Converge PipelineAction::Create<main>] Reflections - Reflections took 69 ms to scan 1 urls, producing 22 keys and 45 values 
[WARN] 2021-06-03 11:23:21.091 [[main]-pipeline-manager] elasticsearch - **WARNING** Detected UNSAFE options in elasticsearch output configuration!
**WARNING** You have enabled encryption but DISABLED certificate verification.
**WARNING** To make sure your data is secure change :ssl_certificate_verification to true
[INFO] 2021-06-03 11:23:21.663 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@es_host:9200/]}}
[WARN] 2021-06-03 11:23:22.532 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://elastic:xxxxxx@es_host:9200/"}
[INFO] 2021-06-03 11:23:22.785 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN] 2021-06-03 11:23:22.789 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO] 2021-06-03 11:23:22.902 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//es_host:9200"]}
[WARN] 2021-06-03 11:23:22.913 [[main]-pipeline-manager] elasticsearch - **WARNING** Detected UNSAFE options in elasticsearch output configuration!
**WARNING** You have enabled encryption but DISABLED certificate verification.
**WARNING** To make sure your data is secure change :ssl_certificate_verification to true
[INFO] 2021-06-03 11:23:22.998 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@es_host:9200/]}}
[WARN] 2021-06-03 11:23:23.081 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://elastic:xxxxxx@es_host:9200/"}
[INFO] 2021-06-03 11:23:23.090 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN] 2021-06-03 11:23:23.092 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO] 2021-06-03 11:23:23.128 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//es_host:9200"]}
[INFO] 2021-06-03 11:23:23.231 [[main]-pipeline-manager] elasticsearch - New ElasticSearch filter client {:hosts=>[{:host=>"es_host:9200", :scheme=>"https"}]}
[ERROR] 2021-06-03 11:23:23.564 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[INFO] 2021-06-03 11:23:23.956 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[WARN] 2021-06-03 11:23:29.645 [[main]-pipeline-manager] elasticsearch - **WARNING** Detected UNSAFE options in elasticsearch output configuration!
**WARNING** You have enabled encryption but DISABLED certificate verification.
**WARNING** To make sure your data is secure change :ssl_certificate_verification to true
[INFO] 2021-06-03 11:23:29.660 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@es_host:9200/]}}
[WARN] 2021-06-03 11:23:29.738 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://elastic:xxxxxx@es_host:9200/"}
[INFO] 2021-06-03 11:23:29.751 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN] 2021-06-03 11:23:29.752 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO] 2021-06-03 11:23:29.840 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//es_host:9200"]}
[WARN] 2021-06-03 11:23:29.843 [[main]-pipeline-manager] elasticsearch - **WARNING** Detected UNSAFE options in elasticsearch output configuration!
**WARNING** You have enabled encryption but DISABLED certificate verification.
**WARNING** To make sure your data is secure change :ssl_certificate_verification to true
[INFO] 2021-06-03 11:23:29.847 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@es_host:9200/]}}
[WARN] 2021-06-03 11:23:29.942 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://elastic:xxxxxx@es_host:9200/"}
[INFO] 2021-06-03 11:23:29.956 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN] 2021-06-03 11:23:29.957 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO] 2021-06-03 11:23:30.031 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//es_host:9200"]}
[INFO] 2021-06-03 11:23:30.035 [[main]-pipeline-manager] elasticsearch - New ElasticSearch filter client {:hosts=>[{:host=>"es_host:9200", :scheme=>"https"}]}
[ERROR] 2021-06-03 11:23:30.134 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[WARN] 2021-06-03 11:23:32.758 [[main]-pipeline-manager] elasticsearch - **WARNING** Detected UNSAFE options in elasticsearch output configuration!
**WARNING** You have enabled encryption but DISABLED certificate verification.
**WARNING** To make sure your data is secure change :ssl_certificate_verification to true
[INFO] 2021-06-03 11:23:32.768 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@es_host:9200/]}}
[WARN] 2021-06-03 11:23:32.825 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://elastic:xxxxxx@es_host:9200/"}
[INFO] 2021-06-03 11:23:32.839 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN] 2021-06-03 11:23:32.840 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO] 2021-06-03 11:23:32.895 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//es_host:9200"]}
[WARN] 2021-06-03 11:23:32.898 [[main]-pipeline-manager] elasticsearch - **WARNING** Detected UNSAFE options in elasticsearch output configuration!
**WARNING** You have enabled encryption but DISABLED certificate verification.
**WARNING** To make sure your data is secure change :ssl_certificate_verification to true
[INFO] 2021-06-03 11:23:32.909 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@es_host:9200/]}}
[WARN] 2021-06-03 11:23:32.999 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://elastic:xxxxxx@es_host:9200/"}
[INFO] 2021-06-03 11:23:33.015 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN] 2021-06-03 11:23:33.017 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO] 2021-06-03 11:23:33.059 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//es_host:9200"]}
[INFO] 2021-06-03 11:23:33.062 [[main]-pipeline-manager] elasticsearch - New ElasticSearch filter client {:hosts=>[{:host=>"es_host:9200", :scheme=>"https"}]}
[ERROR] 2021-06-03 11:23:33.126 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 3, 2021, 3:45pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/4 "2021-06-03T15:45:46Z")

</div>

You could try setting log.level to debug, it may enable additional log messages that are helpful. I would suspect a problem with certificate keys.

---

<div class="post-metadata">

**Author:** ![ankitdevnalkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitdevnalkar/32/46158_2.png) [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\
**Post date:** [June 3, 2021, 4:38pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/5 "2021-06-03T16:38:03Z")

</div>

Here is the debug logs :

> <https://gist.github.com/ankitdevnalkar/56e969d90689c515407290abd38eee8f>

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 3, 2021, 4:47pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/6 "2021-06-03T16:47:30Z")

</div>

> [DEBUG][logstash.javapipeline][main] Pipeline terminated by worker error {:pipeline\_id=\>"main", :exception=\>java.security.cert.CertificateParsingException: signed fields invalid

That is the issue. It is indeed a [problem](https://github.com/openjdk/jdk/blob/52d8215a1ec42d67217505fe3167c70460f5a639/src/java.base/share/classes/sun/security/x509/X509CertImpl.java#L1778) with the certificate. It could be the wrong certificate format, or as I said, a problem with the key.

---

<div class="post-metadata">

**Author:** ![ankitdevnalkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitdevnalkar/32/46158_2.png) [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\
**Post date:** [June 3, 2021, 5:02pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/7 "2021-06-03T17:02:26Z")

</div>

what should be the ideal format for certificate? .p12 will work ?

Also, my ca cert is password protected, is there anyway I can pass CA cert's password ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 3, 2021, 5:27pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/8 "2021-06-03T17:27:25Z")

</div>

Looking at the [code](https://github.com/cheald/manticore/blob/79fbbc51fd5539479b4070eb66cd86de15b2ffb8/lib/manticore/client.rb#L634) I think it is expecting a PKCS#7 format for the ca\_file, but your configuration does not set ca\_file on the elasticsearch filter so I am not sure why it is going through that code at all.

I do not think that file can be password protected. It should not contain the private key.

---

<div class="post-metadata">

**Author:** ![ankitdevnalkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitdevnalkar/32/46158_2.png) [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\
**Post date:** [June 3, 2021, 5:46pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/9 "2021-06-03T17:46:59Z")

</div>

I changed filter config to :

```
   filter
    {
    elasticsearch 
       {
          hosts => ["https://es_host:9200"]
          query => "txid:%{[txid]}"
          ca_file => "/etc/logstash/elastic-stack-ca.p7b"
          ssl => true
          fields => { "_id" => "doc_id" }
          index => "replacement-test*"
          user => 'elastic'
          password => 'mypassword'
       }
    }

```

Now it is giving the below error :

```
[2021-06-03T23:05:51,440][DEBUG][logstash.javapipeline][main] Pipeline terminated by worker error {:pipeline_id=>"main", :exception=>#<Manticore::ResolutionFailure: https>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:37:in `block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:79:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:274:in `call_once'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:158:in `code'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/ping.rb:20:in `ping'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.9.0/lib/logstash/filters/elasticsearch.rb:310:in `test_connection!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.9.0/lib/logstash/filters/elasticsearch.rb:117:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:226:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:225:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:560:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:238:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:183:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/endpoint.conf"], :thread=>"#<Thread:0x22a103eb run>"}
[2021-06-03T23:05:51,450][ERROR][logstash.agent] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 3, 2021, 6:00pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/10 "2021-06-03T18:00:55Z")

</div>

That's good, it has gotten further. Manticore::ResolutionFailure is a DNS resolution failure. It is failing to do a DNS lookup of "https". [Change](https://discuss.elastic.co/t/logstash-elasticsearch-input-using-https-error/251041/2) "https://es\_host:9200" to "es\_host:9200"

---

<div class="post-metadata">

**Author:** ![ankitdevnalkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitdevnalkar/32/46158_2.png) [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\
**Post date:** [June 3, 2021, 6:18pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/11 "2021-06-03T18:18:56Z")

</div>

resolution failure got resolved and came up another one :

`[2021-06-03T23:42:32,006][DEBUG][logstash.javapipeline][main] Pipeline terminated by worker error {:pipeline_id=>"main", :exception=>#<Manticore::UnknownException: Host name 'my_es_ip' does not match the certificate subject provided by the peer (CN=elasticsearch)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:37:in `block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:79:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:274:in `call\_once'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:158:in `code'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `block in perform\_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform\_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/ping.rb:20:in `ping'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.9.0/lib/logstash/filters/elasticsearch.rb:310:in `test_connection!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.9.0/lib/logstash/filters/elasticsearch.rb:117:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:226:in `block in register\_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:225:in `register\_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java\_pipeline.rb:560:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:238:in `start\_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java\_pipeline.rb:183:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134:in `block in start'"], "pipeline.sources"=\>["/etc/logstash/conf.d/endpoint.conf"], :thread=\>"#\<Thread:0x4c47fb4b run\>"}`

---

<div class="post-metadata">

**Author:** ![ankitdevnalkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitdevnalkar/32/46158_2.png) [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\
**Post date:** [June 3, 2021, 6:30pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/12 "2021-06-03T18:30:11Z")

</div>

Going through your [this](https://discuss.elastic.co/t/logstash-unable-to-retrieve-license-information-from-license-server/264979/4) thread, would I really need to re-configure SSL throughout the Stack? 😟

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 3, 2021, 6:53pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/13 "2021-06-03T18:53:43Z")

</div>

For an elasticsearch filter, using a name-matched certificate is mandatory. For an elasticsearch output there is an option to disable name matching, but as the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ssl_certificate_verification) notes, it severely compromises security. (The PDF that documentation links to is a really interesting read.)

In general, if you cannot use a name-matched certificate I would suggest turning off TLS, because you are not getting the security that you think you are.

---

<div class="post-metadata">

**Author:** ![ankitdevnalkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitdevnalkar/32/46158_2.png) [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\
**Post date:** [June 3, 2021, 7:11pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/14 "2021-06-03T19:11:27Z")

</div>

can you tell me how can I turn TLS off? I mean from Logstash or Elasticsearch?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 3, 2021, 7:17pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/15 "2021-06-03T19:17:22Z")

</div>

This [blog](https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash) post explain how TLS is enabled on elasticsearch. Section 2-6-2 has the parameters that turn it on/off.

Once you disable TLS in elasticsearch you would set the ssl option on the elasticsearch filter to false.

HOWEVER, you would be much better off if you generated and used name-matched certificates. Generating the certificates is also covered in that blog post.

---

<div class="post-metadata">

**Author:** ![ankitdevnalkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitdevnalkar/32/46158_2.png) [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\
**Post date:** [June 4, 2021, 1:17pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/16 "2021-06-04T13:17:36Z")

</div>

Hey @Badger, I managed to replace certificates with name-matched certificates. Now it is not giving any exception related to certificate.

However, when I send a log having `baseValueUnitAmount: 980`, it throws two warnings :  
1

```
[INFO] 2021-06-04 18:31:34.837 [[main]>worker0] elasticsearch - New ElasticSearch filter client {:hosts=>[{:host=>{:host=>"es_host:9200", :scheme=>"https", :protocol=>"https", :port=>9200}, :scheme=>"https"}]}
[WARN] 2021-06-04 18:31:35.004 [[main]>worker0] elasticsearch - Failed to query elasticsearch for previous event {:index=>"replacement-test", :error=>"Illegal character in authority at index 8: https://{:host=>\"es_host:9200\", :scheme=>\"https\", :protocol=>\"https\", :port=>9200}:9200/replacement-test%2A/_search?q=txid%my_value&size=1&sort=%40timestamp%3Adesc"}

```

2  
`[WARN] 2021-06-04 17:45:48.669 [[main]>worker1] elasticsearch - Could not index event to Elasticsearch. {:status=>404, :action=>["update", {:_id=>"%{[doc_id]}", :_index=>"replacement-test", :routing=>nil, :_type=>"_doc", :retry_on_conflict=>1}, #<LogStash::Event:0x1cbd73c2>], :response=>{"update"=>{"_index"=>"replacement-test", "_type"=>"_doc", "_id"=>"%{[doc_id]}", "status"=>404, "error"=>{"type"=>"document_missing_exception", "reason"=>"[_doc][%{[doc_id]}]: document missing", "index_uuid"=>"ggzdlW2nQmuMGERrg1iAcA", "shard"=>"0", "index"=>"replacement-test"}}}}`

showing my filter configurations and output configurations for reference :

```
if [baseValueUnitAmount] > 968
  {
   mutate 
   { 
    add_field => {"log" => "update"}
    add_field => {"status" => "this is a updated log"}
   }
  
   elasticsearch 
   {
      hosts => ["es_host:9200"]
      query => "txid:%{[txid]}"
      ca_file => "/etc/logstash/certs_blog/ca/ca.crt"
      ssl => true
      fields => { "_id" => "doc_id" }
      index => "replacement-test"
      user => 'my_user'
      password => 'my_password'

   }
  }
}
output
{
	stdout { codec => rubydebug }
	          if [log] == "update"
            {
            elasticsearch
            {
            codec => json
            hosts => ["es_host:9200"]
            action => "update"
            document_id => "%{[doc_id]}"
            index => "replacement-test"
            manage_template => false
            ssl => true
            ssl_certificate_verification => false
            user => 'my_user'
            password => 'my_password'
            }  
            }
            else
            {
            elasticsearch
            {
            codec => json
            hosts => ["es_host:9200"]
            index => "replacement-test"
            manage_template => false
            ssl => true
            ssl_certificate_verification => false
            user => 'my_user'
            password => 'my_password'
            }
            }
}

```

Moreover, When I saw event JSON in debug logs, I found my copied field `doc_id` is being sent nil. not sure why!

 ![Selection_176](https://us1.discourse-cdn.com/elastic/original/3X/e/f/efde698ec041ba811e4b18b18543b8230b871f39.png)

In addition, I tried to verify my query from dev tools and it gave the expected result

 ![Selection_177](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8bec600145dafb4ba4f746155abb0071a00c2e31.png)

---

<div class="post-metadata">

**Author:** ![ankitdevnalkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitdevnalkar/32/46158_2.png) [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\
**Post date:** [June 4, 2021, 6:43pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/17 "2021-06-04T18:43:35Z")

</div>

I managed to resolve above issues by commenting `ssl => true` and putting `https://` in hosts section.  
Also, `docinfo_fields` option helped to copy `_id` from old documents.

**Note** : `fields` option will not help to copy `_id` field value.

---

<div class="post-metadata">

**Author:** ![ankitdevnalkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitdevnalkar/32/46158_2.png) [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\
**Post date:** [June 4, 2021, 6:49pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/18 "2021-06-04T18:49:05Z")

</div>

appreciate your inputs @Badger !  
it was indeed valuable to solve this issue 🙂 🙌

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 2, 2021, 6:49pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796/19 "2021-07-02T18:49:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.