agentw
August 4, 2021, 9:32pm
1
I upgraded our cluster to 7.14, and after the update for Logstash I received "Pipeline error {:pipeline_id=>"main", :exception=>#<Elasticsearch::Transport::Transport::Errors::Unauthorized: [401]". I was able to isolate it to pipeline configs using the logstash-filter-elasticsearch plugin. The Elasticsearch output plugin works fine, as I see data being indexed. The cluster is using basic auth and TLS. Steps I took to try to fix the issue:
Verified credentials with the _authenticate API.
Verified the role in Kibana had index: read and cluster: manage set
Tried with the superuser account to rule out missing permissions
Updated the logstash-filter-elasticsearch plugin
I do have a second Logstash server running 7.13.4 with the same config, and it is not exhibiting issue. This seems to be an issue with the latest version the this specific plugin. Any help is appreciated.
1 Like
MakoWish
August 5, 2021, 6:23pm
2
+1
I just upgraded to Logstash 7.14.0 and am also seeing this. I have one Logstash server that performs lookups via the Logstash Elasticsearch Filter Plugin, and this plugin is now causing Logstash to crash since the upgrade from 7.13.3 to 7.14.0.
Aug 05 11:16:17 LS1 ServerName[10839]: [2021-08-05T11:16:17,173][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Elasticsearch::Transport::Transport::Errors::Unauthorized: [401] >, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/transport/base.rb:205:in `__raise_transport_error'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/transport/base.rb:333:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/transport/http/manticore.rb:71:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/client.rb:152:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-7.5.0/lib/elasticsearch/api/actions/ping.rb:23:in `ping'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.9.3/lib/logstash/filters/elasticsearch.rb:310:in `test_connection!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.9.3/lib/logstash/filters/elasticsearch.rb:117:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1820:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:586:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/redacted1.conf", "/etc/logstash/conf.d/redacted2.conf", "/etc/logstash/conf.d/redacted3.conf", "/etc/logstash/conf.d/redacted4.conf"], :thread=>"#<Thread:0x198c08c8 run>"}
Yep, same here.
Going to do a rollback, raise a ticket..
MakoWish
August 10, 2021, 5:11pm
4
@Kaj_Noppen ,
Did a rollback work for you? I rolled Logstash back to the previous version we were running (7.13.3), but the issue did not go away.
EDIT:
Just completely purged Logstash from the system, installed 7.13.3, and still having the issue.
Yep, Rollback fixed this instantly. Support is looking for the issue.
Kaj_Noppen
August 11, 2021, 12:11pm
6
Just an FYI, Elastic created some Github issues, so it's a confirmed bug and should be fixed sometime
opened 09:52AM - 10 Aug 21 UTC
bug
regression
status:needs-triage
LS 7.14.0 updated its `elasticsearch` gem dependency to (`-> 7`) **7.5.0** which… leads to a regression with the filter plugin when (user/password) credentials are used. This also affects any other authorization mechanism such as `cloud_auth` and `api_key` as the plugin passes all of those as a header to `Elasticsearch::Client`.
The plugin is setting the [Authorization header with `transport_options: { headers: ... }`](https://github.com/logstash-plugins/logstash-filter-elasticsearch/blob/v3.9.3/lib/logstash/filters/elasticsearch/client.rb#L21) but this isn't working properly:
```
[logstash.filters.elasticsearch][main] New ElasticSearch filter client {:hosts=>["http://localhost:9200"]}
[ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Elasticsearch::Transport::Transport::Errors::Unauthorized: [401] >
```
- [ ] elasticsearch filter reproducer: https://github.com/logstash-plugins/logstash-filter-elasticsearch/issues/147
- [ ] elasticsearch input is also affected
*NOTE: LS has been using elasticsearch gem version 5.0.5 for long time (through all 7.x series and even before).*
opened 09:43AM - 10 Aug 21 UTC
bug
status:needs-triage
**Logstash information**:
Please include the following information:
Logsta… sh version (e.g. `bin/logstash --version`) **7.14.0**
**Description of the problem including expected versus actual behavior**:
The problem relates to a elasticsearch client dependency update, for LS < 7.14.0 the version used was 5.0.5,
in 7.14.0 LS forced a requirement of `>7` and ended up with version 7.5.0 which does not work properly.
**Steps to reproduce**:
`bin/logstash -e "filter { elasticsearch { user => 'elastic' password => 'elastic' hosts => 'http://localhost:9200' } }"`
NOTE: `hosts` needed due broken default on the field: https://github.com/logstash-plugins/logstash-filter-elasticsearch/issues/125
**Provide logs (if relevant)**:
**7.13.0** works while **7.14.0** fails with 401 (failing to pass basic auth header):
```
[2021-08-10T11:34:59,488][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.14.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 25.302-b08 on 1.8.0_302-b08 +indy +jit [linux-x86_64]"}
[2021-08-10T11:34:59,537][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/home/kares/workspace/work/elastic/logstash-7.14.0/data/queue"}
[2021-08-10T11:34:59,557][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/kares/workspace/work/elastic/logstash-7.14.0/data/dead_letter_queue"}
[2021-08-10T11:35:00,147][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-08-10T11:35:00,201][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"f07279a0-4967-4ef8-b508-ba191b55fefb", :path=>"/home/kares/workspace/work/elastic/logstash-7.14.0/data/uuid"}
[2021-08-10T11:35:03,573][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9601}
[2021-08-10T11:35:04,668][INFO ][org.reflections.Reflections] Reflections took 167 ms to scan 1 urls, producing 120 keys and 417 values
[2021-08-10T11:35:09,507][INFO ][logstash.filters.elasticsearch][main] New ElasticSearch filter client {:hosts=>["http://localhost:9200"]}
[2021-08-10T11:35:10,466][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Elasticsearch::Transport::Transport::Errors::Unauthorized: [401] >, :backtrace=>["/home/kares/workspace/work/elastic/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/transport/base.rb:205:in `__raise_transport_error'", "/home/kares/workspace/work/elastic/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/transport/base.rb:333:in `perform_request'", "/home/kares/workspace/work/elastic/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/transport/http/manticore.rb:71:in `perform_request'", "/home/kares/workspace/work/elastic/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/client.rb:152:in `perform_request'", "/home/kares/workspace/work/elastic/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-7.5.0/lib/elasticsearch/api/actions/ping.rb:23:in `ping'", "/home/kares/workspace/work/elastic/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.9.3/lib/logstash/filters/elasticsearch.rb:310:in `test_connection!'", "/home/kares/workspace/work/elastic/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.9.3/lib/logstash/filters/elasticsearch.rb:117:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/home/kares/workspace/work/elastic/logstash-7.14.0/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1820:in `each'", "/home/kares/workspace/work/elastic/logstash-7.14.0/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/home/kares/workspace/work/elastic/logstash-7.14.0/logstash-core/lib/logstash/java_pipeline.rb:586:in `maybe_setup_out_plugins'", "/home/kares/workspace/work/elastic/logstash-7.14.0/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'", "/home/kares/workspace/work/elastic/logstash-7.14.0/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/home/kares/workspace/work/elastic/logstash-7.14.0/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x6c7c724f run>"}
[2021-08-10T11:35:10,470][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
```
1 Like
same issue. just upgraded to 7.14.0 and
The git page says it is fixed. How do I fix this?
Badger
August 12, 2021, 4:07pm
10
The git page says it is fixed in 7.14.1, which has not released yet.
what should I do mean time? that ingestion is not super crtical. hence I was thinking to do some manual work this then if I can't fix it.
should I move to manual fix?
agentw
August 12, 2021, 4:33pm
12
@elasticforme /usr/share/logstash/bin/logstash-plugin update logstash-filter-elasticsearch fixed it for my v7.14.0 host.
2 Likes
system
September 9, 2021, 4:34pm
13
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
