Upgrade to es 6.1.0 trouble

Elastic Stack Elasticsearch
1

Hi,

I try to upgrade the ELK stack from 5.6.3 to 6.1.0.

I get some errors when trying to load the old templates.

15:33:23.975 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Installing elasticsearch template to _template/events
15:33:24.019 [[main]-pipeline-manager] ERROR logstash.outputs.elasticsearch - Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://elasticsearch:9200/_template/events'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:287:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:274:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:369:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:273:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:281:in `block in put'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:338:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:82:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:21:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:57:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:26:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:343:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:354:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:354:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:743:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:364:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:288:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"]}
15:33:24.020 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}



15:33:25.392 [Ruby-0-Thread-49: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.2.3/lib/logstash/inputs/syslog.rb:105] INFO  logstash.inputs.syslog - Starting syslog tcp listener {:address=>"0.0.0.0:5546"}
15:33:30.551 [Ruby-0-Thread-36@[main]>worker1: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:392] WARN  logstash.outputs.elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-2017.12.18", :_type=>"celerylog", :_routing=>nil}, #<LogStash::Event:0x738eb75b>], :response=>{"index"=>{"_index"=>"filebeat-2017.12.18", "_type"=>"celerylog", "_id"=>"aTZCamABrDapfknuLk-F", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [filebeat-2017.12.18] as the final mapping would have more than 1 type: [celerylog, nfvolog]"}}}}
15:33:30.551 [Ruby-0-Thread-36@[main]>worker1: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:392] WARN  logstash.outputs.elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-2017.12.18", :_type=>"vnfmlog", :_routing=>nil}, #<LogStash::Event:0x355475d7>], :response=>{"index"=>{"_index"=>"filebeat-2017.12.18", "_type"=>"vnfmlog", "_id"=>"ajZCamABrDapfknuLk-F", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [filebeat-2017.12.18] as the final mapping would have more than 1 type: [vnfmlog, nfvolog]"}}}}
15:33:30.557 [Ruby-0-Thread-35@[main]>worker0: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:392] WARN  logstash.outputs.elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-2017.12.18", :_type=>"celerylog", :_routing=>nil}, #<LogStash::Event:0x64d18f81>], :response=>{"index"=>{"_index"=>"filebeat-2017.12.18", "_type"=>"celerylog", "_id"=>"azZCamABrDapfknuLk-8", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [filebeat-2017.12.18] as the final mapping would have more than 1 type: [celerylog, nfvolog]"}}}}

Is there anything special regarding the templates I need to do before upgrading to 6.1.0?

Thanks
Mathias

2

Hello again,

I got rid of the filebeat template problem by loading latest filebeat template.

For my own template I still have problems.'
Looking into elasticsearch log I found:

[2017-12-19T06:58:51,927][INFO ][o.e.n.Node ] [fDBNwFz] started
[2017-12-19T06:58:52,235][INFO ][o.e.g.GatewayService ] [fDBNwFz] recovered [5] indices into cluster_state
[2017-12-19T06:58:53,019][INFO ][o.e.c.r.a.AllocationService] [fDBNwFz] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[events-2017.12.18][1]] ...]).
[2017-12-19T06:59:29,603][DEBUG][o.e.a.a.i.t.p.TransportPutIndexTemplateAction] [fDBNwFz] failed to put template [events]
java.lang.IllegalArgumentException: Rejecting mapping update to [2PqVXh_kT6e-4LJkrzVF8A] as the final mapping would have more than 1 type: [log, state, counter, event, generic, notice]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:494) ~[elasticsearch-6.1.0.jar:6.1.0]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:350) ~[elasticsearch-6.1.0.jar:6.1.0]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:274) ~[elasticsearch-6.1.0.jar:6.1.0]

Br Mathias

3

Oh no,

I just read:
https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

Seems like you have changed the entire template structure.
The example indicates that you insert new documents using a new type field.

What if I am not able to change the input.
Are there any other way to use old data?

Br Mathias

4

You need to send your data to different indices like index-type1-*, index-type2-*...

5

So instead of having a template like:
{

"template": "events-*",
"id": "events-*",
"type": "index-pattern",
"version": 1,
"mappings": {
    "log": {
      .
      .
   "state": {
     .
     .
    "counter": {
    .
    .

I should have something like

{
"template": "events-log-",
"id": "events-log-",
"type": "index-pattern",
"version": 1,
"mappings": {
"log": {

{
"template": "events-state-",
"id": "events-state-",
"type": "index-pattern",
"version": 1,
"mappings": {
"state": {

{
"template": "events-counter-",
"id": "events-counter-",
"type": "index-pattern",
"version": 1,
"mappings": {
"counter": {

Is that right?

With several indices, how do I select default index in Kibana?

Thanks
Mathias

6

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

Is that right?

Yes.

With several indices, how do I select default index in Kibana?

There is a little :star: icon close to the index name to make it the default one.

If having multiple indices is an issue, you can reproduce the way it was somewhat done internally in elasticsearch with multiple types (not really like this but you'll get the idea) and index your documents like:

{
  "log": {
     // Your log content here
  },
  "state": {
     // Your state content here
  }, 
  "counter": {
     // Your counter content here
  }
}

But it's less efficient IMO.

7

Hi,

I am still struggeling.
I have a new index for each type but I get the following error:

14:20:21.483 [Ruby-0-Thread-40@[main]>worker0: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:392] WARN logstash.outputs.elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"events-state-2017.12.19", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x7b701c76], :response=>{"index"=>{"_index"=>"events-state-2017.12.19", "_type"=>"doc", "_id"=>"8mclb2ABzTth-5wZkhFh", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [events-state-2017.12.19] as the final mapping would have more than 1 type: [doc, state]"}}}}

Then I found this:
https://discuss.elastic.co/t/rejecting-mapping-update-for-new-index/109576

Does it mean that I should not use templates at all in ES6?
(I though templates are required to get correct types)

I have the following template for state.

{
    "template": "events-state-*",
    "id": "events-state-*",
    "type": "index-pattern",
    "version": 1,
    "mappings": {
        "state": {
            "dynamic_templates": [

.
.
.

Can you explain how the template should be written?

Kind regards
Mathias

8

I understand now.

Just do not define any mapping for each index. Just a new index.

Thanks
Mathias

9

in es 6 one index only have one type...

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.