Events are not indexing after upgradtion 5.6 to 6.0

(Sumesh) #1

Hello

I have upgraded elastic 5.6 to 6.0 and after that events are not indexing from any sources.

The below is a strip from logstash log. I have tried with and without index mapping.
Where can I find the updated index mapping templates?

[2017-11-20T11:37:44,696][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-2017.11.20", :_type=>"wineventlog", :_routing=>nil}, #LogStash::Event:0x2f9e59e5], :response=>{"index"=>{"_index"=>"winlogbeat-2017.11.20", "_type"=>"wineventlog", "_id"=>"kAiT2F8B8FIdclYYevZJ", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [winlogbeat-2017.11.20] as the final mapping would have more than 1 type: [winlogbeat, wineventlog]"}}}}

Regards
Sumesh

(David Pilato) #2

It seems like your template created a type named winlogbeat and logstash is using wineventlog.

You need to either remove the template and replace it with what is needed or update your logstash configuration to use winlogbeat.

(Sumesh) #3

What I did was , followed this post Help please.. breaking changes?
and
Changed the logstash output to this:
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

I was not using any index templates for mapping, still I dont understand why it started working!!!

Regards

(David Pilato) #4

What are the existing index templates?
What is your logstash configuration?

(Sumesh) #5

I have only one index template that is for logstash-*

{
  "logstash": {
    "order": 0,
    "version": 60001,
    "index_patterns": [
       "logstash-*"
    ],
    "settings": {
      "index": {
        "refresh_interval": "5s"
      }

Logstash :

input {
  beats {
    port => 5044
  }
}


output {

{
 elasticsearch {
    hosts => "http://localhost:9200"
    manage_template => true
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}
(David Pilato) #6

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

I edited your post.

You index template is not complete I think. Could you share the full one please?

I think your logstash config is wrong. Could you share the real one please?

(Sumesh) #7

Yes David am posting it here

input {
  beats {
port => 5044
  }
}

filter{} portion

if [type]=="wineventlog"
        {
        mutate { add_field => { "indexType" => "WINDOWS-LOGS" }
                                     }
                }

output{}

if [indexType]== "WINDOWS-LOGS"
{
 elasticsearch {
    hosts => "http://localhost:9200"
    manage_template => true
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

  }
}

Thanks

(David Pilato) #8

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

I’m pretty sure this logstash config does not work.
Which means you’re not copy/pasting it.

What is the mapping for your index?

(Sumesh) #9

Hi David, my logstash configuration is huge that's why I have copied only the relevant portion. The remaining events processing and added to the indexes.

I have not specified any mapping as it is a new installation. So it might be taking the default mapping?

Regards
Sumesh

(David Pilato) #10

Can you share the current mapping for this index?

(Sumesh) #11

Sorry, how to get that?
Is it same as _/template/* ?

Regards
Sumesh

(David Pilato) #12 
GET winlogbeat-2017.11.20/_mapping
(Sumesh) #13

Hello, thanks. Here is the mapping.

https://pastebin.com/8kLy5JU3

Thanks

Sumesh

(David Pilato) #14

This is the mapping for 21st not 20th.

Do you still have the issue today?

(Sumesh) #15

Hello David

I have deleted that by mistake, anyway it is working fine today. Many thanks.

Regards
Sumesh

(Sumesh) #16

Hi David

If I haven't put any default index templates, how will elasticsearch map the data to an index?

Regards
Sumesh

(David Pilato) #17

It will guess. Which is sometime (often?) bad.
In practice, I feel better to define a mapping by my self so I have a finer control.

But if you are using Logstash or Beats, index templates are provided OOTB.

(Sumesh) #18

Thanks David
I understood.

I have multiple sensors such as beats, json file input, syslog, tcp, snmp etc.

Regards

(system) closed #19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.