Index problem .keyword

[AnotherGuy]

Hi,

I go a big problem today and i don't know how to resolve it.

I work on ELK 7.17.1 on the same computer and with filebeat on another computer which sends me log in live with a docker.

I import my template Dashboard so it's good but i found this error:

So i checked the index filebeat-* with the Data Visualizer and i have this :

All of my type: text have a .keyword with the entire log in it, but i want to be in the log_data_name type: text not in the .keyword.
I found a "solution" it's to set the

"fielddata": true

source:

1. how to set fielddata true on created index · Issue #584 · elastic/elasticsearch-ruby · GitHub
2. Text type family | Elasticsearch Guide [master] | Elastic

but i don't know how to do it without the curl PUT ...

So my question is, how my logs can write directly on the type: text?

[AnotherGuy]

Update!

I got this error:

Error executing runtime field or scripted field on index pattern filebeat-*

if (doc['log_data_name'].value == 'car_mode')
{
  
        ^---- HERE

Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [log_data_name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

[stephenb (Stephen Brown)]

Most likely You did not run filebeat setup before you started filebeat per the quick start instructions step 4 therefore the correct index template and mappings did not get loaded. Which means you are getting the "default mapping" which is the behavior you are seeing.

You will need to

• Stop filebeat
• cleanup / delete the index in Kibana
• run filebeat setup -e
• start filebeat to reload the data.

Note filebeat will not reload a file it has already loaded unless you clean out the filebeat data directory

[AnotherGuy]

Ok i do that but for the sudo filebeat setup -e i modif my filebeat.yml
i comment logstash.output and uncomment elasticsearch.output, but setup the filbeat i can uncomment my logstash.output ?

[stephenb (Stephen Brown)]

Ok

You did not say you were using logstash in the middle so there is more to do...

@AnotherGuy

but setup the filbeat i can uncomment my logstash.output ?

Yes comment out the logstash output during setup and set elasticsearch output.
Then comment out the elasticsearch output and uncomment the logstash output when running through logstash.

I usually highly recommend to get everything working using the direct architecture.

Filebeat -> Elasticsearch

Before trying and moving on to the more complex architecture.

Filebeat -> logstash -> Elasticsearch

Also since you are using logstash your pipeline needs to be "filebeat" aware please see this doc

You pipeline should look something like

input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://yourhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://yourhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      user => "elastic"
      password => "secret"
    }
  }
}

[AnotherGuy]

I have that

input {
  beats {
    port => 5044
    id => "from_filebeat"
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.66.11:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

[stephenb (Stephen Brown)]

Yup but if you ever use a module with a pipeline that will not work / support it.
If you are not using a module or pipeline you are fine.. The if / else supports both.

You should add the

manage_template => false

Ohh @AnotherGuy Welcome to the community!

[AnotherGuy]

I got a new problem with my stream log because i removed
-/var/lib/filebeat/registry/filebeat/*
-/var/log/filebeat/*

I try to resolve it and i'll be back.
Thank for the help

[AnotherGuy]

Okay i resolv the problem, i removed filebeat & follow the install Filebeat quick start: installation and configuration | Filebeat Reference [7.17] | Elastic

after the setup it's work's !