Index problem .keyword

Elastic Stack Kibana
1

Hi,

I go a big problem today and i don't know how to resolve it.

I work on ELK 7.17.1 on the same computer and with filebeat on another computer which sends me log in live with a docker.

I import my template Dashboard so it's good but i found this error:
image

So i checked the index filebeat-* with the Data Visualizer and i have this :

image
image1463×86 8.16 KB

All of my type: text have a .keyword with the entire log in it, but i want to be in the log_data_name type: text not in the .keyword.
I found a "solution" it's to set the

"fielddata": true

source:

  1. how to set fielddata true on created index · Issue #584 · elastic/elasticsearch-ruby · GitHub
  2. Text type family | Elasticsearch Guide [master] | Elastic

but i don't know how to do it without the curl PUT ...

So my question is, how my logs can write directly on the type: text?

2

Update!

I got this error:

Error executing runtime field or scripted field on index pattern filebeat-*

if (doc['log_data_name'].value == 'car_mode')
{
  
        ^---- HERE

Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [log_data_name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

3

Most likely You did not run filebeat setup before you started filebeat per the quick start instructions step 4 therefore the correct index template and mappings did not get loaded. Which means you are getting the "default mapping" which is the behavior you are seeing.

You will need to

  • Stop filebeat
  • cleanup / delete the index in Kibana
  • run filebeat setup -e
  • start filebeat to reload the data.

Note filebeat will not reload a file it has already loaded unless you clean out the filebeat data directory

1 Like
4

Ok i do that but for the sudo filebeat setup -e i modif my filebeat.yml
i comment logstash.output and uncomment elasticsearch.output, but setup the filbeat i can uncomment my logstash.output ?

5

Ok :slight_smile:

You did not say you were using logstash in the middle so there is more to do...

Yes comment out the logstash output during setup and set elasticsearch output.
Then comment out the elasticsearch output and uncomment the logstash output when running through logstash.

I usually highly recommend to get everything working using the direct architecture.

Filebeat -> Elasticsearch

Before trying and moving on to the more complex architecture.

Filebeat -> logstash -> Elasticsearch

Also since you are using logstash your pipeline needs to be "filebeat" aware please see this doc

You pipeline should look something like

input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://yourhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://yourhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      user => "elastic"
      password => "secret"
    }
  }
}
6

I have that

input {
  beats {
    port => 5044
    id => "from_filebeat"
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.66.11:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}
7

Yup but if you ever use a module with a pipeline that will not work / support it.
If you are not using a module or pipeline you are fine.. The if / else supports both.

You should add the

manage_template => false

Ohh @AnotherGuy Welcome to the community!

1 Like
8

I got a new problem with my stream log because i removed
-/var/lib/filebeat/registry/filebeat/*
-/var/log/filebeat/*

I try to resolve it and i'll be back.
Thank for the help

10

Okay i resolv the problem, i removed filebeat & follow the install Filebeat quick start: installation and configuration | Filebeat Reference [7.17] | Elastic

after the setup it's work's !

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.