# Data allways stack in field.keyword than field

**URL:** <https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937>\
**Category:** Kibana\
**Tags:** docker\
**Created:** [August 11, 2022, 2:48pm UTC](https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937 "2022-08-11T14:48:18Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dreinale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dreinale/32/106570_2.png) [@Dreinale](https://discuss.elastic.co/u/Dreinale)\
**Post date:** [August 11, 2022, 2:48pm UTC](https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937/1 "2022-08-11T14:48:19Z")

</div>

Hi,

I try to install & configure elk 8.3.3 without security on the docker "host" and filebeat on another docker.

All works, i have the live logs, i imported my dashboards, but i have this:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/e/8e59935fd1db7c86ad49e521d82a9d6858cd6dbd.png)

and the problem is because all of my logs are in field.keyword and not field.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/e/de61b0da664037aa8f711c6a9cb1560572cd8e60.png)

I checked my dataView (filebeat-\*) and i got this warning:

> Mapping conflict  
> A field is defined as several types (string, integer, etc) across the indices that match this pattern. You may still be able to use these conflict fields in parts of Kibana, but they will be unavailable for functions that require Kibana to know their type. Correcting this issue will require reindexing your data.

I go to **[http://192.168.66.214:9200/\_all/\_mapping](http://192.168.66.214:9200/_all/_mapping)** for see if my index have other type than _keyword_

```auto
"log_data_direction":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"log_data_info1":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"log_data_info2":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"log_data_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"log_data_numerical":{"type":"float"},"log_data_value":{"type":"text","fields":{"keyword":{"type":"keyword"

```

And it's ok.

there is my .conf of my logstash:

```auto
input {
  beats {
    port => 5044
    id => "from_filebeat"
    ssl => false
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    manage_template => false
    action => "create"
    }
    stdout { codec => rubydebug }
}

filter {
  dissect {
    mapping => {
      "message" => "%{log_time} %{log_absolutetime} %{log_model} %{log_zone} %{log_data_direction} %{log_data_name} %{log_data_value}"
    }
  }

  date {
    match => ["log_time", "HH:mm:ss.SSS", "ISO8601"]
  }

  ruby {
    init => "@current_car_mode = ''
             @transfer_car_mode = ''
             @start_driving_time = 0
             @current_weather = 'rain'
             @current_speed_limit = 300.0
             @current_speed = 0.0
             @start_over_speed_limit_time = 0
             @current_distraction = '0'
             @start_distraction_time = 0
             @start_reaction_time = 0
             @start_transfer_time = 0
             @current_driving_level = 'senior'
             #@test_reaction_time = 0
             #@test_transfer_time = 0
             @first_time = 0"
    code => "if @first_time == 0
               @first_time = 1
               new_event = event.clone()
               new_event.set('log_data_name', 'distraction_time');
               new_event.set('log_data_value', '0')
               new_event.set('log_data_info1', @current_weather)
               new_event.set('log_data_info2', @current_car_mode)
               new_event.set('log_data_numerical', 0)
               new_event_block.call(new_event)

               new_event = event.clone()
               new_event.set('log_data_name', 'over_speed_limit_time');
               new_event.set('log_data_value', '0')
               new_event.set('log_data_info1', @current_weather)
               new_event.set('log_data_info2', @current_car_mode)
               new_event.set('log_data_numerical', 0)
               new_event_block.call(new_event)

               @start_driving_time = event.get('log_absolutetime').to_f 
               @start_over_speed_limit_time = event.get('log_absolutetime').to_f
               @start_distraction_time = event.get('log_absolutetime').to_f
               @start_reaction_time = event.get('log_absolutetime').to_f
               @start_transfer_time = event.get('log_absolutetime').to_f
             end

             event.set('log_data_numerical', 0.0)
             if event.get('log_data_name') == 'car_mode' and event.get('log_model') == 'public_plugin_vehicle_data'
               if @current_car_mode == ''
                 @start_driving_time = event.get('log_absolutetime').to_f
               else
                 if event.get('log_data_value') != @current_car_mode
                   new_event = event.clone
                   new_event.set('log_data_name', 'driving_time');
                   new_event.set('log_data_value', event.get('log_absolutetime').to_f - @start_driving_time)
                   new_event.set('log_data_info1', @current_weather)
                   new_event.set('log_data_info2', @current_car_mode)
                   new_event.set('log_data_numerical', event.get('log_absolutetime').to_f - @start_driving_time)
                   new_event_block.call(new_event)
                   @start_driving_time = event.get('log_absolutetime').to_f
                 end
               end
               
               if @transfer_car_mode != ''
                 new_event = event.clone
                 new_event.set('log_data_name', 'transfer_time');
                 new_event.set('log_data_value', event.get('log_absolutetime').to_f - @start_transfer_time)
                 new_event.set('log_data_info1', @current_driving_level)
                 new_event.set('log_data_info2', @current_car_mode)
                 new_event.set('log_data_numerical', event.get('log_absolutetime').to_f - @start_transfer_time)
                 new_event_block.call(new_event)
                 @start_reaction_time = 0
                 @start_transfer_time = 0
                 @transfer_car_mode = ''
               end

               if event.get('log_data_value') == 'mrm'
                 new_event = event.clone
                 new_event.set('log_data_name', 'mrm_detection');
                 new_event.set('log_data_value', 'true')
                 new_event.set('log_data_info1', @current_driving_level)
                 new_event.set('log_data_info2', @current_car_mode)
                 new_event_block.call(new_event)
               end

               @current_car_mode = event.get('log_data_value')

             elsif event.get('log_data_name') == 'authority_transfer_request' and event.get('log_model') == 'public_plugin_smart_cabin'
               if event.get('log_data_value') == 'request_start'
                 @start_reaction_time = event.get('log_absolutetime').to_f
                 @start_transfer_time = event.get('log_absolutetime').to_f
                 @transfer_car_mode = ''
               elsif event.get('log_data_value') == 'request_taken_into_account'
                 if @start_reaction_time != 0 and @transfer_car_mode == ''
                   new_event = event.clone
                   new_event.set('log_data_name', 'reaction_time');
                   new_event.set('log_data_value', event.get('log_absolutetime').to_f - @start_reaction_time)
                   new_event.set('log_data_info1', @current_driving_level)
                   new_event.set('log_data_info2', @current_car_mode)
                   new_event.set('log_data_numerical', event.get('log_absolutetime').to_f - @start_reaction_time)
                   new_event_block.call(new_event)
                   @start_transfer_time = event.get('log_absolutetime').to_f
                   @transfer_car_mode = @current_car_mode
                 end
               elsif event.get('log_data_value') == 'request_cancel'
                   @start_reaction_time = 0
                   @start_transfer_time = 0
                   @transfer_car_mode = ''
               elsif event.get('log_data_value') == 'request_end'
                   #new_event = event.clone
                   #new_event.set('log_data_name', 'transfer_time');
                   #new_event.set('log_data_value', event.get('log_absolutetime').to_f - @start_transfer_time)
                   #new_event.set('log_data_info1', @current_driving_level)
                   #new_event.set('log_data_info2', @transfer_car_mode)
                   #new_event.set('log_data_numerical', event.get('log_absolutetime').to_f - @start_transfer_time)
                   #new_event_block.call(new_event)
                   #@start_transfer_time = 0
                   #@start_reaction_time = 0  
                   #@transfer_car_mode = ''
               end

             elsif event.get('log_data_name') == 'user.user_driving_level' and event.get('log_zone') == 'user_driver'
               @current_driving_level = event.get('log_data_value')
             
             elsif event.get('log_data_name') == 'weather' and event.get('log_model') == 'public_plugin_ros_driving_environment'
               if @current_car_mode != ''
                 new_event = event.clone
                 new_event.set('log_data_name', 'driving_time');
                 new_event.set('log_data_value', event.get('log_absolutetime').to_f - @start_driving_time)
                 new_event.set('log_data_info1', @current_weather)
                 new_event.set('log_data_info2', @current_car_mode)
                 new_event.set('log_data_numerical', event.get('log_absolutetime').to_f - @start_driving_time)
                 new_event_block.call(new_event)
                 @start_driving_time = event.get('log_absolutetime').to_f
               end
               if @current_distraction == '1'
                 new_event = event.clone
                 new_event.set('log_data_name', 'distraction_time');
                 new_event.set('log_data_value', event.get('log_absolutetime').to_f - @start_distraction_time)
                 new_event.set('log_data_info1', @current_weather)
                 new_event.set('log_data_info2', @current_car_mode)
                 new_event.set('log_data_numerical', event.get('log_absolutetime').to_f - @start_distraction_time)
                 new_event_block.call(new_event)
                 @start_distraction_time = event.get('log_absolutetime').to_f
               end
               if (@current_speed > @current_speed_limit)
                 new_event = event.clone
                 new_event.set('log_data_name', 'over_speed_limit_time');
                 new_event.set('log_data_value', event.get('log_absolutetime').to_f - @start_over_speed_limit_time)
                 new_event.set('log_data_info1', @current_weather)
                 new_event.set('log_data_info2', @current_car_mode)
                 new_event.set('log_data_numerical', event.get('log_absolutetime').to_f - @start_over_speed_limit_time)
                 new_event_block.call(new_event)
                 @start_over_speed_limit_time = event.get('log_absolutetime').to_f
               end
               @current_weather = event.get('log_data_value')

             elsif event.get('log_data_name') == 'distraction.distracted'
               if event.get('log_data_value') == '1'
                 if @current_distraction == '0'
                   @start_distraction_time = event.get('log_absolutetime').to_f
                 end
                 event.set('log_data_numerical', '1');
                 @current_distraction = '1'

                 #new_event = event.clone
                 #new_event.set('log_data_name', 'reaction_time');
                 #new_event.set('log_data_value', @test_reaction_time)
                 #new_event.set('log_data_info1', @current_weather)
                 #new_event.set('log_data_info2', 'autonomous_driving')
                 #new_event.set('log_data_numerical', @test_reaction_time)
                 #@test_reaction_time = @test_reaction_time + 2
                 #new_event_block.call(new_event)

               else
                 if @current_distraction == '1'
                   new_event = event.clone
                   new_event.set('log_data_name', 'distraction_time');
                   new_event.set('log_data_value', event.get('log_absolutetime').to_f - @start_distraction_time)
                   new_event.set('log_data_info1', @current_weather)
                   new_event.set('log_data_info2', @current_car_mode)
                   new_event.set('log_data_numerical', event.get('log_absolutetime').to_f - @start_distraction_time)
                   new_event_block.call(new_event)
                 end
                 event.set('log_data_numerical', '0');
                 @current_distraction = '0'

                 #new_event = event.clone
                 #new_event.set('log_data_name', 'transfer_time');
                 #new_event.set('log_data_value', @test_transfer_time)
                 #new_event.set('log_data_info1', @current_weather)
                 #new_event.set('log_data_info2', 'autonomous_driving')
                 #new_event.set('log_data_numerical', @test_transfer_time)
                 #@test_transfer_time = @test_transfer_time + 2
                 #new_event_block.call(new_event)

               end
             
             elsif event.get('log_data_name') == 'speed_limit' and event.get('log_model') == 'public_plugin_vehicle_data'
               if (@current_speed > @current_speed_limit)
                 @current_speed_limit = event.get('log_data_value').to_f
                 if @current_speed_limit >= @current_speed
                   new_event = event.clone
                   new_event.set('log_data_name', 'over_speed_limit_time');
                   new_event.set('log_data_value', event.get('log_absolutetime').to_f - @start_over_speed_limit_time)
                   new_event.set('log_data_info1', @current_weather)
                   new_event.set('log_data_info2', @current_car_mode)
                   new_event.set('log_data_numerical', event.get('log_absolutetime').to_f - @start_over_speed_limit_time)
                   new_event_block.call(new_event)
                 end
               else
                 @current_speed_limit = event.get('log_data_value').to_f
                 if @current_speed_limit < @current_speed
                   @start_over_speed_limit_time = event.get('log_absolutetime').to_f
                 end               
               end
             
             elsif event.get('log_data_name') == 'speed' and event.get('log_model') == 'public_plugin_vehicle_data'
               if @current_speed > @current_speed_limit
                 @current_speed = event.get('log_data_value').to_f
                 if @current_speed_limit >= @current_speed
                   new_event = event.clone
                   new_event.set('log_data_name', 'over_speed_limit_time');
                   new_event.set('log_data_value', event.get('log_absolutetime').to_f - @start_over_speed_limit_time)
                   new_event.set('log_data_info1', @current_weather)
                   new_event.set('log_data_info2', @current_car_mode)
                   new_event.set('log_data_numerical', event.get('log_absolutetime').to_f - @start_over_speed_limit_time)
                   new_event_block.call(new_event)
                 end
               else
                 @current_speed = event.get('log_data_value').to_f
                 if @current_speed_limit < @current_speed
                   @start_over_speed_limit_time = event.get('log_absolutetime').to_f
                 end               
               end

             elsif event.get('log_data_name') == 'lane_crossing_detection' and event.get('log_data_value') != 'lane_crossing' and event.get('log_model') == 'public_plugin_vehicle_data'
               new_event = event.clone
               new_event.set('log_data_name', 'lane_crossing');
               new_event.set('log_data_value', 'true')
               new_event.set('log_data_info1', @current_weather)
               new_event.set('log_data_info2', @current_car_mode)
               new_event_block.call(new_event)

             end"
  }

  mutate {
    convert => ["log_data_numerical","float"]
  }
}

```

I do all step for install & configure filebeat & elk.

I followed this topic **[https://discuss.elastic.co/t/index-problem-keyword/308112](https://.keyword)** but nothing.

i do the `filebeat setup -e` with elasticsearch port

and after i set the logstash port and i launched filebeat.

After i retry but with the logstash port and this command:

> filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["192.168.66.214:9200"]'

and it's allways the same problem.

If someone have the answer that can help me a lot !

---

<div class="post-metadata">

**Author:** ![JLeysens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jleysens/32/67404_2.png) [@JLeysens](https://discuss.elastic.co/u/JLeysens)\
**Post date:** [August 12, 2022, 9:25am UTC](https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937/2 "2022-08-12T09:25:52Z")

</div>

Hello @Dreinale !

Are you able to follow the setup instructions and getting filebeat -\> ES -\> Kibana working?

Per this comment: [Index problem .keyword - #5 by stephenb](https://discuss.elastic.co/t/index-problem-keyword/308112/5)

---

<div class="post-metadata">

**Author:** ![Dreinale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dreinale/32/106570_2.png) [@Dreinale](https://discuss.elastic.co/u/Dreinale)\
**Post date:** [August 12, 2022, 9:37am UTC](https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937/3 "2022-08-12T09:37:54Z")

</div>

Hi @JLeysens ,

yes i that work but i need logstash for parsing my data.

---

<div class="post-metadata">

**Author:** ![JLeysens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jleysens/32/67404_2.png) [@JLeysens](https://discuss.elastic.co/u/JLeysens)\
**Post date:** [August 12, 2022, 9:45am UTC](https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937/4 "2022-08-12T09:45:08Z")

</div>

Are you using a module that requires a pipeline to be applied?

> [@Index problem .keyword](https://discuss.elastic.co/t/index-problem-keyword/308112/7):
>
> Yup but if you ever use a module with a pipeline that will not work / support it. If you are not using a module or pipeline you are fine.. The if / else supports both. You should add the manage\_template =\> false Ohh @AnotherGuy Welcome to the community!

In the first comment I shared the logstash configuration takes this into consideration

---

<div class="post-metadata">

**Author:** ![Dreinale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dreinale/32/106570_2.png) [@Dreinale](https://discuss.elastic.co/u/Dreinale)\
**Post date:** [August 12, 2022, 9:50am UTC](https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937/5 "2022-08-12T09:50:57Z")

</div>

i remember i enabled 2 modules (system & nginx)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/a/ca39ac7b6feac0543eeec5905f2399d1a4dbbfd9.png)

But i just disabled them and i's allways the same

---

<div class="post-metadata">

**Author:** ![JLeysens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jleysens/32/67404_2.png) [@JLeysens](https://discuss.elastic.co/u/JLeysens)\
**Post date:** [August 12, 2022, 1:11pm UTC](https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937/6 "2022-08-12T13:11:14Z")

</div>

Hmm, if you see the logstash `output` config here it is applying `pipeline`s to the incoming data.

> **[Use ingest pipelines for parsing | Logstash Reference \[7.17\] | Elastic](https://www.elastic.co/guide/en/logstash/7.17/use-ingest-pipelines.html)**

Try updating your logstash output to match? Specifically something like:

```auto
output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "<your-host>"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "<your-host>"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      user => "elastic"
      password => "secret"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 12, 2022, 1:59pm UTC](https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937/7 "2022-08-12T13:59:19Z")

</div>

@JLeysens @Dreinale

Actually if you are using modules + logstash + datastreams + 8.x the output is a little different our docs are wrong and there is a tiny bug

Please look at this thread

> [@Only write ops with an op\_type of create are allowed in data streams"}}}}](https://discuss.elastic.co/t/only-write-ops-with-an-op-type-of-create-are-allowed-in-data-streams/308182/2):
>
> Try this there is an error in the docs... Clean up the data stream and try again... output { if [@metadata][pipeline] { elasticsearch { hosts =\> "http://localhost:9200" pipeline =\> "%{[@metadata][pipeline]}" user =\> "elastic" password =\> "password" manage\_template =\> false index =\> "%{[@metadata][beat]}-%{[@metadata][version]}" action =\> "create" } } else { elasticsearch { hosts =\> "http://localhost:9200" user =\> "elastic" …

```auto
output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      pipeline => "%{[@metadata][pipeline]}"
      user => "elastic"
      password => "password"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      action => "create" <!--- Important
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      user => "elastic"
      password => "password"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      action => "create"
    }
  }
} 

```

I think I explain why somewhere in there... 🙂

---

<div class="post-metadata">

**Author:** ![Dreinale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dreinale/32/106570_2.png) [@Dreinale](https://discuss.elastic.co/u/Dreinale)\
**Post date:** [August 12, 2022, 2:23pm UTC](https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937/8 "2022-08-12T14:23:13Z")

</div>

Ohh my god you found the solution !

the problem was there

```auto
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

```

we need to write it like this

```auto
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"

```

Thank you so much @JLeysens & @stephenb !

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 12, 2022, 4:04pm UTC](https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937/9 "2022-08-12T16:04:59Z")

</div>

Yes, you need to understand that you need to write to the datastream and in 7.x it was the write alias and that's how it works... Pretty common mistake.

Glad you got it working!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 9, 2022, 4:05pm UTC](https://discuss.elastic.co/t/data-allways-stack-in-field-keyword-than-field/311937/10 "2022-09-09T16:05:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.