Creating a seperate index for conflicting logs

Elastic Stack Beats
1

log a has field "ts" of type float
log b has field "ts" of type date
This is causing a WARN - cannot create index

Ideally, I don't want to have to drop one of the logs, so I wanted to create a index specifically for log b which is only coming from a certain pod. This are the indeces I have made

- index: "index-a-%{+yyyyMM}" 
  when.not.regexp:
      kubernetes.pod.name: "mypod-.*"
  when.regexp:
     kubernetes.namespace: "mynamespace-*"

- index: "index-b-%{+yyyyMM}" 
   when.regexp:
      kubernetes.pod.name: "mypod-.*"
      kubernetes.namespace: "mynamespace-.*"

With the above config, I am still getting the WARN, and index-b is not created. Any help would be greatly appreciated

2

From Beats to Elasticsearch

3

Removed filebeat

4

From Elasticsearch to Beats

5

Added filebeat

6

if you are ok with using same index just moving the conflicting field.
you can use the rename processor.
you can tell it to move the field_x to field_a if condition A was met

7

Hi @calbertan Welcome to the Elastic community.

There might be multiple cases -

  1. Before re ingesting you need to delete old index so that it will create a new with specific field type. Make sure a and b log going in dedicated index.
  2. After getting error just check the mapping of an index. Usually this happen if you trying to ingest multi type of data on single field.

Some solutions -

  1. Maintain 2 ts field with different type.
  2. You can even try object field to maintain both data.