Using keyword type for mapping

[ybegus]

I cxreate indices on daily base. I use the same mapping types for all of them. Mapping contains 23 fields. 6 of them are keyword. Also I prepared index pattern to process them together. I coilected data for several days and I found the following problem.
Keyword fields may be represented by 2 fields like
fieldA
fieldA.keyword
where fieldA.keyword is aggregatabke and fieldA is not or by one field fieldA where fieldA is aggregatable.

As a result I could not combine these data to the same dashboard

I found that including daily indices into Kibana. Why does it happen and how to avoid it?

I use version 6.2.3

Thank you in advance,

Yefim

[dadoonet (David Pilato)]

Probably because you sent a document which contains a field fieldA which was not part of your mapping so elasticsearch tried to guess automatically which ended up with the 2 fields.

You can reject such documents by using a strict mapping if you wish.

More information at https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html

[ybegus]

Thanks for quick reply. I appreciate it.

It sounds reasonable but this is not a case unfortunately. All fields and their types belong to pre-defined mapping json-object. According to nature of documents each document contains subset of these fields but each field of any document belongs to this mapping json-object. I use Node,js (elasticsearch module) to create mapping, indices and insert documents to Elasticsearch. I checked in Kibana that types of all fields correspond to types I set in mapping for each field but fields with a keyword type sometimes (every 1-3 days) may be changed and have extra field - field.keyword or vise versa field.keyword may disappear and original field became aggregatable. As a result index pattern in Kibana get yellow status and visualizations have lack of data.

[dadoonet (David Pilato)]

Can you share the index templates you have if any?
Can you share a "normal" mapping and a "bad" one?

[ybegus]

I've got "normal" ('use_name') and "bad" (user_name, username.keyword) mapping results using the same mapping json-object (based on mapping (user_name: { type: 'keyword' }) in Node.js. It happened when I created a new index and sent mapping for this index. I implemented index template and sent mapping just one time creating this template. After that new indices re-used this mapping. After these changes i see just "normal" mapping on Kibana side at least for a couple days since I implemented it. You may find my mapping below. A code below is a part of Node.js

. . .
const mappings = {
log: {
properties: {
ts: { type: 'date' },
opID: { type: 'keyword' },
trqID: { type: 'keyword' },
pmID: { type: 'keyword' },
vpID: { type: 'keyword' },
user_name: { type: 'keyword' },
tpath: { type: 'keyword' },
retCode: { type: 'integer' },
fd: { type: 'integer' },
length: { type: 'long' },
offset: { type: 'long' },
read_size: { type: 'long' },
write_size: { type: 'long' },
flags: { type: 'integer' },
mode: { type: 'integer' },
old_tpath: { type: 'text' },
new_tpath: { type: 'text' },
mount_path: { type: 'text' },
truncate: { type: 'long' },
devType: {type: 'integer' },
uid: { type: 'integer' },
gid: { type: 'integer' },
stpath: { type: 'text' },
dtpath: { type: 'text' },
atime: {
type: 'nested',
properties: {
sec: { type: 'long' },
nsec: { type: 'long' },
},
},
mtime: {
type: 'nested',
properties: {
sec: { type: 'long' },
nsec: { type: 'long' },
},
},
},
},
};

. . .

[dadoonet (David Pilato)]

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

You may find my mapping below

Actually, I'd like you to run:

GET indek_ok/_mapping
GET indek_ko/_mapping

Also:

GET _templates

[ybegus]

I updated my code implementing index-template and I don't see this behavior anymore. Current mapping doesn't contain field types like field.keyword. Nevertheless I still could not explain this behavior. I updated my post.