Type Mismatch between Ingest Pipeline and Kibana

Elastic Stack Elasticsearch
1

Hello Team,

We are facing an issue with Elastic Stack and Kabana type mismatch. Kabana is not picking up type as NUMBER rather shows them as string. Here is what we've tried,

  • Initially used ingest pipeline node grok processor to parse data from CSV file as type "DATA", the index type on Kabana is being shown as Type String.
  • Changed the values to Type NUMBER (for values which are like 1.0, 0.003 ), the data is not being parsed at all.
  • Now changed the value to BASE64 , ingest processor node, parses the data, however Kabana still shows them as type string. Tried with new ingest pipeline, new name for the tags, still the values in Kabana are shown as string.

Any help / suggestion please on why sometimes ingest pipeline grok processor are not picking up NUMBER and then Kabana not showing the type as NUMBER..

2

Run

GET yourindexname/_mapping

And check what is the mapping for that field. If text, then that explains and you need to adapt your mapping.

3

Thanks Dave..

Looking at the mapping , here is what I have,

"L75Pc": {
"type": "keyword",
"ignore_above": 1024
},
and so on for all the fields..

Grok processor has it defined as BASE16FLOAT. Is there way we can change the field type..?

4

You need to provide your own mapping and reindex.

5

Thanks Dave..

At which level, the reindex needs to be done. Is it withing ingest grok processor ?
Is Kibana picking up the default data type (which is string), we tried deleting the index and re-creating, however still we get the data type as string.

Is there any pointers or example you can share. Any pointers to specific document will also be helpful.

6

Basically do:

DELETE myindex
PUT myindex
{
  "mappings": {
    "doc": {
      "properties": {
        "foo": {
          "type": "integer"
        }
      }
    }
  }
}
PUT myindex/doc/1
{
  "foo": 1
}

That should work.
Once you understand what I wrote as an example, apply this to your documents and generate a correct mapping.
Then index your source (whatever it is) in this index myindex or use the reindex API to read from your bad index and index in myindex.

Note that if you have been using the correct type in JSON, that would be even better.
IE: instead of indexing:

{
  "foo": "1"
}

Index

{
  "foo": 1
}
7

Thanks Dave.

We have now removed the indexes completely and added the following ingest grok pattern, which works fine as long as data is available...
{
"description": " test ingest pipeline data",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"^%{Data:abc.timestamp}\,?%{DATA:Count:int}?, \?%{DATA:Average:float}?"
]
}
}
]

However, whenever there is no data, it throws an ingest error as empty string. Do you know if there is better way to specify ignore empty strings (other than the '?'). Also is there way we can turn on debug for ingest pipelines.

Thanks,

8

Please format your code.

Do you know if there is better way to specify ignore empty strings (other than the '?').

Read: Handling failures in pipelines | Elasticsearch Guide [master] | Elastic

Also is there way we can turn on debug for ingest pipelines.

This might help: Simulate pipeline API | Elasticsearch Guide [master] | Elastic

9

Thanks Dave,

The latest issue seems more of a specific problem,

As it stands we now have a Grok ingest pipeline,

{
"description": "Pipeline for parsing log",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["^%{DATA:abc.timestamp}\|%{DATA:abc.org}:%{DATA:abc.xycr}:%{INT:abc.counter}\|%{DATA:Try}\|%{DATA:ID}?$”
]
}
{
"convert": {
"field": "Try",
"type": "float",
"ignore_missing": true
}
},
{
"convert": {
"field": "ID",
"type": "integer",
"ignore_missing": true
}

The issue we are now facing is on the Kibana front where the Variable Try and ID are shown as Text rather than Numbers.

We did multiple testing after clearing up the indices, and noticed that whenever the values are present, then they are treated as Numbers in Kibana and if the values are not present, I think it is being treated as Null String and in Kibana it is represented as Text. Is there way these variable can always be treated as numbers.

In the GROK processor, we did try representing them as NUMBER rather than DATA, however it throws an exception, which I think NUMBER always expect some values in it.

We are running ELK Version 5.6.0. Do you know if there is way around it..?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.