How to make a text field aggregate-able in Kibana

Elastic Stack Kibana
1

Hi,

I am using Kibana 6.8.1 version.

We are using EFK. There is a text field name Message, I want to make visualization by using this field. However the field is not showing in option. I have tried to do it by assigning the type as keyword, but getting error.

PUT app-000064
{
"mappings": {
"_doc": {
"message": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
}
}
}
}

Error:

{
"error": {
"root_cause": [
{
"type": "resource_already_exists_exception",
"reason": "index [app-000064/kZ-MR9qsTKal0qvEul9gIA] already exists",
"index_uuid": "kZ-MR9qsTKal0qvEul9gIA",
"index": "app-000064"
}
],
"type": "resource_already_exists_exception",
"reason": "index [app-000064/kZ-MR9qsTKal0qvEul9gIA] already exists",
"index_uuid": "kZ-MR9qsTKal0qvEul9gIA",
"index": "app-000064"
},
"status": 400
}

Message Log:

image
image742×31 2.89 KB

I want to use the status, Response Time, Request Object these fields to make visualization.

Please help. Thank you

2

Hi @Pratishruti. Can you share the mapping of your index?

If I understand correctly the Status, Response Time, and Request Object fields do not yet exist in Elasticsearch. The data exists only in the message field. If this is correct, you will need to define a Dissect pattern to create the fields you want from the message field. It sounds like you are using Filebeat, so the Dissect processor would be added to the Filebeat configuration.

But, maybe we can make this easier. I would also love to know the source of the logs. Is there an existing module that matches the source of your logs? If so, you don't need to define a custom Dissect pattern.

3

Hello @nickpeihl , Thank you for your response.

Mapping of index:

{
  "app-000075" : {
    "mappings" : {
      "_doc" : {
        "_meta" : {
          "version" : "2020.01.23.0"
        },
        "dynamic_templates" : [
          {
            "aushape_generic_nested_fields" : {
              "path_match" : "aushape.data.*.*.*",
              "mapping" : {
                "index" : true,
                "type" : "text"
              }
            }
          },
          {
            "aushape_generic_fields" : {
              "path_match" : "aushape.data.*.*",
              "mapping" : {
                "index" : true,
                "type" : "text"
              }
            }
          },
          {
            "aushape_generic_records" : {
              "path_match" : "aushape.data.*",
              "mapping" : {
                "type" : "object"
              }
            }
          },
          {
            "message_field" : {
              "match" : "message",
              "match_mapping_type" : "string",
              "mapping" : {
                "index" : true,
                "norms" : false,
                "type" : "text"
              }
            }
          },
          {
            "string_fields" : {
              "match" : "*",
              "match_mapping_type" : "string",
              "mapping" : {
                "fields" : {
                  "raw" : {
                    "ignore_above" : 256,
                    "index" : false,
                    "type" : "keyword"
                  }
                },
                "index" : true,
                "norms" : true,
                "type" : "text"
              }
            }
          }
        ],
        "date_detection" : false,
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "fields" : {
              "raw" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            },
            "format" : "yyyy-MM-dd HH:mm:ss,SSSZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime"
          },
          "aushape" : {
            "properties" : {
              "data" : {
                "properties" : {
                  "avc" : {
                    "type" : "nested"
                  },
                  "execve" : {
                    "type" : "text"
                  },
                  "netfilter_cfg" : {
                    "type" : "nested"
                  },
                  "obj_pid" : {
                    "type" : "nested"
                  },
                  "path" : {
                    "type" : "nested"
                  }
                }
              },
              "error" : {
                "type" : "text"
              },
              "node" : {
                "type" : "keyword"
              },
              "serial" : {
                "type" : "long"
              },
              "text" : {
                "type" : "text"
              },
              "trimmed" : {
                "type" : "keyword"
              }
            }
          },
          "docker" : {
            "properties" : {
              "command" : {
                "type" : "keyword"
              },
              "container_id" : {
                "type" : "keyword"
              },
              "container_id_short" : {
                "type" : "keyword"
              },
              "container_image" : {
                "type" : "keyword"
              },
              "container_name" : {
                "type" : "text",
                "fields" : {
                  "raw" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "operation" : {
                "type" : "keyword"
              },
              "pid" : {
                "type" : "keyword"
              },
              "reason" : {
                "type" : "keyword"
              },
              "result" : {
                "type" : "keyword"
              },
              "sauid" : {
                "type" : "keyword"
              },
              "user" : {
                "type" : "keyword"
              }
            }
          },
          "file" : {
            "type" : "text",
            "fields" : {
              "raw" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "geoip" : {
            "dynamic" : "true",
            "properties" : {
              "location" : {
                "type" : "geo_point"
              }
            }
          },
          "hostname" : {
            "type" : "keyword"
          },
          "ipaddr4" : {
            "type" : "ip",
            "fields" : {
              "raw" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "ipaddr6" : {
            "type" : "ip"
          },
          "kubernetes" : {
            "properties" : {
              "container_image" : {
                "type" : "text",
                "fields" : {
                  "raw" : {
                    "type" : "keyword",
                    "index" : false,
                    "ignore_above" : 256
                  }
                }
              },
              "container_image_id" : {
                "type" : "text",
                "fields" : {
                  "raw" : {
                    "type" : "keyword",
                    "index" : false,
                    "ignore_above" : 256
                  }
                }
              },
              "container_name" : {
                "type" : "text",
                "fields" : {
                  "raw" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "event" : {
                "properties" : {
                  "count" : {
                    "type" : "integer"
                  },
                  "firstTimestamp" : {
                    "type" : "date",
                    "format" : "yyyy-MM-dd HH:mm:ss,SSSZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime"
                  },
                  "involvedObject" : {
                    "properties" : {
                      "apiVersion" : {
                        "type" : "keyword"
                      },
                      "kind" : {
                        "type" : "keyword"
                      },
                      "name" : {
                        "type" : "keyword"
                      },
                      "namespace" : {
                        "type" : "keyword"
                      },
                      "resourceVersion" : {
                        "type" : "keyword"
                      },
                      "uid" : {
                        "type" : "keyword"
                      }
                    }
                  },
                  "metadata" : {
                    "properties" : {
                      "name" : {
                        "type" : "keyword"
                      },
                      "namespace" : {
                        "type" : "keyword"
                      },
                      "resourceVersion" : {
                        "type" : "keyword"
                      },
                      "selfLink" : {
                        "type" : "keyword"
                      },
                      "uid" : {
                        "type" : "keyword"
                      }
                    }
                  },
                  "reason" : {
                    "type" : "keyword"
                  },
                  "source_component" : {
                    "type" : "keyword"
                  },
                  "type" : {
                    "type" : "keyword"
                  },
                  "verb" : {
                    "type" : "keyword"
                  }
                }
              },
              "flat_labels" : {
                "type" : "keyword",
                "norms" : true
              },
              "host" : {
                "type" : "keyword",
                "norms" : true
              },
              "labels" : {
                "properties" : {
                  "app" : {
                    "properties" : {
                      "kubernetes" : {
                        "properties" : {
                          "io/component" : {
                            "type" : "text",
                            "fields" : {
                              "raw" : {
                                "type" : "keyword",
                                "index" : false,
                                "ignore_above" : 256
                              }
                            }
                          },
                          "io/instance" : {
                            "type" : "text",
                            "fields" : {
                              "raw" : {
                                "type" : "keyword",
                                "index" : false,
                                "ignore_above" : 256
                              }
                            }
                          },
                          "io/managed-by" : {
                            "type" : "text",
                            "fields" : {
                              "raw" : {
                                "type" : "keyword",
                                "index" : false,
                                "ignore_above" : 256
                              }
                            }
                          },
                          "io/name" : {
                            "type" : "text",
                            "fields" : {
                              "raw" : {
                                "type" : "keyword",
                                "index" : false,
                                "ignore_above" : 256
                              }
                            }
                          },
                          "io/part-of" : {
                            "type" : "text",
                            "fields" : {
                              "raw" : {
                                "type" : "keyword",
                                "index" : false,
                                "ignore_above" : 256
                              }
                            }
                          },
                          "io/version" : {
                            "type" : "text",
                            "fields" : {
                              "raw" : {
                                "type" : "keyword",
                                "index" : false,
                                "ignore_above" : 256
                              }
                            }
                          }
                        }
                      }
                    }
                  },
                  "app_kubernetes_io_component" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "app_kubernetes_io_instance" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "app_kubernetes_io_managed-by" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "app_kubernetes_io_name" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "app_kubernetes_io_part-of" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "app_kubernetes_io_version" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  }
                }
              },
              "master_url" : {
                "type" : "keyword"
              },
              "namespace_id" : {
                "type" : "keyword",
                "norms" : true
              },
              "namespace_labels" : {
                "properties" : {
                  "istio-injection" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "kiali_io/member-of" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "kiali_io_member-of" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "kubernetes_io/metadata_name" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "kubernetes_io_metadata_name" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "maistra_io/ignore-namespace" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "maistra_io/member-of" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "maistra_io_ignore-namespace" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "maistra_io_member-of" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "name" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  },
                  "usesDatagrid" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "index" : false,
                        "ignore_above" : 256
                      }
                    }
                  }
                }
              },
              "namespace_name" : {
                "type" : "keyword",
                "norms" : true
              },
              "pod_id" : {
                "type" : "keyword",
                "norms" : true
              },
              "pod_ip" : {
                "type" : "text",
                "fields" : {
                  "raw" : {
                    "type" : "keyword",
                    "index" : false,
                    "ignore_above" : 256
                  }
                }
              },
              "pod_name" : {
                "type" : "keyword",
                "norms" : true
              }
            }
          },
          "level" : {
            "type" : "keyword"
          },
          "log_type" : {
            "type" : "text",
            "fields" : {
              "raw" : {
                "type" : "keyword",
                "index" : false,
                "ignore_above" : 256
              }
            }
          },
          "message" : {
            "type" : "text",
            "norms" : false
          },
          "namespace_name" : {
            "type" : "keyword",
            "doc_values" : false
          },
          "namespace_uuid" : {
            "type" : "keyword"
          },
          "offset" : {
            "type" : "long"
          },
          "openshift" : {
            "properties" : {
              "cluster_id" : {
                "type" : "text",
                "fields" : {
                  "raw" : {
                    "type" : "keyword",
                    "index" : false,
                    "ignore_above" : 256
                  }
                }
              },
              "sequence" : {
                "type" : "long"
              }
            }
          },
          "ovirt" : {
            "properties" : {
              "class" : {
                "type" : "keyword"
              },
              "cluster_name" : {
                "type" : "keyword"
              },
              "correlationid" : {
                "type" : "keyword"
              },
              "engine_fqdn" : {
                "type" : "keyword"
              },
              "entity" : {
                "type" : "keyword"
              },
              "host_id" : {
                "type" : "keyword"
              },
              "module_lineno" : {
                "type" : "keyword"
              },
              "thread" : {
                "type" : "keyword"
              }
            }
          },
          "pid" : {
            "type" : "keyword"
          },
          "pipeline_metadata" : {
            "properties" : {
              "@version" : {
                "type" : "keyword"
              },
              "collector" : {
                "properties" : {
                  "hostname" : {
                    "type" : "keyword"
                  },
                  "inputname" : {
                    "type" : "keyword"
                  },
                  "ipaddr4" : {
                    "type" : "ip",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "ipaddr6" : {
                    "type" : "ip"
                  },
                  "name" : {
                    "type" : "keyword"
                  },
                  "original_raw_message" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "received_at" : {
                    "type" : "date",
                    "format" : "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime"
                  },
                  "version" : {
                    "type" : "keyword"
                  }
                }
              },
              "normalizer" : {
                "properties" : {
                  "hostname" : {
                    "type" : "keyword"
                  },
                  "inputname" : {
                    "type" : "keyword"
                  },
                  "ipaddr4" : {
                    "type" : "ip",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "ipaddr6" : {
                    "type" : "ip"
                  },
                  "name" : {
                    "type" : "keyword"
                  },
                  "original_raw_message" : {
                    "type" : "text",
                    "fields" : {
                      "raw" : {
                        "type" : "keyword",
                        "ignore_above" : 256
                      }
                    }
                  },
                  "received_at" : {
                    "type" : "date",
                    "format" : "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime"
                  },
                  "version" : {
                    "type" : "keyword"
                  }
                }
              },
              "trace" : {
                "type" : "text",
                "analyzer" : "whitespace"
              }
            }
          },
          "service" : {
            "type" : "keyword"
          },
          "systemd" : {
            "properties" : {
              "k" : {
                "properties" : {
                  "KERNEL_DEVICE" : {
                    "type" : "keyword"
                  },
                  "KERNEL_SUBSYSTEM" : {
                    "type" : "keyword"
                  },
                  "UDEV_DEVLINK" : {
                    "type" : "keyword"
                  },
                  "UDEV_DEVNODE" : {
                    "type" : "keyword"
                  },
                  "UDEV_SYSNAME" : {
                    "type" : "keyword"
                  }
                }
              },
              "t" : {
                "properties" : {
                  "AUDIT_LOGINUID" : {
                    "type" : "keyword"
                  },
                  "AUDIT_SESSION" : {
                    "type" : "keyword"
                  },
                  "BOOT_ID" : {
                    "type" : "keyword"
                  },
                  "CAP_EFFECTIVE" : {
                    "type" : "keyword"
                  },
                  "CMDLINE" : {
                    "type" : "keyword"
                  },
                  "COMM" : {
                    "type" : "keyword"
                  },
                  "EXE" : {
                    "type" : "keyword"
                  },
                  "GID" : {
                    "type" : "keyword"
                  },
                  "HOSTNAME" : {
                    "type" : "keyword"
                  },
                  "LINE_BREAK" : {
                    "type" : "keyword"
                  },
                  "MACHINE_ID" : {
                    "type" : "keyword"
                  },
                  "PID" : {
                    "type" : "keyword"
                  },
                  "SELINUX_CONTEXT" : {
                    "type" : "keyword"
                  },
                  "SOURCE_REALTIME_TIMESTAMP" : {
                    "type" : "keyword"
                  },
                  "STREAM_ID" : {
                    "type" : "keyword"
                  },
                  "SYSTEMD_CGROUP" : {
                    "type" : "keyword"
                  },
                  "SYSTEMD_INVOCATION_ID" : {
                    "type" : "keyword"
                  },
                  "SYSTEMD_OWNER_UID" : {
                    "type" : "keyword"
                  },
                  "SYSTEMD_SESSION" : {
                    "type" : "keyword"
                  },
                  "SYSTEMD_SLICE" : {
                    "type" : "keyword"
                  },
                  "SYSTEMD_UNIT" : {
                    "type" : "keyword"
                  },
                  "SYSTEMD_USER_UNIT" : {
                    "type" : "keyword"
                  },
                  "TRANSPORT" : {
                    "type" : "keyword"
                  },
                  "UID" : {
                    "type" : "keyword"
                  }
                }
              },
              "u" : {
                "properties" : {
                  "CODE_FILE" : {
                    "type" : "keyword"
                  },
                  "CODE_FUNCTION" : {
                    "type" : "keyword"
                  },
                  "CODE_LINE" : {
                    "type" : "keyword"
                  },
                  "ERRNO" : {
                    "type" : "keyword"
                  },
                  "MESSAGE_ID" : {
                    "type" : "keyword"
                  },
                  "RESULT" : {
                    "type" : "keyword"
                  },
                  "SYSLOG_FACILITY" : {
                    "type" : "keyword"
                  },
                  "SYSLOG_IDENTIFIER" : {
                    "type" : "keyword"
                  },
                  "SYSLOG_PID" : {
                    "type" : "keyword"
                  },
                  "UNIT" : {
                    "type" : "keyword"
                  }
                }
              }
            }
          },
          "tags" : {
            "type" : "text",
            "analyzer" : "whitespace"
          },
          "tlog" : {
            "properties" : {
              "id" : {
                "type" : "long"
              },
              "in_bin" : {
                "type" : "short"
              },
              "in_txt" : {
                "type" : "text"
              },
              "out_bin" : {
                "type" : "short"
              },
              "out_txt" : {
                "type" : "text"
              },
              "pos" : {
                "type" : "long"
              },
              "session" : {
                "type" : "long"
              },
              "term" : {
                "type" : "keyword"
              },
              "timing" : {
                "type" : "keyword"
              },
              "user" : {
                "type" : "keyword"
              },
              "ver" : {
                "type" : "long"
              }
            }
          },
          "viaq_index_name" : {
            "type" : "keyword"
          },
          "viaq_msg_id" : {
            "type" : "keyword"
          }
        }
      }
    }
  }
}

We are using OpenShift Container Platform based Kibana, so the log source is OCP. We are not using Filebeat.

Can you please describe more about define a Dissect pattern, and Is it like that for every field of message that we want to use while making visualization, we need to define a Dissect pattern or need to mention it manually. Because I have given an example of Status , Response Time , and Request Object, there are multiple fields present as message will be different as per the logs.

Is there any other way available to use message field as aggregateable.

Thank You

4

Thanks for the additional information. I understand now that you are using OCP which appear to use Fluent instead of Filebeat for ingesting logs. Fluent is not an Elastic product and I am unfamiliar with it. So perhaps the dissect processor is not an option for you.

It may be possible to construct Runtime fields to access the properties in the message field. Runtime fields are slower than indexed fields, but you should be able to aggregate on Runtime fields in Kibana.

5

Hello @nickpeihl , Thanks for the information. As we are using Elasticsearch version 5.6.5, Runtime fields option is not available for this version. And as said previously we are using OCP based Kibana. so it is the latest version as per our OCP version.

Can you please suggest any other alternative to use message field as aggregateable.

Thank You

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.