Kibana filter ,fluentd


(Pmr) #1

Issue in displaying numeric data in kibana .

Client side Fluentd configuration

type tail tag underutilized.log path /var/log/utilization/monthly_summary.log format /^,HOST,(?[^,]+),DATE,(?[^,]+),STREAK,(?[^,]+),CPU_DAY_UTIL,(?[^,]+)$/ pos_file /var/spool/fluentd/underutilized.log.position.txt

<match underutilized.**>
type copy

type file path /var/log/fluentd/underutilized.log compress gzip type forward buffer_path /var/spool/fluentd/underutilized.*.buffer buffer_type file flush_interval 30s send_timeout 30s

heartbeat_type tcp
heartbeat_interval 10m
<server>
  name dstsrvr01
  host dstsrvr01.abc.com
  port 24224
</server>

[user1@srvr05 ~]$ tail -3 /var/log/utilization/monthly_summary.log
,HOST,srvr05,DATE,2018-10-10,STREAK,123,CPU_DAY_UTIL,0%
,HOST,srvr05,DATE,2018-10-11,STREAK,124,CPU_DAY_UTIL,0%
,HOST,srvr05,DATE,2018-10-12,STREAK,125,CPU_DAY_UTIL,0%

Destination Server dstsrvr01 Kibana configuration

in /etc/td-agent/td-agent.conf

<match underutilized.**>
type elasticsearch
logstash_format true
flush_interval 5s
logstash_prefix underutilized

We are able to see the generated indexes if we run curl in dstsrvr01
curl 'localhost:9200/_cat/indices?v' | grep underutilized
yellow open underutilized-2018.09.02 S9PURAzIT2-SSo6QuIPH_Q 5 1 1554 0 390.9kb 390.9kb
yellow open underutilized-2018.08.29 itVOHT7_Qg6B-RNSAY40Bw 5 1 1563 0 404.7kb 404.7kb
yellow open underutilized-2018.06.10 LD0rp2rvQ8yW4j4kZjjR7g 5 1 62 0 138.5kb 138.5kb
yellow open underutilized-2018.07.09 7GljpUOQQByol0gv-C5ZSg 5 1 2043 0 463.8kb 463.8kb

We are displaying these data through Kibana .We are particular on the STREAK value ,which we want to be in integer format . But here it is in STRING format bydefault
and not able to apply numerical filter (eg STREAK > 40).Is ther any option to convert a particular field to integer .

Client version: fluentd-0.10.31-1.el5
Server version: td-agent-2.3.6-0.el7.x86_64
kibana-5.6.9-1.x86_64


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.