Cannot visualise data!

Elastic Stack Kibana
1

Hello i am new into kibana i was looking at this forum for solution but couldn't find solution.
I found a similar question and topic but it didn't helped me:
Similar problem

So i have EFK stack with fluentd or fluentbit (same situtuation) and i have some logger which logs messages to output by this code:

def on_message(client, userdata, msg):
todump = {"topic": msg.topic, "message": str(msg.payload, 'utf-8')}
print(json.dumps(todump))

So fluentd is capturing data and sending it to elasticsearch and than i access this in kabana and i cannot visualise my data.

This is table view and json view of this data:

chrome_2018-08-20_10-54-26 chrome_2018-08-20_10-54-09

For example i want to be able to visualise data log.Value from it and i can't i am trying many solutions without getting into anything , pls help.

2

Firstly, edited the Kibana URl- its not safe to have it here.
I think you need to process all the fields and include mappings ingest-node or logstash for processing, mappings either via template or on index creation.

Index templates allow you to define templates that will automatically be applied when new indices are created. The templates include both settings and mappings and a simple pattern template that controls whether the template should be applied to the new index.
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

Are you trying to visualize on log fields ? If it's the log field, you may want the json filter for example w/ ingest node. There are many use-cases where it is important to enrich incoming data. Node ingest implements a new type of ES node, which performs this enrichment prior to indexing.

Thanks
Rashmi

3

Thank you for your atention @rashmi.
Kibana url was just some droplets with my use case i posted it to give deeper insight.
I am quite new in this sort of knowledge and yes i am trying to visualise on log fields.
I do not understand what you wrote here.
Could you please write it somehow understandable for me ?
I want to be able to visualise on log fields let's say value. But i am not able to do it out of the box.
What should i apply to make it possible?

4

I shall tag my colleague @ppisljar for more info on visualizations here. He can shed more light on this

5

log is just one field with json in it ... so kibana can't access log.Value. You would need to index your data differently, so each of the fields inside log json would be a separate field indexed by elastic search.

I suggest to ask in elasticsearch forum or check with fluentbeat if anything can be configured there.

6

Am i not in elasitc forums ? Can some one provide any help on this topic ?

7

As mentioned above, you will need to extract the data from the log string before indexing it into Elasticsearch if you want to build visualisations on it. If you were using Filebeat and/or Logstash I could show you how to do it, but I have unfortunately no experience at all using FluentD. If you can specify an ingest pipeline from FluentD, you could probably also extract the data using an ingest pipeline.

8

Okey so i have some point here this inget pipeline. Is someone here familiar with it ?

9

Yes, we are familiar with it. Before diving into this I would however recommend verifying that FluentD Elasticsearch output is able to specify an ingest pipeline. As it is a reasonably recent addition, it may not be supported. Based on what I can see here, it seems like it may not be supported, which means that you would need to handle this in your FluentD configuration.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.