Visualize mongo query performance in Kibana


(Swapnil Jaiswal) #1

Hi,
I have successfully configured filebeat clients to stream data to centralized ELK server. Now i want to to visualize the mongo query performance data in various dashboards. For this is followed the instructions as below :

On ELK server

mkdir mongo_kibana_template ; cd mongo_kibana_template
curl -L -O http://download.elastic.co/beats/dashboards/beats-dashboards-1.1.1.zip
unzip beats-dashboards-1.1.1.zip
cd beats-dashboards-1.1.1/
./load.sh 

I also appended the following snippet of config on the filebeat client running on Mongo server.

On Filebeat client

protocols:  
  mongodb:
    ports: [27017, 27019]
    send_request: true     # index the request payload
    send_response: true    # index the response payload
    max_docs: 10           # maximum number of documents to index per request/response
    max_doc_length: 1024   # maximum document size to index

Besides the above rest of the config like TLS certificate addition, log file path were added as usual.

Post the above steps I got new dashboard and visualizations on my existing ELK server
but Data was missing.

Please point out if I am missing a step ?
Please note : data is being streamed to the ELK server normally from the FileBeat client.
(verified by checking checking KIBANA for logs from specific hostname.)
But same is not being picked up by the dashboard.

Sample dashboard i got following the above steps was like following but Without Data :frowning:
http://goo.gl/jTPG63

Reference links for above steps :
Ref1: https://goo.gl/tcW954
Ref2: https://goo.gl/89gMNE

PFB : FileBeat client config
Filebeat.yml

PFA the screenshots

Data is being streamed normally to the ELK server since same is searchable based on Mongo Server hostname - as below




Index Mapping config warning -- post running load.sh





Dashboard data not being fetched;
Error : Could not locate that index-pattern-field (id: resource)

System info :

Filebeat Client 
filebeat-1.1.1-1.x86_64
OS: CentOS release 6.7 (Final)
Linux ip-172-31-57-210 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux


ELK server 
OS : Ubuntu 14.04.3 LTS (GNU/Linux 3.19.0-49-generic x86_64)
Kibana : 4.3.0

Elastic search details :
  "name" : "Zip-Zap",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.2.0",
    "build_hash" : "8ff36d139e16f8720f2947ef62c8167a888992fe",
    "build_timestamp" : "2016-01-27T13:32:39Z",
    "build_snapshot" : false,
    "lucene_version" : "5.4.1"
  },

Config File for filebeat client running mongodb


(ruflin) #2

Could it be that you are mixing filebeat and packetbeat? It seems like you have the configuration for both beats in on config file. The protocols part is for packetbeat, and not filebeat.


(system) #3