I'm trying to send a log file. FB-->ES-->Kibana.
All the configurations are done.When i try for filebeat-* index pattern in kibana,it can't fetch the mapping.The screen is like this
Can you suggest a solution to this?
Can you suggest a solution to this?
Do you have any indices in Elasticsearch that matches the filebeat-* pattern?
No.. I haven't created any. How do i check for the same?
You need to index some data to get the indices created, which should resolve the issue.
how do i do that? can you please help me with this?
currently when i do GET /_cat/indices?v this is what i get
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .monitoring-kibana-2-2016.11.29 RHJ508yuTM63FMPc3NGUeQ 1 1 3719 0 817.3kb 817.3kb
yellow open .monitoring-kibana-2-2016.11.28 RA_2KqZZQEyMhZQ5Rv2fsQ 1 1 3684 0 761.7kb 761.7kb
yellow open .kibana 4blvngEtS2GQ9CqxHBPM3w 1 1 12 2 109kb 109kb
yellow open .monitoring-kibana-2-2016.11.30 sKHiSdtyQqWkvriNUorStw 1 1 2795 0 569.5kb 569.5kb
yellow open .monitoring-kibana-2-2016.11.21 8WmsFxuJT6iyBJBpPiX09g 1 1 1734 0 465.7kb 465.7kb
yellow open my-index gztHHJqSQKenNsotULctXg 5 1 1 0 3.9kb 3.9kb
yellow open .monitoring-es-2-2016.11.22 4id5-NxwTO6zsBKUD8tUSw 1 1 23272 64 9.7mb 9.7mb
yellow open .reporting-2016.11.20 5S-sBhbcRfmgwt5XwAYE0w 5 1 1 0 213kb 213kb
yellow open .monitoring-es-2-2016.11.24 u7Kn6HpwQ2uE7ENxoSH2Qg 1 1 31411 180 13.5mb 13.5mb
yellow open .monitoring-kibana-2-2016.11.24 4DPc2bRyR-aSWXEvMALonQ 1 1 3774 0 843.9kb 843.9kb
yellow open .monitoring-es-2-2016.11.21 UAT3Z9a9RfmO8Qu7M-7Q1g 1 1 9286 126 4.1mb 4.1mb
yellow open .monitoring-es-2-2016.11.28 jeBrGjLKRDSnnRvEIpg31w 1 1 36337 352 16.3mb 16.3mb
yellow open .monitoring-es-2-2016.11.29 OeWv5HVsShK2zWO-oA1NGw 1 1 41524 96 17.4mb 17.4mb
yellow open .monitoring-kibana-2-2016.11.25 kx_te3sOTw6K7DyPrUbhLQ 1 1 1150 0 274.7kb 274.7kb
yellow open .monitoring-data-2 FFiCFcWGQueHjioLi9puKQ 1 1 3 0 11.4kb 11.4kb
yellow open .monitoring-kibana-2-2016.11.22 qlA38y0RT_uKtL-6zxYNjQ 1 1 2916 0 634.6kb 634.6kb
yellow open .monitoring-es-2-2016.11.25 35M-gyX9SDODRJ5O_2ApwA 1 1 19746 0 8.5mb 8.5mb
yellow open .monitoring-es-2-2016.11.30 fQKHSr7iQuqbx9G5-8ZgkQ 1 1 33346 0 14.9mb 14.9mb
Try to get Filebeat to start indexing into Elasticsearch, assuming it is configured to use that index pattern.
how do i get filebeat to start indexing into elasticsearch?
i can share the yml files if you want...
Please share your filebeat config file and the filebeat logs.
###################### Filebeat Configuration Example #########################
This file is an example configuration file highlighting only the most common
options. The filebeat.full.yml file from the same directory contains all the
supported options with more comments. You can use it as a reference.
You can find the full configuration reference here:
https://www.elastic.co/guide/en/beats/filebeat/index.html
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
Each - is a prospector. Most options can be set at the prospector level, so
you can use different prospectors for various configurations.
Below are the prospector specific configurations.
-
input_type: log
Paths that should be crawled and fetched. Glob based paths.
paths:
- C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\RDS69_PAD32_PRB46_F1_WEU_Iteration1.log
#- c:\programdata\elasticsearch\logs*
Exclude lines. A list of regular expressions to match. It drops the lines that are
matching any regular expression from the list.
#exclude_lines: ["^DBG"]
Include lines. A list of regular expressions to match. It exports the lines that are
matching any regular expression from the list.
#include_lines: ["^ERR", "^WARN"]
Exclude files. A list of regular expressions to match. Filebeat drops the files that
are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: [".gz$"]
Optional additional fields. These field can be freely picked
to add additional information to the crawled log files for filtering
#fields:
level: debug
review: 1
Multiline options
Mutiline can be used for log messages spanning multiple lines. This is common
for Java Stack Traces or C-Line Continuation
The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
#multiline.pattern: ^[
Defines if the pattern set under pattern should be negated or not. Default is false.
#multiline.negate: false
Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
that was (not) matched before or after or as long as a pattern is not matched based on negate.
Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
#multiline.match: after
- C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\RDS69_PAD32_PRB46_F1_WEU_Iteration1.log
#================================ General =====================================
The name of the shipper that publishes the network data. It can be used to group
all the transactions sent by a single shipper in the web interface.
#name:
The tags of the shipper are included in their own field with each
transaction published.
#tags: ["service-X", "web-tier"]
Optional fields that you can specify to add additional information to the
output.
#fields:
env: staging
#================================ Outputs =====================================
Configure what outputs to use when sending the data collected by the beat.
Multiple outputs may be used.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
Array of hosts to connect to.
hosts: ["localhost:9200"]
#template.name : "filebeat"
#template.path : "C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\software_ELK\Filebeat\filebeat.template.json"
#template.overwrite : true
Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
#output.logstash:
The Logstash hosts
#hosts: ["localhost:5044"]
#protocol: "http"
Optional SSL. By default is off.
List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
#================================ Logging =====================================
Sets log level. The default log level is info.
Available log levels are: critical, error, warning, info, debug
#logging.level: debug
At debug level, you can selectively enable logging only for some components.
To enable all selectors use ["*"]. Examples of other selectors are "beat",
"publish", "service".
#logging.selectors: ["*"]
2016-11-30T11:10:27+05:30 INFO Home path: [C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\software_ELK\Filebeat] Config path: [C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\software_ELK\Filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\software_ELK\Filebeat\logs]
2016-11-30T11:10:27+05:30 INFO Setup Beat: filebeat; Version: 5.0.1
2016-11-30T11:10:27+05:30 INFO Loading template enabled. Reading template file: C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\software_ELK\Filebeat\filebeat.template.json
2016-11-30T11:10:27+05:30 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\software_ELK\Filebeat\filebeat.template-es2x.json
2016-11-30T11:10:27+05:30 INFO Elasticsearch url: http://localhost:9200
2016-11-30T11:10:27+05:30 INFO Activated elasticsearch as output plugin.
2016-11-30T11:10:27+05:30 INFO Publisher name: 3INLAPARADKA
2016-11-30T11:10:27+05:30 INFO Flush Interval set to: 1s
2016-11-30T11:10:27+05:30 INFO Max Bulk Size set to: 50
2016-11-30T11:10:27+05:30 INFO filebeat start running.
2016-11-30T11:10:27+05:30 INFO Registry file set to: C:\ProgramData\filebeat\registry
2016-11-30T11:10:27+05:30 INFO Loading registrar data from C:\ProgramData\filebeat\registry
2016-11-30T11:10:27+05:30 INFO States Loaded from registrar: 0
2016-11-30T11:10:27+05:30 INFO Loading Prospectors: 1
2016-11-30T11:10:27+05:30 INFO Starting Registrar
2016-11-30T11:10:27+05:30 INFO Load previous states from registry into memory
2016-11-30T11:10:27+05:30 INFO Previous states loaded: 0
2016-11-30T11:10:27+05:30 INFO Start sending events to output
2016-11-30T11:10:27+05:30 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2016-11-30T11:10:27+05:30 INFO Loading Prospectors completed. Number of prospectors: 1
2016-11-30T11:10:27+05:30 INFO All prospectors are initialised and running with 0 states to persist
2016-11-30T11:10:27+05:30 INFO Starting prospector of type: log
2016-11-30T11:10:27+05:30 INFO Harvester started for file: C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\RDS69_PAD32_PRB46_F1_WEU_Iteration1.log
2016-11-30T11:10:32+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:10:33+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:10:35+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:10:27+05:30 INFO Home path: [C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\software_ELK\Filebeat] Config path: [C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\software_ELK\Filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\software_ELK\Filebeat\logs]
2016-11-30T11:10:27+05:30 INFO Setup Beat: filebeat; Version: 5.0.1
2016-11-30T11:10:27+05:30 INFO Loading template enabled. Reading template file: C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\software_ELK\Filebeat\filebeat.template.json
2016-11-30T11:10:27+05:30 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\software_ELK\Filebeat\filebeat.template-es2x.json
2016-11-30T11:10:27+05:30 INFO Elasticsearch url: http://localhost:9200
2016-11-30T11:10:27+05:30 INFO Activated elasticsearch as output plugin.
2016-11-30T11:10:27+05:30 INFO Publisher name: 3INLAPARADKA
2016-11-30T11:10:27+05:30 INFO Flush Interval set to: 1s
2016-11-30T11:10:27+05:30 INFO Max Bulk Size set to: 50
2016-11-30T11:10:27+05:30 INFO filebeat start running.
2016-11-30T11:10:27+05:30 INFO Registry file set to: C:\ProgramData\filebeat\registry
2016-11-30T11:10:27+05:30 INFO Loading registrar data from C:\ProgramData\filebeat\registry
2016-11-30T11:10:27+05:30 INFO States Loaded from registrar: 0
2016-11-30T11:10:27+05:30 INFO Loading Prospectors: 1
2016-11-30T11:10:27+05:30 INFO Starting Registrar
2016-11-30T11:10:27+05:30 INFO Load previous states from registry into memory
2016-11-30T11:10:27+05:30 INFO Previous states loaded: 0
2016-11-30T11:10:27+05:30 INFO Start sending events to output
2016-11-30T11:10:27+05:30 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2016-11-30T11:10:27+05:30 INFO Loading Prospectors completed. Number of prospectors: 1
2016-11-30T11:10:27+05:30 INFO All prospectors are initialised and running with 0 states to persist
2016-11-30T11:10:27+05:30 INFO Starting prospector of type: log
2016-11-30T11:10:27+05:30 INFO Harvester started for file: C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\RDS69_PAD32_PRB46_F1_WEU_Iteration1.log
2016-11-30T11:10:32+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:10:33+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:10:35+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:10:39+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:10:47+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:10:57+05:30 INFO Non-zero metrics in the last 30s: filebeat.harvester.started=1 libbeat.es.publish.read_bytes=1965 libbeat.publisher.published_events=836 filebeat.harvester.running=1 filebeat.harvester.open_files=1 libbeat.es.publish.write_bytes=605
2016-11-30T11:11:03+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:11:27+05:30 INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_bytes=393 libbeat.es.publish.write_bytes=121
2016-11-30T11:11:35+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:11:57+05:30 INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_bytes=393 libbeat.es.publish.write_bytes=121
2016-11-30T11:12:27+05:30 INFO No non-zero metrics in the last 30s
2016-11-30T11:12:35+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:12:57+05:30 INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_bytes=393 libbeat.es.publish.write_bytes=121
2016-11-30T11:13:27+05:30 INFO No non-zero metrics in the last 30s
2016-11-30T11:13:35+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:13:57+05:30 INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_bytes=393 libbeat.es.publish.write_bytes=121
2016-11-30T11:14:27+05:30 INFO No non-zero metrics in the last 30s
2016-11-30T11:14:35+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:14:57+05:30 INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_bytes=393 libbeat.es.publish.write_bytes=121
2016-11-30T11:15:27+05:30 INFO No non-zero metrics in the last 30s
2016-11-30T11:15:32+05:30 INFO File is inactive: C:\Here\CMST-1932-EPT_Monitoring_mechanism_and_Error_Handling\RDS69_PAD32_PRB46_F1_WEU_Iteration1.log. Closing because close_inactive of 5m0s reached.
2016-11-30T11:15:36+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2016-11-30T11:15:57+05:30 INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=1 libbeat.es.publish.read_bytes=393 filebeat.harvester.running=-1 filebeat.harvester.open_files=-1 libbeat.es.publish.write_bytes=121
2016-11-30T11:16:27+05:30 INFO No non-zero metrics in the last 30s
Looks like elasticsearch needs username / password but you didn't configure any.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.