Hello can some please help me get the condition set correctly. I want to create multiple index based on source file. So i used document_type to identify the different input file. When i try to specify condition i get error. I am not using logstash.
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- input_type: log
json.keys_under_root: true
json.add_error_key: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
#- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
#- C:\temp\sunny\*
#- C:\temp\DK\*
#- C:\temp\WW\*
#- C:\temp\NW\*
#- C:\temp\HS\*
- C:\TestRail\TestRailResults.json
document_type: TestRailResults
- C:\TestRail\TestRailResults1.json
document_type: TestRailResults1
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
index: "sunny_try"
indices:
- index: "TestRail"
when.document_type : TestRailResults1
Update code as below, now i dont get error and new index is created based on field.type. But now the first file is not getting pushed. below is msg i get in log file
In log file i get this msg:
No output is defined to store the topology. The server fields might not be filled.
filebeat.prospectors:
- input_type: log
json.keys_under_root: true
json.add_error_key: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
#- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
#- C:\temp\sunny\*
#- C:\temp\DK\*
#- C:\temp\WW\*
#- C:\temp\NW\*
#- C:\temp\HS\*
- C:\TestRail\TestRailResults.json
#document_type: TestRailResults
- C:\TestRail\TestRailResults1.json
fields:
type: "TestRailResults1"
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
I think what is happening is the following, Filebeat has already read the file at some point, and the content never change after that, so for Filebeat, it doesn't have to reread it.
Filebeat keeps track of the offset of the watched files in data/registry; I suggest you delete that file, and you restart Logstash, by doing so you should see events in your Elasticsearch indices.
Also try increasing Filebeat verbosity with the following options, this will help with further debugging.
Hello Pier, thks for ur reply. But i had changed the file to see filebeat fetch it again. But it did not happen. In the log i get no output is defined to stor topology. I will try to do verbose log and see if it helps to identify the root cause.
Also i dont use logstash. I output directly to elasticsearch.
What i intend to do is, i have two files that get updated daily. I want to create separate index for both files based on type of file. Not able to achieve that.
I dont want to extract dynamic field and apply filter. I am ok with adding custom field while reading the file using fields and then use it as filter to create separate index. Is this not possible with filebeat?
Sorry Pier, i am new to this, please bear with me. I am positing the entire filebeat.yml file and output log file. Please help me correct the config. I tried as you mentioned but still only one index was created. i wil attach the input files as well.
Filebeat yml file:
###################### Filebeat Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The filebeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
- input_type: log
json.keys_under_root: true
json.add_error_key: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- C:\TestRail\TestRailResults12.json
fields:
document_type: "testrailresults12"
- input_type: log
json.keys_under_root: true
json.add_error_key: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- C:\TestRail\TestRailResults13.json
fields:
document_type: "testrailresults13"
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ["^DBG"]
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ["^ERR", "^WARN"]
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: [".gz$"]
# Optional additional fields. These field can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
### Multiline options
# Mutiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
#multiline.pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
#multiline.negate: false
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
#multiline.match: after
#================================ General =====================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
#================================ Outputs =====================================
# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
index: "filebeat-%{[beat.version]}-%{[fields][document_type]}"
#indices:
#- index: "testrailnew12"
#when.equals:
#fields.type: "testrailresults12"
#- index: "testrailnew13"
#when.equals:
#fields.type: "testrailresults13"
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
#output.logstash:
# The Logstash hosts
# hosts: ["localhost:5044"]
#output.console:
# enabled: true
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# ClientCertificate Key
#ssl.key: "/etc/pki/client/cert.key"
#================================ Logging =====================================
# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
2018-03-30T11:26:48-07:00 DBG Disable stderr logging
2018-03-30T11:26:48-07:00 INFO Home path: [C:\Users\sgorasia\Downloads\filebeat] Config path: [C:\Users\sgorasia\Downloads\filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\Users\sgorasia\Downloads\filebeat\logs]
2018-03-30T11:26:48-07:00 INFO Setup Beat: filebeat; Version: 5.5.1
2018-03-30T11:26:48-07:00 DBG Processors:
2018-03-30T11:26:48-07:00 DBG Initializing output plugins
2018-03-30T11:26:48-07:00 INFO Loading template enabled. Reading template file: C:\Users\sgorasia\Downloads\filebeat\filebeat.template.json
2018-03-30T11:26:48-07:00 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: C:\Users\sgorasia\Downloads\filebeat\filebeat.template-es2x.json
2018-03-30T11:26:48-07:00 INFO Loading template enabled for Elasticsearch 6.x. Reading template file: C:\Users\sgorasia\Downloads\filebeat\filebeat.template-es6x.json
2018-03-30T11:26:48-07:00 INFO Elasticsearch url: http://localhost:9200
2018-03-30T11:26:48-07:00 INFO Activated elasticsearch as output plugin.
2018-03-30T11:26:48-07:00 DBG Create output worker
2018-03-30T11:26:48-07:00 DBG No output is defined to store the topology. The server fields might not be filled.
2018-03-30T11:26:48-07:00 INFO Publisher name: 5CG722343K
2018-03-30T11:26:48-07:00 INFO Flush Interval set to: 1s
2018-03-30T11:26:48-07:00 INFO Max Bulk Size set to: 50
2018-03-30T11:26:48-07:00 DBG create bulk processing worker (interval=1s, bulk size=50)
2018-03-30T11:26:48-07:00 INFO filebeat start running.
2018-03-30T11:26:48-07:00 INFO Registry file set to: C:\ProgramData\filebeat\registry
2018-03-30T11:26:48-07:00 INFO Loading registrar data from C:\ProgramData\filebeat\registry
2018-03-30T11:26:48-07:00 DBG Windows is interactive: false
2018-03-30T11:26:48-07:00 INFO States Loaded from registrar: 6
2018-03-30T11:26:48-07:00 INFO Loading Prospectors: 2
2018-03-30T11:26:48-07:00 DBG File Configs: [C:\TestRail\TestRailResults12.json]
2018-03-30T11:26:48-07:00 DBG exclude_files: []
2018-03-30T11:26:48-07:00 INFO Prospector with previous states loaded: 0
2018-03-30T11:26:48-07:00 INFO Starting prospector of type: log; id: 13305660674938637783
2018-03-30T11:26:48-07:00 DBG File Configs: [C:\TestRail\TestRailResults13.json]
2018-03-30T11:26:48-07:00 DBG exclude_files: []
2018-03-30T11:26:48-07:00 INFO Prospector with previous states loaded: 0
2018-03-30T11:26:48-07:00 INFO Starting prospector of type: log; id: 5053548534983409642
2018-03-30T11:26:48-07:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 2
2018-03-30T11:26:48-07:00 INFO Starting Registrar
2018-03-30T11:26:48-07:00 INFO Start sending events to output
2018-03-30T11:26:48-07:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018-03-30T11:26:48-07:00 DBG Start next scan
2018-03-30T11:26:48-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults12.json
2018-03-30T11:26:48-07:00 DBG Start harvester for new file: C:\TestRail\TestRailResults12.json
2018-03-30T11:26:48-07:00 DBG Start next scan
2018-03-30T11:26:48-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults13.json
2018-03-30T11:26:48-07:00 DBG New state added for C:\TestRail\TestRailResults12.json
2018-03-30T11:26:48-07:00 DBG Start harvester for new file: C:\TestRail\TestRailResults13.json
2018-03-30T11:26:48-07:00 DBG Setting offset for file based on seek: C:\TestRail\TestRailResults12.json
2018-03-30T11:26:48-07:00 DBG Setting offset for file: C:\TestRail\TestRailResults12.json. Offset: 0
2018-03-30T11:26:48-07:00 DBG Prospector states cleaned up. Before: 1, After: 1
2018-03-30T11:26:48-07:00 DBG New state added for C:\TestRail\TestRailResults13.json
2018-03-30T11:26:48-07:00 INFO Harvester started for file: C:\TestRail\TestRailResults12.json
2018-03-30T11:26:48-07:00 DBG Setting offset for file based on seek: C:\TestRail\TestRailResults13.json
2018-03-30T11:26:48-07:00 DBG Setting offset for file: C:\TestRail\TestRailResults13.json. Offset: 0
2018-03-30T11:26:48-07:00 DBG Prospector states cleaned up. Before: 1, After: 1
2018-03-30T11:26:48-07:00 INFO Harvester started for file: C:\TestRail\TestRailResults13.json
2018-03-30T11:26:48-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.
2018-03-30T11:26:48-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.
2018-03-30T11:26:49-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.
2018-03-30T11:26:49-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.
2018-03-30T11:26:51-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.
2018-03-30T11:26:51-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.
2018-03-30T11:26:53-07:00 DBG Flushing spooler because of timeout. Events flushed: 3
2018-03-30T11:26:53-07:00 DBG Publish: {
"@timestamp": "2018-03-30T18:26:48.868Z",
"Automated": "No",
"AutomationId": "",
"AutomationScenarioName": "",
"Defects": "",
"EpicId": "76580",
"Estimate": "",
"Level": "System",
"Milestone": "Night's Watch - Sprint 4",
"Name": "itdoesnotwork",
"Priority": "Medium",
"References": "113740",
"Type": "API/Services",
"beat": {
"hostname": "5CG722343K",
"name": "5CG722343K",
"version": "5.5.1"
},
"fields": {
"document_type": "testrailresults13"
},
"input_type": "log",
"offset": 263,
"source": "C:\TestRail\TestRailResults13.json",
"type": "log"
}
2018-03-30T11:26:53-07:00 DBG output worker: publish 1 events
2018-03-30T11:26:53-07:00 DBG ES Ping(url=http://localhost:9200, timeout=1m30s)
2018-03-30T11:26:53-07:00 DBG Ping status code: 200
2018-03-30T11:26:53-07:00 INFO Connected to Elasticsearch version 5.5.1
2018-03-30T11:26:53-07:00 INFO Trying to load template for client: http://localhost:9200
2018-03-30T11:26:53-07:00 DBG HEAD http://localhost:9200/_template/filebeat
2018-03-30T11:26:53-07:00 INFO Template already exists and will not be overwritten.
2018-03-30T11:26:54-07:00 DBG PublishEvents: 1 events have been published to elasticsearch in 243ms.
2018-03-30T11:26:54-07:00 DBG send completed
2018-03-30T11:26:54-07:00 DBG Events sent: 3
2018-03-30T11:26:54-07:00 DBG Processing 3 events
2018-03-30T11:26:54-07:00 DBG New state added for C:\TestRail\TestRailResults12.json
2018-03-30T11:26:54-07:00 DBG New state added for C:\TestRail\TestRailResults13.json
2018-03-30T11:26:54-07:00 DBG Registrar states cleaned up. Before: 8, After: 8
2018-03-30T11:26:54-07:00 DBG Write registry file: C:\ProgramData\filebeat\registry
2018-03-30T11:26:54-07:00 DBG Registry file updated. 8 states written.
Yes exactly it always reads the second file only. the file is json type but here its not allowed to attach file. So could not attach the input files. The file content is exactly similar to other file but the name field has different value.
Internally Filebeat detect LINE by checking if a line terminates by \n character, I understand there is only one line in your document but I think that there is no newline character at the end of that line. So Filebeat thinks there is nothing to read in that document.
Open the file in your editor go at the end of the line, press return and save the file.
Any application that write theses document will have to terminate each line with a \n, every logging library does that.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.