ERR failed to initialize elasticsearch plugin as output: missing condition

Hello can some please help me get the condition set correctly. I want to create multiple index based on source file. So i used document_type to identify the different input file. When i try to specify condition i get error. I am not using logstash.

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.


- input_type: log
  json.keys_under_root: true
  json.add_error_key: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
#- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    #- C:\temp\sunny\*
    #- C:\temp\DK\*
    #- C:\temp\WW\*
    #- C:\temp\NW\*
    #- C:\temp\HS\*
    - C:\TestRail\TestRailResults.json
  document_type: TestRailResults
    - C:\TestRail\TestRailResults1.json
  document_type: TestRailResults1

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  index: "sunny_try"
  indices:
    - index: "TestRail"
      when.document_type : TestRailResults1

Update code as below, now i dont get error and new index is created based on field.type. But now the first file is not getting pushed. below is msg i get in log file
In log file i get this msg:
No output is defined to store the topology. The server fields might not be filled.

updated code:

#=========================== Filebeat prospectors =============================

filebeat.prospectors:


- input_type: log
  json.keys_under_root: true
  json.add_error_key: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
#- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    #- C:\temp\sunny\*
    #- C:\temp\DK\*
    #- C:\temp\WW\*
    #- C:\temp\NW\*
    #- C:\temp\HS\*
    - C:\TestRail\TestRailResults.json
  #document_type: TestRailResults
    - C:\TestRail\TestRailResults1.json
  fields:
   type: "TestRailResults1"

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  index: "sunny_try"
  indices:
    - index: "testrail"
      when.equals:
       fields.type: "TestRailResults1"

Hello @sunny1,

I think what is happening is the following, Filebeat has already read the file at some point, and the content never change after that, so for Filebeat, it doesn't have to reread it.

Filebeat keeps track of the offset of the watched files in data/registry; I suggest you delete that file, and you restart Logstash, by doing so you should see events in your Elasticsearch indices.

Also try increasing Filebeat verbosity with the following options, this will help with further debugging.

./filebeat -c yourconfig.yml -v -e -d "*"

Hello Pier, thks for ur reply. But i had changed the file to see filebeat fetch it again. But it did not happen. In the log i get no output is defined to stor topology. I will try to do verbose log and see if it helps to identify the root cause.

Also i dont use logstash. I output directly to elasticsearch.

What i intend to do is, i have two files that get updated daily. I want to create separate index for both files based on type of file. Not able to achieve that.

-Sunny

In this case it's similar to what I've replied in Using a regex in the custom field of Filebeat

Pier,

I dont want to extract dynamic field and apply filter. I am ok with adding custom field while reading the file using fields and then use it as filter to create separate index. Is this not possible with filebeat?

--Sunny

In this case you can do the following, in your filebeat YAML configure the fields for each inputs:

- type: log
  enabled: true
  paths:
    - /var/log/*.log
  fields:
    document_type: "mycustomtype"

In the elasticsearch output you can use field reference to access this field and configure the index dynamically:

 index: "filebeat-%{[beat.version]}-%{[fields][document_type]}"

You will also have to uncomment or adjust the following options to make sure the template is correctly applied to your data.

setup.template.name: "filebeat-%{[beat.version]}"
setup.template.pattern: "filebeat-%{[beat.version]}-*"

This will on the fly create the right index based on your data.

Sorry Pier, i am new to this, please bear with me. I am positing the entire filebeat.yml file and output log file. Please help me correct the config. I tried as you mentioned but still only one index was created. i wil attach the input files as well.

Filebeat yml file:

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat prospectors =============================

filebeat.prospectors:


- input_type: log
  json.keys_under_root: true
  json.add_error_key: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
  
    - C:\TestRail\TestRailResults12.json
    
  fields:
   document_type: "testrailresults12"

- input_type: log
  json.keys_under_root: true
  json.add_error_key: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:   
    - C:\TestRail\TestRailResults13.json
    
  fields:
   document_type: "testrailresults13"

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ["^DBG"]

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ["^ERR", "^WARN"]

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: [".gz$"]

  # Optional additional fields. These field can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after


#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  index: "filebeat-%{[beat.version]}-%{[fields][document_type]}"
  #indices:
  
    #- index: "testrailnew12"
      #when.equals:
       #fields.type: "testrailresults12"
       
    #- index: "testrailnew13"
      #when.equals:
       #fields.type: "testrailresults13"      
     
       
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
 # hosts: ["localhost:5044"]
  
#output.console:
 # enabled: true
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # ClientCertificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

Output log:

2018-03-30T11:26:48-07:00 DBG Disable stderr logging
2018-03-30T11:26:48-07:00 INFO Home path: [C:\Users\sgorasia\Downloads\filebeat] Config path: [C:\Users\sgorasia\Downloads\filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\Users\sgorasia\Downloads\filebeat\logs]
2018-03-30T11:26:48-07:00 INFO Setup Beat: filebeat; Version: 5.5.1
2018-03-30T11:26:48-07:00 DBG Processors:
2018-03-30T11:26:48-07:00 DBG Initializing output plugins
2018-03-30T11:26:48-07:00 INFO Loading template enabled. Reading template file: C:\Users\sgorasia\Downloads\filebeat\filebeat.template.json
2018-03-30T11:26:48-07:00 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: C:\Users\sgorasia\Downloads\filebeat\filebeat.template-es2x.json
2018-03-30T11:26:48-07:00 INFO Loading template enabled for Elasticsearch 6.x. Reading template file: C:\Users\sgorasia\Downloads\filebeat\filebeat.template-es6x.json
2018-03-30T11:26:48-07:00 INFO Elasticsearch url: http://localhost:9200
2018-03-30T11:26:48-07:00 INFO Activated elasticsearch as output plugin.
2018-03-30T11:26:48-07:00 DBG Create output worker
2018-03-30T11:26:48-07:00 DBG No output is defined to store the topology. The server fields might not be filled.
2018-03-30T11:26:48-07:00 INFO Publisher name: 5CG722343K
2018-03-30T11:26:48-07:00 INFO Flush Interval set to: 1s
2018-03-30T11:26:48-07:00 INFO Max Bulk Size set to: 50
2018-03-30T11:26:48-07:00 DBG create bulk processing worker (interval=1s, bulk size=50)
2018-03-30T11:26:48-07:00 INFO filebeat start running.
2018-03-30T11:26:48-07:00 INFO Registry file set to: C:\ProgramData\filebeat\registry
2018-03-30T11:26:48-07:00 INFO Loading registrar data from C:\ProgramData\filebeat\registry
2018-03-30T11:26:48-07:00 DBG Windows is interactive: false
2018-03-30T11:26:48-07:00 INFO States Loaded from registrar: 6
2018-03-30T11:26:48-07:00 INFO Loading Prospectors: 2
2018-03-30T11:26:48-07:00 DBG File Configs: [C:\TestRail\TestRailResults12.json]
2018-03-30T11:26:48-07:00 DBG exclude_files: []
2018-03-30T11:26:48-07:00 INFO Prospector with previous states loaded: 0
2018-03-30T11:26:48-07:00 INFO Starting prospector of type: log; id: 13305660674938637783
2018-03-30T11:26:48-07:00 DBG File Configs: [C:\TestRail\TestRailResults13.json]
2018-03-30T11:26:48-07:00 DBG exclude_files: []
2018-03-30T11:26:48-07:00 INFO Prospector with previous states loaded: 0
2018-03-30T11:26:48-07:00 INFO Starting prospector of type: log; id: 5053548534983409642
2018-03-30T11:26:48-07:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 2
2018-03-30T11:26:48-07:00 INFO Starting Registrar
2018-03-30T11:26:48-07:00 INFO Start sending events to output
2018-03-30T11:26:48-07:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018-03-30T11:26:48-07:00 DBG Start next scan
2018-03-30T11:26:48-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults12.json
2018-03-30T11:26:48-07:00 DBG Start harvester for new file: C:\TestRail\TestRailResults12.json
2018-03-30T11:26:48-07:00 DBG Start next scan
2018-03-30T11:26:48-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults13.json
2018-03-30T11:26:48-07:00 DBG New state added for C:\TestRail\TestRailResults12.json
2018-03-30T11:26:48-07:00 DBG Start harvester for new file: C:\TestRail\TestRailResults13.json
2018-03-30T11:26:48-07:00 DBG Setting offset for file based on seek: C:\TestRail\TestRailResults12.json
2018-03-30T11:26:48-07:00 DBG Setting offset for file: C:\TestRail\TestRailResults12.json. Offset: 0
2018-03-30T11:26:48-07:00 DBG Prospector states cleaned up. Before: 1, After: 1
2018-03-30T11:26:48-07:00 DBG New state added for C:\TestRail\TestRailResults13.json
2018-03-30T11:26:48-07:00 INFO Harvester started for file: C:\TestRail\TestRailResults12.json
2018-03-30T11:26:48-07:00 DBG Setting offset for file based on seek: C:\TestRail\TestRailResults13.json
2018-03-30T11:26:48-07:00 DBG Setting offset for file: C:\TestRail\TestRailResults13.json. Offset: 0
2018-03-30T11:26:48-07:00 DBG Prospector states cleaned up. Before: 1, After: 1
2018-03-30T11:26:48-07:00 INFO Harvester started for file: C:\TestRail\TestRailResults13.json
2018-03-30T11:26:48-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.
2018-03-30T11:26:48-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.
2018-03-30T11:26:49-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.
2018-03-30T11:26:49-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.
2018-03-30T11:26:51-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.
2018-03-30T11:26:51-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.
2018-03-30T11:26:53-07:00 DBG Flushing spooler because of timeout. Events flushed: 3
2018-03-30T11:26:53-07:00 DBG Publish: {
"@timestamp": "2018-03-30T18:26:48.868Z",
"Automated": "No",
"AutomationId": "",
"AutomationScenarioName": "",
"Defects": "",
"EpicId": "76580",
"Estimate": "",
"Level": "System",
"Milestone": "Night's Watch - Sprint 4",
"Name": "itdoesnotwork",
"Priority": "Medium",
"References": "113740",
"Type": "API/Services",
"beat": {
"hostname": "5CG722343K",
"name": "5CG722343K",
"version": "5.5.1"
},
"fields": {
"document_type": "testrailresults13"
},
"input_type": "log",
"offset": 263,
"source": "C:\TestRail\TestRailResults13.json",
"type": "log"
}
2018-03-30T11:26:53-07:00 DBG output worker: publish 1 events
2018-03-30T11:26:53-07:00 DBG ES Ping(url=http://localhost:9200, timeout=1m30s)
2018-03-30T11:26:53-07:00 DBG Ping status code: 200
2018-03-30T11:26:53-07:00 INFO Connected to Elasticsearch version 5.5.1
2018-03-30T11:26:53-07:00 INFO Trying to load template for client: http://localhost:9200
2018-03-30T11:26:53-07:00 DBG HEAD http://localhost:9200/_template/filebeat
2018-03-30T11:26:53-07:00 INFO Template already exists and will not be overwritten.
2018-03-30T11:26:54-07:00 DBG PublishEvents: 1 events have been published to elasticsearch in 243ms.
2018-03-30T11:26:54-07:00 DBG send completed
2018-03-30T11:26:54-07:00 DBG Events sent: 3
2018-03-30T11:26:54-07:00 DBG Processing 3 events
2018-03-30T11:26:54-07:00 DBG New state added for C:\TestRail\TestRailResults12.json
2018-03-30T11:26:54-07:00 DBG New state added for C:\TestRail\TestRailResults13.json
2018-03-30T11:26:54-07:00 DBG Registrar states cleaned up. Before: 8, After: 8
2018-03-30T11:26:54-07:00 DBG Write registry file: C:\ProgramData\filebeat\registry
2018-03-30T11:26:54-07:00 DBG Registry file updated. 8 states written.

Lets find out what worked, from the debug log we see that the document has the fields/document_type

I only see a document from C:\TestRail\TestRailResults13.json So the other file is not read, What is the content of TestRailResult12.json?

Yes exactly it always reads the second file only. the file is json type but here its not allowed to attach file. So could not attach the input files. The file content is exactly similar to other file but the name field has different value.

So its either one of the following:

  1. The offset is saved in data/registry so it doesn't read it again.
  2. OR there is no end of line (\n) at the end of the file so it doesn't know that an event is completely written in the file.

Sorry was in process to post complete file but it was more than 7000 charater so had to truncate it.

Let me see how i can post the entire file

Here some of it. I have more in log file but its just repeat of what i already posted.

018-03-30T11:26:55-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.
2018-03-30T11:26:55-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.
2018-03-30T11:26:58-07:00 DBG Run prospector
2018-03-30T11:26:58-07:00 DBG Start next scan
2018-03-30T11:26:58-07:00 DBG Flushing spooler because of timeout. Events flushed: 0
2018-03-30T11:26:58-07:00 DBG Run prospector
2018-03-30T11:26:58-07:00 DBG Start next scan
2018-03-30T11:26:58-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults13.json
2018-03-30T11:26:58-07:00 DBG Update existing file for harvesting: C:\TestRail\TestRailResults13.json, offset: 263
2018-03-30T11:26:58-07:00 DBG Harvester for file is still running: C:\TestRail\TestRailResults13.json
2018-03-30T11:26:58-07:00 DBG Prospector states cleaned up. Before: 1, After: 1
2018-03-30T11:26:58-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults12.json
2018-03-30T11:26:58-07:00 DBG Update existing file for harvesting: C:\TestRail\TestRailResults12.json, offset: 0
2018-03-30T11:26:58-07:00 DBG Harvester for file is still running: C:\TestRail\TestRailResults12.json
2018-03-30T11:26:58-07:00 DBG Prospector states cleaned up. Before: 1, After: 1
2018-03-30T11:27:03-07:00 DBG Flushing spooler because of timeout. Events flushed: 0
2018-03-30T11:27:03-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.
2018-03-30T11:27:03-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.
2018-03-30T11:27:08-07:00 DBG Flushing spooler because of timeout. Events flushed: 0
2018-03-30T11:27:08-07:00 DBG Run prospector
2018-03-30T11:27:08-07:00 DBG Start next scan
2018-03-30T11:27:08-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults13.json
2018-03-30T11:27:08-07:00 DBG Update existing file for harvesting: C:\TestRail\TestRailResults13.json, offset: 263
2018-03-30T11:27:08-07:00 DBG Harvester for file is still running: C:\TestRail\TestRailResults13.json
2018-03-30T11:27:08-07:00 DBG Prospector states cleaned up. Before: 1, After: 1
2018-03-30T11:27:08-07:00 DBG Run prospector
2018-03-30T11:27:08-07:00 DBG Start next scan
2018-03-30T11:27:08-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults12.json
2018-03-30T11:27:08-07:00 DBG Update existing file for harvesting: C:\TestRail\TestRailResults12.json, offset: 0
2018-03-30T11:27:08-07:00 DBG Harvester for file is still running: C:\TestRail\TestRailResults12.json
2018-03-30T11:27:08-07:00 DBG Prospector states cleaned up. Before: 1, After: 1
2018-03-30T11:27:13-07:00 DBG Flushing spooler because of timeout. Events flushed: 0
2018-03-30T11:27:13-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.
2018-03-30T11:27:13-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.
2018-03-30T11:27:18-07:00 DBG Flushing spooler because of timeout. Events flushed: 0
2018-03-30T11:27:18-07:00 DBG Run prospector
2018-03-30T11:27:18-07:00 DBG Start next scan
2018-03-30T11:27:18-07:00 DBG Run prospector
2018-03-30T11:27:18-07:00 DBG Start next scan

Can you add a new line at the end of this file? The log says it stock at the beginning.

The file 12 has only one line as of now:

{"Defects":"","Name":"yesitworks","Type":"API/Services","Priority":"Medium","Estimate":"","Milestone":"Night's Watch - Sprint 4","References":"113740","Level":"System","AutomationScenarioName":"","Automated":"No","AutomationId":"","EpicId":"76580"}

Do you want me to add another line after this ? Because the file beat is not reading even this line to start with.

The file content for both files are almost same only the name field has different values.

Internally Filebeat detect LINE by checking if a line terminates by \n character, I understand there is only one line in your document but I think that there is no newline character at the end of that line. So Filebeat thinks there is nothing to read in that document.

Open the file in your editor go at the end of the line, press return and save the file.

Any application that write theses document will have to terminate each line with a \n, every logging library does that.

Sure doing it now.