# ERR failed to initialize elasticsearch plugin as output: missing condition

**URL:** <https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 29, 2018, 10:26pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162 "2018-03-29T22:26:44Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 29, 2018, 10:26pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/1 "2018-03-29T22:26:44Z")

</div>

Hello can some please help me get the condition set correctly. I want to create multiple index based on source file. So i used document\_type to identify the different input file. When i try to specify condition i get error. I am not using logstash.

```
filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log
  json.keys_under_root: true
  json.add_error_key: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
#- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    #- C:\temp\sunny\*
    #- C:\temp\DK\*
    #- C:\temp\WW\*
    #- C:\temp\NW\*
    #- C:\temp\HS\*
    - C:\TestRail\TestRailResults.json
  document_type: TestRailResults
    - C:\TestRail\TestRailResults1.json
  document_type: TestRailResults1

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  index: "sunny_try"
  indices:
    - index: "TestRail"
      when.document_type : TestRailResults1
```

---

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 29, 2018, 11:17pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/2 "2018-03-29T23:17:21Z")

</div>

Update code as below, now i dont get error and new index is created based on field.type. But now the first file is not getting pushed. below is msg i get in log file  
In log file i get this msg:  
No output is defined to store the topology. The server fields might not be filled.

updated code:

#=========================== Filebeat prospectors =============================

```
filebeat.prospectors:

- input_type: log
  json.keys_under_root: true
  json.add_error_key: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
#- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    #- C:\temp\sunny\*
    #- C:\temp\DK\*
    #- C:\temp\WW\*
    #- C:\temp\NW\*
    #- C:\temp\HS\*
    - C:\TestRail\TestRailResults.json
  #document_type: TestRailResults
    - C:\TestRail\TestRailResults1.json
  fields:
   type: "TestRailResults1"

```

#-------------------------- Elasticsearch output ------------------------------  
output.elasticsearch:  
# Array of hosts to connect to.  
hosts: ["localhost:9200"]

```
  # Optional protocol and basic auth credentials.
  index: "sunny_try"
  indices:
    - index: "testrail"
      when.equals:
       fields.type: "TestRailResults1"
```

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 1:49pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/3 "2018-03-30T13:49:40Z")

</div>

Hello @sunny1,

I think what is happening is the following, Filebeat has already read the file at some point, and the content never change after that, so for Filebeat, it doesn't have to reread it.

Filebeat keeps track of the offset of the watched files in data/registry; I suggest you delete that file, and you restart Logstash, by doing so you should see events in your Elasticsearch indices.

Also try increasing Filebeat verbosity with the following options, this will help with further debugging.

```auto
./filebeat -c yourconfig.yml -v -e -d "*"

```

---

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 30, 2018, 5:34pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/4 "2018-03-30T17:34:17Z")

</div>

Hello Pier, thks for ur reply. But i had changed the file to see filebeat fetch it again. But it did not happen. In the log i get no output is defined to stor topology. I will try to do verbose log and see if it helps to identify the root cause.

Also i dont use logstash. I output directly to elasticsearch.

What i intend to do is, i have two files that get updated daily. I want to create separate index for both files based on type of file. Not able to achieve that.

-Sunny

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 5:36pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/5 "2018-03-30T17:36:18Z")

</div>

In this case it's similar to what I've replied in [Using a regex in the custom field of Filebeat](https://discuss.elastic.co/t/using-a-regex-in-the-custom-field-of-filebeat/126252/2?u=pierhugues)

---

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 30, 2018, 5:40pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/6 "2018-03-30T17:40:00Z")

</div>

Pier,

I dont want to extract dynamic field and apply filter. I am ok with adding custom field while reading the file using fields and then use it as filter to create separate index. Is this not possible with filebeat?

--Sunny

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 6:20pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/7 "2018-03-30T18:20:35Z")

</div>

In this case you can do the following, in your filebeat YAML configure the fields for each inputs:

```auto
- type: log
  enabled: true
  paths:
    - /var/log/*.log
  fields:
    document_type: "mycustomtype"

```

In the elasticsearch output you can use field reference to access this field and configure the index dynamically:

```auto
 index: "filebeat-%{[beat.version]}-%{[fields][document_type]}"

```

You will also have to uncomment or adjust the following options to make sure the template is correctly applied to your data.

```auto
setup.template.name: "filebeat-%{[beat.version]}"
setup.template.pattern: "filebeat-%{[beat.version]}-*"

```

This will on the fly create the right index based on your data.

---

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 30, 2018, 6:34pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/8 "2018-03-30T18:34:35Z")

</div>

Sorry Pier, i am new to this, please bear with me. I am positing the entire filebeat.yml file and output log file. Please help me correct the config. I tried as you mentioned but still only one index was created. i wil attach the input files as well.

Filebeat yml file:

###################### Filebeat Configuration Example #########################

```
# This file is an example configuration file highlighting only the most common
# options. The filebeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

- input_type: log
  json.keys_under_root: true
  json.add_error_key: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
  
    - C:\TestRail\TestRailResults12.json
    
  fields:
   document_type: "testrailresults12"

- input_type: log
  json.keys_under_root: true
  json.add_error_key: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:   
    - C:\TestRail\TestRailResults13.json
    
  fields:
   document_type: "testrailresults13"

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ["^DBG"]

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ["^ERR", "^WARN"]

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: [".gz$"]

  # Optional additional fields. These field can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  # level: debug
  # review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  index: "filebeat-%{[beat.version]}-%{[fields][document_type]}"
  #indices:
  
    #- index: "testrailnew12"
      #when.equals:
       #fields.type: "testrailresults12"
       
    #- index: "testrailnew13"
      #when.equals:
       #fields.type: "testrailresults13"      
     
       
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
 # hosts: ["localhost:5044"]
  
#output.console:
 # enabled: true
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # ClientCertificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
```

---

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 30, 2018, 6:36pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/9 "2018-03-30T18:36:03Z")

</div>

Output log:

2018-03-30T11:26:48-07:00 DBG Disable stderr logging  
2018-03-30T11:26:48-07:00 INFO Home path: [C:\Users\sgorasia\Downloads\filebeat] Config path: [C:\Users\sgorasia\Downloads\filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\Users\sgorasia\Downloads\filebeat\logs]  
2018-03-30T11:26:48-07:00 INFO Setup Beat: filebeat; Version: 5.5.1  
2018-03-30T11:26:48-07:00 DBG Processors:  
2018-03-30T11:26:48-07:00 DBG Initializing output plugins  
2018-03-30T11:26:48-07:00 INFO Loading template enabled. Reading template file: C:\Users\sgorasia\Downloads\filebeat\filebeat.template.json  
2018-03-30T11:26:48-07:00 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: C:\Users\sgorasia\Downloads\filebeat\filebeat.template-es2x.json  
2018-03-30T11:26:48-07:00 INFO Loading template enabled for Elasticsearch 6.x. Reading template file: C:\Users\sgorasia\Downloads\filebeat\filebeat.template-es6x.json  
2018-03-30T11:26:48-07:00 INFO Elasticsearch url: [http://localhost:9200](http://localhost:9200)  
2018-03-30T11:26:48-07:00 INFO Activated elasticsearch as output plugin.  
2018-03-30T11:26:48-07:00 DBG Create output worker  
2018-03-30T11:26:48-07:00 DBG No output is defined to store the topology. The server fields might not be filled.  
2018-03-30T11:26:48-07:00 INFO Publisher name: 5CG722343K  
2018-03-30T11:26:48-07:00 INFO Flush Interval set to: 1s  
2018-03-30T11:26:48-07:00 INFO Max Bulk Size set to: 50  
2018-03-30T11:26:48-07:00 DBG create bulk processing worker (interval=1s, bulk size=50)  
2018-03-30T11:26:48-07:00 INFO filebeat start running.  
2018-03-30T11:26:48-07:00 INFO Registry file set to: C:\ProgramData\filebeat\registry  
2018-03-30T11:26:48-07:00 INFO Loading registrar data from C:\ProgramData\filebeat\registry  
2018-03-30T11:26:48-07:00 DBG Windows is interactive: false  
2018-03-30T11:26:48-07:00 INFO States Loaded from registrar: 6  
2018-03-30T11:26:48-07:00 INFO Loading Prospectors: 2  
2018-03-30T11:26:48-07:00 DBG File Configs: [C:\TestRail\TestRailResults12.json]  
2018-03-30T11:26:48-07:00 DBG exclude\_files: []  
2018-03-30T11:26:48-07:00 INFO Prospector with previous states loaded: 0  
2018-03-30T11:26:48-07:00 INFO Starting prospector of type: log; id: 13305660674938637783  
2018-03-30T11:26:48-07:00 DBG File Configs: [C:\TestRail\TestRailResults13.json]  
2018-03-30T11:26:48-07:00 DBG exclude\_files: []  
2018-03-30T11:26:48-07:00 INFO Prospector with previous states loaded: 0  
2018-03-30T11:26:48-07:00 INFO Starting prospector of type: log; id: 5053548534983409642  
2018-03-30T11:26:48-07:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 2  
2018-03-30T11:26:48-07:00 INFO Starting Registrar  
2018-03-30T11:26:48-07:00 INFO Start sending events to output  
2018-03-30T11:26:48-07:00 INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s  
2018-03-30T11:26:48-07:00 DBG Start next scan  
2018-03-30T11:26:48-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults12.json  
2018-03-30T11:26:48-07:00 DBG Start harvester for new file: C:\TestRail\TestRailResults12.json  
2018-03-30T11:26:48-07:00 DBG Start next scan  
2018-03-30T11:26:48-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults13.json  
2018-03-30T11:26:48-07:00 DBG New state added for C:\TestRail\TestRailResults12.json  
2018-03-30T11:26:48-07:00 DBG Start harvester for new file: C:\TestRail\TestRailResults13.json  
2018-03-30T11:26:48-07:00 DBG Setting offset for file based on seek: C:\TestRail\TestRailResults12.json  
2018-03-30T11:26:48-07:00 DBG Setting offset for file: C:\TestRail\TestRailResults12.json. Offset: 0  
2018-03-30T11:26:48-07:00 DBG Prospector states cleaned up. Before: 1, After: 1  
2018-03-30T11:26:48-07:00 DBG New state added for C:\TestRail\TestRailResults13.json  
2018-03-30T11:26:48-07:00 INFO Harvester started for file: C:\TestRail\TestRailResults12.json  
2018-03-30T11:26:48-07:00 DBG Setting offset for file based on seek: C:\TestRail\TestRailResults13.json  
2018-03-30T11:26:48-07:00 DBG Setting offset for file: C:\TestRail\TestRailResults13.json. Offset: 0  
2018-03-30T11:26:48-07:00 DBG Prospector states cleaned up. Before: 1, After: 1  
2018-03-30T11:26:48-07:00 INFO Harvester started for file: C:\TestRail\TestRailResults13.json  
2018-03-30T11:26:48-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.  
2018-03-30T11:26:48-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.  
2018-03-30T11:26:49-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.  
2018-03-30T11:26:49-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.  
2018-03-30T11:26:51-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.  
2018-03-30T11:26:51-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.  
2018-03-30T11:26:53-07:00 DBG Flushing spooler because of timeout. Events flushed: 3  
2018-03-30T11:26:53-07:00 DBG Publish: {  
"@timestamp": "2018-03-30T18:26:48.868Z",  
"Automated": "No",  
"AutomationId": "",  
"AutomationScenarioName": "",  
"Defects": "",  
"EpicId": "76580",  
"Estimate": "",  
"Level": "System",  
"Milestone": "Night's Watch - Sprint 4",  
"Name": "itdoesnotwork",  
"Priority": "Medium",  
"References": "113740",  
"Type": "API/Services",  
"beat": {  
"hostname": "5CG722343K",  
"name": "5CG722343K",  
"version": "5.5.1"  
},  
"fields": {  
"document\_type": "testrailresults13"  
},  
"input\_type": "log",  
"offset": 263,  
"source": "C:\TestRail\TestRailResults13.json",  
"type": "log"  
}  
2018-03-30T11:26:53-07:00 DBG output worker: publish 1 events  
2018-03-30T11:26:53-07:00 DBG ES Ping(url=http://localhost:9200, timeout=1m30s)  
2018-03-30T11:26:53-07:00 DBG Ping status code: 200  
2018-03-30T11:26:53-07:00 INFO Connected to Elasticsearch version 5.5.1  
2018-03-30T11:26:53-07:00 INFO Trying to load template for client: [http://localhost:9200](http://localhost:9200)  
2018-03-30T11:26:53-07:00 DBG HEAD [http://localhost:9200/\_template/filebeat](http://localhost:9200/_template/filebeat)  
2018-03-30T11:26:53-07:00 INFO Template already exists and will not be overwritten.  
2018-03-30T11:26:54-07:00 DBG PublishEvents: 1 events have been published to elasticsearch in 243ms.  
2018-03-30T11:26:54-07:00 DBG send completed  
2018-03-30T11:26:54-07:00 DBG Events sent: 3  
2018-03-30T11:26:54-07:00 DBG Processing 3 events  
2018-03-30T11:26:54-07:00 DBG New state added for C:\TestRail\TestRailResults12.json  
2018-03-30T11:26:54-07:00 DBG New state added for C:\TestRail\TestRailResults13.json  
2018-03-30T11:26:54-07:00 DBG Registrar states cleaned up. Before: 8, After: 8  
2018-03-30T11:26:54-07:00 DBG Write registry file: C:\ProgramData\filebeat\registry  
2018-03-30T11:26:54-07:00 DBG Registry file updated. 8 states written.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 6:41pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/10 "2018-03-30T18:41:06Z")

</div>

Lets find out what worked, from the debug log we see that the document has the `fields/document_type`

> [@sunny1](#):
>
> "fields": {
> 
> "document\_type": "testrailresults13"
> 
> },

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 6:43pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/11 "2018-03-30T18:43:53Z")

</div>

I only see a document from `C:\TestRail\TestRailResults13.json` So the other file is not read, What is the content of `TestRailResult12.json`?

---

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 30, 2018, 6:45pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/12 "2018-03-30T18:45:09Z")

</div>

Yes exactly it always reads the second file only. the file is json type but here its not allowed to attach file. So could not attach the input files. The file content is exactly similar to other file but the name field has different value.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 6:47pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/13 "2018-03-30T18:47:27Z")

</div>

> [@sunny1](#):
>
> 2018-03-30T11:26:48-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.

So its either one of the following:

1. The offset is saved in `data/registry` so it doesn't read it again.
2. OR there is no **end of line** (`\n`) at the end of the file so it doesn't know that an event is completely written in the file.

---

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 30, 2018, 6:53pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/14 "2018-03-30T18:53:01Z")

</div>

Sorry was in process to post complete file but it was more than 7000 charater so had to truncate it.

Let me see how i can post the entire file

---

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 30, 2018, 6:55pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/15 "2018-03-30T18:55:37Z")

</div>

Here some of it. I have more in log file but its just repeat of what i already posted.

018-03-30T11:26:55-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.  
2018-03-30T11:26:55-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.  
2018-03-30T11:26:58-07:00 DBG Run prospector  
2018-03-30T11:26:58-07:00 DBG Start next scan  
2018-03-30T11:26:58-07:00 DBG Flushing spooler because of timeout. Events flushed: 0  
2018-03-30T11:26:58-07:00 DBG Run prospector  
2018-03-30T11:26:58-07:00 DBG Start next scan  
2018-03-30T11:26:58-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults13.json  
2018-03-30T11:26:58-07:00 DBG Update existing file for harvesting: C:\TestRail\TestRailResults13.json, offset: 263  
2018-03-30T11:26:58-07:00 DBG Harvester for file is still running: C:\TestRail\TestRailResults13.json  
2018-03-30T11:26:58-07:00 DBG Prospector states cleaned up. Before: 1, After: 1  
2018-03-30T11:26:58-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults12.json  
2018-03-30T11:26:58-07:00 DBG Update existing file for harvesting: C:\TestRail\TestRailResults12.json, offset: 0  
2018-03-30T11:26:58-07:00 DBG Harvester for file is still running: C:\TestRail\TestRailResults12.json  
2018-03-30T11:26:58-07:00 DBG Prospector states cleaned up. Before: 1, After: 1  
2018-03-30T11:27:03-07:00 DBG Flushing spooler because of timeout. Events flushed: 0  
2018-03-30T11:27:03-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.  
2018-03-30T11:27:03-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.  
2018-03-30T11:27:08-07:00 DBG Flushing spooler because of timeout. Events flushed: 0  
2018-03-30T11:27:08-07:00 DBG Run prospector  
2018-03-30T11:27:08-07:00 DBG Start next scan  
2018-03-30T11:27:08-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults13.json  
2018-03-30T11:27:08-07:00 DBG Update existing file for harvesting: C:\TestRail\TestRailResults13.json, offset: 263  
2018-03-30T11:27:08-07:00 DBG Harvester for file is still running: C:\TestRail\TestRailResults13.json  
2018-03-30T11:27:08-07:00 DBG Prospector states cleaned up. Before: 1, After: 1  
2018-03-30T11:27:08-07:00 DBG Run prospector  
2018-03-30T11:27:08-07:00 DBG Start next scan  
2018-03-30T11:27:08-07:00 DBG Check file for harvesting: C:\TestRail\TestRailResults12.json  
2018-03-30T11:27:08-07:00 DBG Update existing file for harvesting: C:\TestRail\TestRailResults12.json, offset: 0  
2018-03-30T11:27:08-07:00 DBG Harvester for file is still running: C:\TestRail\TestRailResults12.json  
2018-03-30T11:27:08-07:00 DBG Prospector states cleaned up. Before: 1, After: 1  
2018-03-30T11:27:13-07:00 DBG Flushing spooler because of timeout. Events flushed: 0  
2018-03-30T11:27:13-07:00 DBG End of file reached: C:\TestRail\TestRailResults13.json; Backoff now.  
2018-03-30T11:27:13-07:00 DBG End of file reached: C:\TestRail\TestRailResults12.json; Backoff now.  
2018-03-30T11:27:18-07:00 DBG Flushing spooler because of timeout. Events flushed: 0  
2018-03-30T11:27:18-07:00 DBG Run prospector  
2018-03-30T11:27:18-07:00 DBG Start next scan  
2018-03-30T11:27:18-07:00 DBG Run prospector  
2018-03-30T11:27:18-07:00 DBG Start next scan

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 6:57pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/16 "2018-03-30T18:57:19Z")

</div>

> [@sunny1](#):
>
> 2018-03-30T11:27:08-07:00 DBG Update existing file for harvesting: C:\TestRail\TestRailResults12.json, offset: 0

Can you add a new line at the end of this file? The log says it stock at the beginning.

---

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 30, 2018, 6:59pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/17 "2018-03-30T18:59:41Z")

</div>

The file 12 has only one line as of now:

`{"Defects":"","Name":"yesitworks","Type":"API/Services","Priority":"Medium","Estimate":"","Milestone":"Night's Watch - Sprint 4","References":"113740","Level":"System","AutomationScenarioName":"","Automated":"No","AutomationId":"","EpicId":"76580"}`

---

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 30, 2018, 7:01pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/18 "2018-03-30T19:01:04Z")

</div>

Do you want me to add another line after this ? Because the file beat is not reading even this line to start with.

The file content for both files are almost same only the name field has different values.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 7:06pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/19 "2018-03-30T19:06:03Z")

</div>

Internally Filebeat detect `LINE` by checking if a line terminates by `\n` character, I understand there is only one line in your document but I think that there is no newline character at the end of that line. So Filebeat thinks there is nothing to read in that document.

Open the file in your editor go at the end of the line, press return and save the file.

Any application that write theses document will have to terminate each line with a `\n`, every logging library does that.

---

<div class="post-metadata">

**Author:** ![sunny1](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@sunny1](https://discuss.elastic.co/u/sunny1)\
**Post date:** [March 30, 2018, 7:09pm UTC](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162/20 "2018-03-30T19:09:11Z")

</div>

Sure doing it now.

[Next page](https://discuss.elastic.co/t/err-failed-to-initialize-elasticsearch-plugin-as-output-missing-condition/126162.md?page=2)