# My "Logstash.conf" gives errors

**URL:** <https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041>\
**Category:** Logstash\
**Created:** [June 12, 2017, 12:55pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041 "2017-06-12T12:55:39Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![MartinNouv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinnouv/32/18995_2.png) [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\
**Post date:** [June 12, 2017, 12:55pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/1 "2017-06-12T12:55:39Z")

</div>

Hello there !

I'm trying to configure an ELK to have a better overview of my CSV files.

But when I launch my conf file, I get some issues I can't solve despite my searches and my tries for solving.

My conf file :

input {  
file {  
path =\> "C:/...\*.csv"  
start\_position =\> beginning  
sincedb\_path =\> "/dev/null"  
}  
}

filter {  
csv {  
columns =\> ["IP Address", "DNS Name", "Scan Period"]  
separator =\> ";"  
}  
}

output {  
stdout { codec =\> rubydebug }  
elasticsearch {  
host =\> "localhost:9200"  
index =\> "csv\_index"  
}  
}

The result next a _logstash -f "\*.csv"_ command line :

[2017-06-12T14:52:17,639][ERROR][logstash.outputs.elasticsearch] Unknown setting  
'host' for elasticsearch  
[2017-06-12T14:52:17,655][ERROR][logstash.agent] Cannot create pipeli  
ne {:reason=\>"Something is wrong with your configuration."}  
[2017-06-12T14:52:17,967][INFO][logstash.outputs.elasticsearch] Elasticsearch p  
ool URLs updated {:changes=\>{:removed=\>[], :added=\>[http://logstash\_system:xxxxx  
x@localhost:9200/\_xpack/monitoring/?system\_id=logstash&system\_api\_version=2&inte  
rval=1s]}}  
[2017-06-12T14:52:17,967][INFO][logstash.outputs.elasticsearch] Running health  
check to see if an Elasticsearch connection is working {:healthcheck\_url=\>http:/  
/logstash\_system:xxxxxx@localhost:9200/, :path=\>"/"}  
[2017-06-12T14:52:18,232][WARN][logstash.outputs.elasticsearch] Restored connec  
tion to ES instance {:url=\>#\<URI::HTTP:0x6cfddc20 URL:http://logstash\_system:xxx  
xxx@localhost:9200/\>}  
[2017-06-12T14:52:18,232][INFO][logstash.outputs.elasticsearch] New Elasticsear  
ch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>[#\<URI::HTTP:0x77  
6b50d8 URL:[http://localhost:9200](http://localhost:9200)\>]}  
[2017-06-12T14:52:18,248][INFO][logstash.outputs.elasticsearch] Elasticsearch p  
ool URLs updated {:changes=\>{:removed=\>[], :added=\>[http://logstash\_system:xxxxx  
x@localhost:9200/\_xpack/monitoring/?system\_id=logstash&system\_api\_version=2&inte  
rval=1s]}}  
[2017-06-12T14:52:18,248][INFO][logstash.outputs.elasticsearch] Running health  
check to see if an Elasticsearch connection is working {:healthcheck\_url=\>http:/  
/logstash\_system:xxxxxx@localhost:9200/, :path=\>"/"}  
[2017-06-12T14:52:18,263][WARN][logstash.outputs.elasticsearch] Restored connec  
tion to ES instance {:url=\>#\<URI::HTTP:0x3ff94539 URL:http://logstash\_system:xxx  
xxx@localhost:9200/\>}  
[2017-06-12T14:52:18,263][INFO][logstash.outputs.elasticsearch] New Elasticsear  
ch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>[#\<URI::HTTP:0x72  
e9016 URL:[http://localhost:9200](http://localhost:9200)\>]}  
[2017-06-12T14:52:18,263][INFO][logstash.pipeline] Starting pipeline {"  
id"=\>".monitoring-logstash", "pipeline.workers"=\>1, "pipeline.batch.size"=\>2, "p  
ipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>2}  
[2017-06-12T14:52:18,279][INFO][logstash.pipeline] Pipeline .monitoring  
-logstash started  
[2017-06-12T14:52:18,513][INFO][logstash.agent] Successfully started  
Logstash API endpoint {:port=\>9600}  
[2017-06-12T14:52:28,285][ERROR][logstash.inputs.metrics] Failed to create mon  
itoring event {:message=\>"For path: events", :error=\>"LogStash::Instrument::Metr  
icStore::MetricNotFound"}

I tried everything, nothing effective. So a little help could be great.

Thanks in advance, and have a nice day !

---

<div class="post-metadata">

**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\
**Post date:** [June 12, 2017, 1:03pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/2 "2017-06-12T13:03:46Z")

</div>

> [@MartinNouv](#):
>
> The result next a logstash -f "\*.csv" command line

The _-f_ flag is for defining your config path and not your input files, like so

```auto
/bin/logstash -f /path/to/config_folder/
```

Try this and post any potential errors that should arise.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 12, 2017, 1:10pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/3 "2017-06-12T13:10:26Z")

</div>

> [@MartinNouv](#):
>
> host =\> "localhost:9200"

As indicated by the first line in the error message, `host` is no longer a valid parameter for the Elasticsearch output. If you look in [the documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts) you will see that is now supposed to be `hosts` instead. make sure you check against the correct version of the documentation if you are following old tutorials.

---

<div class="post-metadata">

**Author:** ![MartinNouv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinnouv/32/18995_2.png) [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\
**Post date:** [June 12, 2017, 1:18pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/4 "2017-06-12T13:18:50Z")

</div>

I fixed what you said. Same error.

[2017-06-12T15:15:58,515][ERROR][logstash.inputs.metrics] Failed to create mon  
itoring event {:message=\>"For path: events", :error=\>"LogStash::Instrument::Metr  
icStore::MetricNotFound"}

---

<div class="post-metadata">

**Author:** ![MartinNouv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinnouv/32/18995_2.png) [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\
**Post date:** [June 12, 2017, 1:19pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/5 "2017-06-12T13:19:52Z")

</div>

Same answer as above.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 12, 2017, 1:27pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/6 "2017-06-12T13:27:57Z")

</div>

Which version of Logstash are you using? Do you have X-Pack installed?

---

<div class="post-metadata">

**Author:** ![MartinNouv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinnouv/32/18995_2.png) [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\
**Post date:** [June 12, 2017, 1:31pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/7 "2017-06-12T13:31:12Z")

</div>

I've got the last version (5.4.0) with X-pack installed.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 12, 2017, 1:34pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/8 "2017-06-12T13:34:36Z")

</div>

Have you added the [required configuration](https://www.elastic.co/guide/en/x-pack/current/monitoring-logstash.html) to the `logstash.yml` file?

---

<div class="post-metadata">

**Author:** ![erion](https://avatars.discourse-cdn.com/v4/letter/e/779978/32.png) [@erion](https://discuss.elastic.co/u/erion)\
**Post date:** [June 12, 2017, 1:40pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/9 "2017-06-12T13:40:15Z")

</div>

try:  
output {  
stdout { codec =\> rubydebug } }

elastisearch {  
hosts =\> ["localhost:9200"]  
...  
}  
try this two modify if send you error change localhost with your IP:9200.

---

<div class="post-metadata">

**Author:** ![MartinNouv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinnouv/32/18995_2.png) [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\
**Post date:** [June 13, 2017, 2:01pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/10 "2017-06-13T14:01:40Z")

</div>

Hi again !

I tried to reconfigure the .yml like Christian asked, but no, nothing changes.

---

<div class="post-metadata">

**Author:** ![MartinNouv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinnouv/32/18995_2.png) [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\
**Post date:** [June 14, 2017, 8:28am UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/11 "2017-06-14T08:28:26Z")

</div>

I' thinking about one thing right now. In the YML of Logstash, should I remove the hashtags to apply the changes I've made with the required configuration ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 14, 2017, 9:05am UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/12 "2017-06-14T09:05:28Z")

</div>

Can you share your logstash.yml file. Please make sure it is formatted as code.

---

<div class="post-metadata">

**Author:** ![MartinNouv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinnouv/32/18995_2.png) [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\
**Post date:** [June 14, 2017, 9:23am UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/13 "2017-06-14T09:23:41Z")

</div>

Sure, there it is.

```
   # Settings file in YAML
    #
    # Settings can be specified either in hierarchical form, e.g.:
    #
    # pipeline:
    # batch:
    # size: 125
    # delay: 5
    #
    # Or as flat keys:
    #
    # pipeline.batch.size: 125
    # pipeline.batch.delay: 5
    #
    # ------------ Node identity ------------
    #
    # Use a descriptive name for the node:
    #
    # node.name: test
    #
    # If omitted the node name will default to the machine's host name
    #
    # ------------ Data path ------------------
    #
    # Which directory should be used by logstash and its plugins
    # for any persistent needs. Defaults to LOGSTASH_HOME/data
    #
    # path.data:
    #
    # ------------ Pipeline Settings --------------
    #
    # Set the number of workers that will, in parallel, execute the filters+outputs
    # stage of the pipeline.
    #
    # This defaults to the number of the host's CPU cores.
    #
    # pipeline.workers: 2
    #
    # How many workers should be used per output plugin instance
    #
    # pipeline.output.workers: 1
    #
    # How many events to retrieve from inputs before sending to filters+workers
    #
    # pipeline.batch.size: 125
    #
    # How long to wait before dispatching an undersized batch to filters+workers
    # Value is in milliseconds.
    #
    # pipeline.batch.delay: 5
    #
    # Force Logstash to exit during shutdown even if there are still inflight
    # events in memory. By default, logstash will refuse to quit until all
    # received events have been pushed to the outputs.
    #
    # WARNING: enabling this can lead to data loss during shutdown
    #
    # pipeline.unsafe_shutdown: false
    #
    # ------------ Pipeline Configuration Settings --------------
    #
    # Where to fetch the pipeline configuration for the main pipeline
    #
    # path.config:
    #
    # Pipeline configuration string for the main pipeline
    #
    # config.string:
    #
    # At startup, test if the configuration is valid and exit (dry run)
    #
    # config.test_and_exit: false
    #
    # Periodically check if the configuration has changed and reload the pipeline
    # This can also be triggered manually through the SIGHUP signal
    #
    # config.reload.automatic: false
    #
    # How often to check if the pipeline configuration has changed (in seconds)
    #
    # config.reload.interval: 3
    #
    # Show fully compiled configuration as debug log message
    # NOTE: --log.level must be 'debug'
    #
    # config.debug: false
    #
    # ------------ Queuing Settings --------------
    #
    # Internal queuing model, "memory" for legacy in-memory based queuing and
    # "persisted" for disk-based acked queueing. Defaults is memory
    #
    # queue.type: memory
    #
    # If using queue.type: persisted, the directory path where the data files will be stored.
    # Default is path.data/queue
    #
    # path.queue:
    #
    # If using queue.type: persisted, the page data files size. The queue data consists of
    # append-only data files separated into pages. Default is 250mb
    #
    # queue.page_capacity: 250mb
    #
    # If using queue.type: persisted, the maximum number of unread events in the queue.
    # Default is 0 (unlimited)
    #
    # queue.max_events: 0
    #
    # If using queue.type: persisted, the total capacity of the queue in number of bytes.
    # If you would like more unacked events to be buffered in Logstash, you can increase the
    # capacity using this setting. Please make sure your disk drive has capacity greater than
    # the size specified here. If both max_bytes and max_events are specified, Logstash will pick
    # whichever criteria is reached first
    # Default is 1024mb or 1gb
    #
    # queue.max_bytes: 1024mb
    #
    # If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint
    # Default is 1024, 0 for unlimited
    #
    # queue.checkpoint.acks: 1024
    #
    # If using queue.type: persisted, the maximum number of written events before forcing a checkpoint
    # Default is 1024, 0 for unlimited
    #
    # queue.checkpoint.writes: 1024
    #
    # If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page
    # Default is 1000, 0 for no periodic checkpoint.
    #
    # queue.checkpoint.interval: 1000
    #
    # ------------ Metrics Settings --------------
    #
    # Bind address for the metrics REST endpoint
    #
    # http.host: "127.0.0.1"
    #
    # Bind port for the metrics REST endpoint, this option also accept a range
    # (9600-9700) and logstash will pick up the first available ports.
    #
    # http.port: 9600-9700
    #
    # ------------ Debugging Settings --------------
    #
    # Options for log.level:
    # * fatal
    # * error
    # * warn
    # * info (default)
    # * debug
    # * trace
    #
    # log.level: info
    # path.logs:
    #
    # ------------ Other Settings --------------
    #
    # Where to find custom plugins
    # path.plugins: []
    #xpack.monitoring.elasticsearch.url: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] 
    #xpack.monitoring.elasticsearch.username: "elastic" 
    #xpack.monitoring.elasticsearch.password: "changeme"
```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 14, 2017, 9:25am UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/14 "2017-06-14T09:25:38Z")

</div>

None of that config will be used as it is all commented out, so you need to remove hashtags for the relevant parts.

---

<div class="post-metadata">

**Author:** ![MartinNouv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinnouv/32/18995_2.png) [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\
**Post date:** [June 14, 2017, 9:33am UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/15 "2017-06-14T09:33:05Z")

</div>

Yeah it's what I thought. I'll fix that and I'll keep you posted.

---

<div class="post-metadata">

**Author:** ![MartinNouv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinnouv/32/18995_2.png) [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\
**Post date:** [June 14, 2017, 1:28pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/16 "2017-06-14T13:28:38Z")

</div>

Ok, there's progress ! I've touched the yml in more details, and no more big errors like before. Therefore,

```
[2017-06-14T15:01:18,111][ERROR][logstash.outputs.elasticsearch] Unknown setting
 'host' for elasticsearch
[2017-06-14T15:01:18,111][FATAL][logstash.runner] The given configurat
ion is invalid. Reason: Something is wrong with your configuration.

```

So I can see it's from the host option from the elastic search function, but i thought it was ok.

The output from my conf file :

```
output {
    stdout { codec => rubydebug }
    elasticsearch {
        host => ["https://127.0.0.1:9200"]
        index => "csv_index"
    }
}
```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 14, 2017, 2:24pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/17 "2017-06-14T14:24:52Z")

</div>

The parameter should be ['hosts'](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts), not 'host'.

---

<div class="post-metadata">

**Author:** ![MartinNouv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinnouv/32/18995_2.png) [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\
**Post date:** [June 14, 2017, 2:54pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/18 "2017-06-14T14:54:02Z")

</div>

```
Configuration OK
[2017-06-14T16:30:23,248][INFO][logstash.runner] Using config.test_an
d_exit mode. Config Validation Result: OK. Exiting Logstash

```

And I don't have anything about Logstash in ElasticSearch's logs.

---

<div class="post-metadata">

**Author:** ![alanr4](https://avatars.discourse-cdn.com/v4/letter/a/45deac/32.png) [@alanr4](https://discuss.elastic.co/u/alanr4)\
**Post date:** [June 14, 2017, 3:12pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/19 "2017-06-14T15:12:15Z")

</div>

After installing xpac encounter this error:

[2017-06-14T15:11:13,598][ERROR][logstash.inputs.file] Unknown setting 'user' for file  
[2017-06-14T15:11:13,601][ERROR][logstash.inputs.file] Unknown setting 'password' for file  
[2017-06-14T15:11:13,608][ERROR][logstash.agent] Cannot create pipeline {:reason=\>"Something is wrong with your configuration."}

Any tips?

---

<div class="post-metadata">

**Author:** ![MartinNouv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinnouv/32/18995_2.png) [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\
**Post date:** [June 16, 2017, 8:25am UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041/20 "2017-06-16T08:25:55Z")

</div>

It's me again.

Next i disabled the "config.test\_and\_exit mode", I fall on a error next i launch the logstash command ( the pipeline is created first fyi)

It's kinda strange because it looks like the pipeline is operationnal, despite this error what appears to be in the plugin.

Any ideas ? And of course, thanks again for your help !

[Next page](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041.md?page=2)