# Logstash skips file while reading config
**URL:** https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770
**Category:** Logstash
**Created:** [October 30, 2019, 12:22am UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770 "2019-10-30T00:22:35Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [October 30, 2019, 12:22am UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770/1 "2019-10-30T00:22:36Z")
Logstash just starts and shuts down due to configpathloader error. Please explain what the issue is. Below is the error -
[DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=\>["/home/mehak/Documents/logstash-7.4.0/CONTRIBUTORS", "/home/mehak/Documents/logstash-7.4.0/Gemfile", "/home/mehak/Documents/logstash-7.4.0/Gemfile.lock", "/home/mehak/Documents/logstash-7.4.0/LICENSE.txt", "/home/mehak/Documents/logstash-7.4.0/NOTICE.TXT", "/home/mehak/Documents/logstash-7.4.0/bin", "/home/mehak/Documents/logstash-7.4.0/config", "/home/mehak/Documents/logstash-7.4.0/data", "/home/mehak/Documents/logstash-7.4.0/lib", "/home/mehak/Documents/logstash-7.4.0/logs", "/home/mehak/Documents/logstash-7.4.0/logstash-core", "/home/mehak/Documents/logstash-7.4.0/logstash-core-plugin-api", "/home/mehak/Documents/logstash-7.4.0/modules", "/home/mehak/Documents/logstash-7.4.0/tools", "/home/mehak/Documents/logstash-7.4.0/vendor", "/home/mehak/Documents/logstash-7.4.0/x-pack"]}
[2019-10-29T13:14:14,614][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config\_file=\>"/home/mehak/Documents/logstash-7.4.0/logstash.conf"}
[2019-10-29T13:14:14,791][DEBUG][logstash.agent] Converging pipelines state {:actions\_count=\>1}
[2019-10-29T13:14:14,817][DEBUG][logstash.agent] Executing action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main}
[2019-10-29T13:14:15,125][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, input, filter, output at line 2, column 1 (byte 2) after \n", :backtrace=\>["/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:49:in `compile\_graph'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/agent.rb:326:in `block in converge\_state'"]}
Below is the logstash.config file-
filebeat.inputs:
- type: log
paths:
- /var/log/system.log
- /var/log/wifi.log
- type: log
paths:
- "/var/log/apache2/\*"
fields:
apache: true
fields\_under\_root: true
output.elasticsearch:
hosts: ["[https://localhost:9200](https://localhost:9200/)"]
index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
ssl.certificate\_authorities: ["/etc/pki/root/ca.pem"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key"
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [October 30, 2019, 8:35am UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770/2 "2019-10-30T08:35:28Z")
I moved your post to Logstash, as it is a Logstash related question.
---
### Author:  [@st3inbeiss](https://discuss.elastic.co/u/st3inbeiss)
#### Post date: [October 30, 2019, 9:36am UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770/3 "2019-10-30T09:36:14Z")
If I get this right, you try to use a beats config in logstash. Logstash uses another format for its config files. Something along the lines:
```
input{
}
filter{
}
output{
}
```
See here: [https://www.elastic.co/guide/en/logstash/current/configuration.html](https://www.elastic.co/guide/en/logstash/current/configuration.html)
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [October 30, 2019, 5:51pm UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770/4 "2019-10-30T17:51:56Z")
Hi @st3inbeiss, I fixed the logstahs.conf file and it looks like this now-
# Read input from filebeat by listening to port 5044 on which filebeat will send the data
input {
beats {
type =\> "test"
port =\> "5044"
}
}
filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
if [message] =~ "\tat" {
grok {
match =\> ["message", "^(\tat)"]
add\_tag =\> ["stacktrace"]
}
}
}
output {
stdout {
codec =\> rubydebug
}
# Sending properly parsed log events to elasticsearch
elasticsearch {
hosts =\> ["localhost:9200"]
}
}
But i still have the skipping files error
2019-10-30T10:44:43,602][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-10-30T10:44:43,609][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.4.0"}
[2019-10-30T10:44:43,687][DEBUG][logstash.agent] Setting up metric collection
[2019-10-30T10:44:43,812][DEBUG][logstash.instrument.periodicpoller.os] Starting {:polling\_interval=\>5, :polling\_timeout=\>120}
[2019-10-30T10:44:44,273][DEBUG][logstash.instrument.periodicpoller.jvm] Starting {:polling\_interval=\>5, :polling\_timeout=\>120}
[2019-10-30T10:44:44,804][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=\>"ParNew"}
[2019-10-30T10:44:44,817][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=\>"ConcurrentMarkSweep"}
[2019-10-30T10:44:44,861][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Starting {:polling\_interval=\>5, :polling\_timeout=\>120}
[2019-10-30T10:44:44,887][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Starting {:polling\_interval=\>5, :polling\_timeout=\>120}
[2019-10-30T10:44:45,014][DEBUG][logstash.agent] Starting agent
[2019-10-30T10:44:45,298][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=\>
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 30, 2019, 6:17pm UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770/5 "2019-10-30T18:17:23Z")
> [@Mehak\_Bhargava](#):
>
> [DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern
This is a non-issue. It is a debug message that is routinely logged.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [October 30, 2019, 6:23pm UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770/6 "2019-10-30T18:23:02Z")
This debug message is followed by an error-
[2019-10-30T11:17:48,727][ERROR][logstash.config.sourceloader] No configuration found in the configured sources
This is my pipelines.yml file content and I created two files called first-pipeline.conf and second-pipeline.conf in logstash folder with just input{} filter{} output{} to pass something in.
- [pipeline.id](http://pipeline.id/): test
pipeline.workers: 1
path.config: "/tmp/logstash/first-pipeline.config
pipeline.batch.size: 1
- [pipeline.id](http://pipeline.id/): another\_test
queue.type: persisted
path.config: "/tmp/logstash/second-pipeline.config"
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [October 30, 2019, 6:41pm UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770/7 "2019-10-30T18:41:05Z")
Made progress by creating a pipeline.conf file in logstash and running the command bin/logstash -f pieline.conf
After the DEBUG- Skipping the following files statement, I dont get any errors but still logstash runs for seconds only. What is the issue here? Where would I give location of the logs I want to be passed?
2019-10-30T11:35:15,339][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config\_file=\>"/home/mehak/Documents/logstash-7.4.0/pipeline.conf"}
[2019-10-30T11:35:15,423][DEBUG][logstash.agent] Converging pipelines state {:actions\_count=\>1}
[2019-10-30T11:35:15,466][DEBUG][logstash.agent] Executing action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main}
[2019-10-30T11:35:15,799][DEBUG][org.logstash.secret.store.SecretStoreFactory] Attempting to exists or secret store with implementation: org.logstash.secret.store.backend.JavaKeyStore
[2019-10-30T11:35:16,142][DEBUG][logstash.javapipeline] Starting pipeline {:pipeline\_id=\>"main"}
[2019-10-30T11:35:16,191][WARN][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster\_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2019-10-30T11:35:16,194][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>500, :thread=\>"#\"}
[2019-10-30T11:35:16,240][INFO][logstash.javapipeline][main] Pipeline started {"[pipeline.id](http://pipeline.id/)"=\>"main"}
[2019-10-30T11:35:16,268][DEBUG][logstash.javapipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\"}
[2019-10-30T11:35:16,345][DEBUG][logstash.javapipeline][main] Input plugins stopped! Will shutdown filter/output workers. {:pipeline\_id=\>"main", :thread=\>"#\"}
[2019-10-30T11:35:16,346][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2019-10-30T11:35:16,385][DEBUG][logstash.javapipeline][main] Shutdown waiting for worker thread {:pipeline\_id=\>"main", :thread=\>"#\"}
[2019-10-30T11:35:16,425][DEBUG][logstash.javapipeline][main] Shutdown waiting for worker thread {:pipeline\_id=\>"main", :thread=\>"#\"}
[2019-10-30T11:35:16,432][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>}
[2019-10-30T11:35:16,438][DEBUG][logstash.javapipeline][main] Shutdown waiting for worker thread {:pipeline\_id=\>"main", :thread=\>"#\"}
[2019-10-30T11:35:16,438][DEBUG][logstash.javapipeline][main] Shutdown waiting for worker thread {:pipeline\_id=\>"main", :thread=\>"#\"}
[2019-10-30T11:35:16,448][DEBUG][logstash.javapipeline][main] Pipeline has been shutdown {:pipeline\_id=\>"main", :thread=\>"#\"}
[2019-10-30T11:35:16,491][DEBUG][logstash.agent] Starting puma
[2019-10-30T11:35:16,510][DEBUG][logstash.agent] Trying to start WebServer {:port=\>9600}
[2019-10-30T11:35:16,522][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2019-10-30T11:35:16,555][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2019-10-30T11:35:16,558][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2019-10-30T11:35:16,560][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2019-10-30T11:35:16,578][DEBUG][logstash.agent] Shutting down all pipelines {:pipelines\_count=\>0}
[2019-10-30T11:35:16,579][DEBUG][logstash.agent] Converging pipelines state {:actions\_count=\>0}
[2019-10-30T11:35:16,586][DEBUG][logstash.api.service] [api-service] start
[2019-10-30T11:35:16,808][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}
[2019-10-30T11:35:21,814][INFO][logstash.runner] Logstash shut down
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [October 30, 2019, 6:41pm UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770/8 "2019-10-30T18:41:31Z")
This helped, thanks!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 30, 2019, 7:58pm UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770/9 "2019-10-30T19:58:17Z")
What does your pipeline configuration look like?
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [October 30, 2019, 8:28pm UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770/10 "2019-10-30T20:28:08Z")
Logstash is creating index in elasticsearch but no data is sent. So my question is- how to check pipeline is created and data is being sent from filebeats to logstash port?
Index details:
yellow open logstash-2019.10.30-000001 WnnV21usSDqM18yvaVowZQ 1 1 0 0 283b 283b
No data document in index:
{
"count" : 0,
"\_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
}
}
Here is pipeline.conf file:
// Read input from filebeat by listening to port 5044 on which filebeat will send the data
input {
beats {
type =\> "another\_test"
port =\> "5044"
}
}
filter {
//If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
// if [message] =~ "\tat" {
// grok {
// match =\> ["message", "^(\tat)"]
// add\_tag =\> ["stacktrace"]
// }
//}
}
output {
//stdout {
// codec =\> rubydebug
//}
/ /Sending properly parsed log events to elasticsearch
elasticsearch {
hosts =\> ["localhost:9200"]
}
}
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 27, 2019, 8:28pm UTC](https://discuss.elastic.co/t/logstash-skips-file-while-reading-config/205770/11 "2019-11-27T20:28:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.