# Error when reading log files using logstash

**URL:** <https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891>\
**Category:** Logstash\
**Created:** [March 5, 2019, 12:00pm UTC](https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891 "2019-03-05T12:00:21Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dheerajbtu](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@Dheerajbtu](https://discuss.elastic.co/u/Dheerajbtu)\
**Post date:** [March 5, 2019, 12:00pm UTC](https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891/1 "2019-03-05T12:00:21Z")

</div>

I have a JSON file with below information.  
{"apiId":210966762,"apiVersionId":15552430,"orgId":"eeb3ccb6-a2f4-4c7b-9459-7202183bce03","hostId":"mule.qa.2","receivedTs":"2019-01-28T23:59:54.691-05:00","repliedTs":"2019-01-28T23:59:56.406-05:00"}

I need to add this information to kibana.

I am trying to do it using logstash with below config file. (fileName: logstash.conf)  
input{  
file {  
port =\> 5044  
path =\> "e:\logs\*"  
type =\> "json"  
start\_position =\> "beginning"  
codec =\> "json"  
}  
}

output{  
stdout{  
codec=\> rubydebug  
}  
elasticsearch{  
hosts =\> "localhost:9200"  
index =\> "mulesoft-log-mon-%{+YYYY.MM.dd}"  
}  
}

But I am getting below error when I run logstash.  
PS E:\softwares\ElasticLogstash6.2.3\> .\bin\logstash.bat -f logstash.conf  
Sending Logstash's logs to E:/softwares/ElasticLogstash6.2.3/logs which is now configured via log4j2.properties  
[2019-03-05T16:54:09,394][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"E:/softwares/ElasticLogstash6.2.3/modules/fb\_apache/configuration"}  
[2019-03-05T16:54:09,440][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"E:/softwares/ElasticLogstash6.2.3/modules/netflow/configuration"}  
[2019-03-05T16:54:10,167][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2019-03-05T16:54:11,461][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.2.3"}  
[2019-03-05T16:54:12,840][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
[2019-03-05T16:54:16,388][ERROR][logstash.inputs.file] Unknown setting 'port' for file  
[2019-03-05T16:54:16,466][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Something is wrong with your configuration.", :backtrace=\>["E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/config/mixin.rb:89:in `config_init'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/inputs/base.rb:62:in`initialize'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/plugins/plugin\_factory.rb:89:in `plugin'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/pipeline.rb:112:in`plugin'", "(eval):8:in `<eval>'", "org/jruby/RubyKernel.java:994:in`eval'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/pipeline.rb:84:in `initialize'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/pipeline.rb:169:in`initialize'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/pipeline\_action/create.rb:40:in `execute'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/agent.rb:315:in`block in converge\_state'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/agent.rb:312:in`block in converge\_state'", "org/jruby/RubyArray.java:1734:in `each'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/agent.rb:299:in`converge\_state'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/agent.rb:141:in`with\_pipelines'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/agent.rb:90:in`execute'", "E:/softwares/ElasticLogstash6.2.3/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "E:/softwares/ElasticLogstash6.2.3/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in`block in initialize'"]}  
PS E:\softwares\ElasticLogstash6.2.3\>

Am I missing something..?  
Do we need grok filter to read json content..?

I am doing it on windows machine to setup things on my local env.

Appreciate your help/support.

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [March 5, 2019, 12:02pm UTC](https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891/2 "2019-03-05T12:02:34Z")

</div>

> [@Dheerajbtu](#):
>
> file {  
> port =\> 5044  
> path =\> "e:\logs\*"  
> type =\> "json"  
> start\_position =\> "beginning"  
> codec =\> "json"  
> }

You are sending log info from your local path no need to parse port in input. remove that and try

---

<div class="post-metadata">

**Author:** ![Dheerajbtu](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@Dheerajbtu](https://discuss.elastic.co/u/Dheerajbtu)\
**Post date:** [March 5, 2019, 12:25pm UTC](https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891/3 "2019-03-05T12:25:41Z")

</div>

Hi Ganesh,

I have removed port, following is my conf file  
input{  
file {  
path =\> e:\logs\*.log  
type =\> "json"  
start\_position =\> "beginning"  
codec =\> "json"  
}  
}

output{  
stdout{  
codec=\> rubydebug  
}  
elasticsearch{  
hosts =\> "localhost:9200"  
index =\> "mulesoft-log-mon-%{+YYYY.MM.dd}"  
}  
}

Now we are not getting the error but,  
I dont see logs getting populated/processing.

Following is the console info. It is stuck at this point without any error in idle state.  
PS E:\softwares\ElasticLogstash6.2.3\> .\bin\logstash.bat -f logstash.conf  
Sending Logstash's logs to E:/softwares/ElasticLogstash6.2.3/logs which is now configured via log4j2.properties  
[2019-03-05T17:40:21,885][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"E:/softwares/ElasticLogstash6.2.3/modules/fb\_apache/configuration"}  
[2019-03-05T17:40:21,916][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"E:/softwares/ElasticLogstash6.2.3/modules/netflow/configuration"}  
[2019-03-05T17:40:22,213][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2019-03-05T17:40:23,244][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.2.3"}  
[2019-03-05T17:40:24,097][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
[2019-03-05T17:40:30,142][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}  
[2019-03-05T17:40:30,913][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://localhost:9200/](http://localhost:9200/)]}}  
[2019-03-05T17:40:30,928][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://localhost:9200/](http://localhost:9200/), :path=\>"/"}  
[2019-03-05T17:40:31,219][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"}  
[2019-03-05T17:40:31,344][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>6}  
[2019-03-05T17:40:31,360][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6}  
[2019-03-05T17:40:31,376][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil}  
[2019-03-05T17:40:31,422][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
[2019-03-05T17:40:31,500][INFO][logstash.outputs.elasticsearch] Installing elasticsearch template to \_template/logstash  
[2019-03-05T17:40:33,521][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]}  
[2019-03-05T17:40:34,637][INFO][logstash.pipeline] Pipeline started succesfully {:pipeline\_id=\>"main", :thread=\>"#\<Thread:0x6a7e0b00 run\>"}  
[2019-03-05T17:40:34,795][INFO][logstash.agent] Pipelines running {:count=\>1, :pipelines=\>["main"]}

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [March 5, 2019, 12:27pm UTC](https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891/4 "2019-03-05T12:27:15Z")

</div>

have you check ES index whether index get created or not?

---

<div class="post-metadata">

**Author:** ![Dheerajbtu](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@Dheerajbtu](https://discuss.elastic.co/u/Dheerajbtu)\
**Post date:** [March 5, 2019, 12:49pm UTC](https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891/5 "2019-03-05T12:49:15Z")

</div>

Index is not created

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 5, 2019, 12:52pm UTC](https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891/6 "2019-03-05T12:52:21Z")

</div>

> [@Dheerajbtu](#):
>
> path =\> e:\logs\*.log

Use forward slash instead of backslash

---

<div class="post-metadata">

**Author:** ![Dheerajbtu](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@Dheerajbtu](https://discuss.elastic.co/u/Dheerajbtu)\
**Post date:** [March 5, 2019, 1:06pm UTC](https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891/7 "2019-03-05T13:06:56Z")

</div>

Hi Ganesh & Badger,

Thanks a ton for your response. Index got created and got data into KIBANA.

Thanks a lot for your time 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 2, 2019, 1:07pm UTC](https://discuss.elastic.co/t/error-when-reading-log-files-using-logstash/170891/8 "2019-04-02T13:07:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.