# Data sending to ES without using Filebeat

**URL:** <https://discuss.elastic.co/t/data-sending-to-es-without-using-filebeat/147361>\
**Category:** Logstash\
**Created:** [September 5, 2018, 9:50am UTC](https://discuss.elastic.co/t/data-sending-to-es-without-using-filebeat/147361 "2018-09-05T09:50:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [September 5, 2018, 9:50am UTC](https://discuss.elastic.co/t/data-sending-to-es-without-using-filebeat/147361/1 "2018-09-05T09:50:55Z")

</div>

HI,  
I want to send data from logstash to ES without filebeat. I use docker command to start the logstash service and log is JSON. pLease find the below code and tell me where im commiting mistake,  
input{  
file{  
path =\> "/root/logstash/logs/\*.log"  
#type =\> "json"  
}  
}  
filter {  
grok {  
match =\> ["message", "%{GREEDYDATA:msg}"]  
}  
json {  
source =\> "message"  
}  
}  
output {  
elasticsearch {  
hosts =\> ["xxxxxx:9200"]  
index =\> "report-sample-%{+YYYY.MM.dd}"  
workers =\> 1  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 5, 2018, 9:58am UTC](https://discuss.elastic.co/t/data-sending-to-es-without-using-filebeat/147361/2 "2018-09-05T09:58:07Z")

</div>

There's nothing obviously wrong with your configuration, but:

- Your grok filter is rather useless.
- You have configured the file input to tail the input files. If you want the files to be read from the top you'll have to make some adjustments. See the file input documentation and the countless previous threads on this topic.
- Are the log files actually available in /root/logstash/logs _inside the container_? Does the user that Logstash runs as have access to those files?

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [September 5, 2018, 10:10am UTC](https://discuss.elastic.co/t/data-sending-to-es-without-using-filebeat/147361/3 "2018-09-05T10:10:39Z")

</div>

- Your grok filter is rather useless.  
**Removed this one**
- You have configured the file input to tail the input files. If you want the files to be read from the top you'll have to make some adjustments. See the file input documentation and the countless previous threads on this topic.  
**added starting\_postition =\> "beginning"**
- Are the log files actually available in /root/logstash/logs _inside the container_ ? Does the user that Logstash runs as have access to those files?  
**yes i have provide that chmod 775**

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 5, 2018, 10:28am UTC](https://discuss.elastic.co/t/data-sending-to-es-without-using-filebeat/147361/4 "2018-09-05T10:28:54Z")

</div>

> added starting\_postition =\> "beginning"

Okay, but make sure you clear the current sincedb state (if any). `start_position` only matters the first time Logstash discovers a file.

> yes i have provide that chmod 775

If you increase Logstash's loglevel it'll tell you what the filename patterns expand to. That's a simple way of checking if the problem is that Logstash can't find or read the input files.

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [September 5, 2018, 12:27pm UTC](https://discuss.elastic.co/t/data-sending-to-es-without-using-filebeat/147361/5 "2018-09-05T12:27:19Z")

</div>

how to increase the logstash's loglevel?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 5, 2018, 12:36pm UTC](https://discuss.elastic.co/t/data-sending-to-es-without-using-filebeat/147361/6 "2018-09-05T12:36:16Z")

</div>

There's a command line options for that, or you can set it via logstash.yml. Both methods are described in the documentation.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 3, 2018, 12:36pm UTC](https://discuss.elastic.co/t/data-sending-to-es-without-using-filebeat/147361/7 "2018-10-03T12:36:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.