# Sending files to logstash with Ruby **URL:** \ **Category:** Logstash\ **Created:** [January 17, 2017, 4:45pm UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882 "2017-01-17T16:45:01Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![sjc07](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@sjc07](https://discuss.elastic.co/u/sjc07)\ **Post date:** [January 17, 2017, 4:45pm UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/1 "2017-01-17T16:45:01Z")
I am working on some code to send a number of files from a series of folders to logstash to be filtered and then added to elasticsearch. I have some Ruby code that identifies the log file type and I then want to send it to logstash with the correct filter and let it handle the file. I am struggling to figure out the best way to do this. Should I use the logstash input filter? Or perhaps filebeat? I do not see any documentation that would allow me to send files from a Ruby script to logstash. Any suggestions of examples would be very welcome. Thanks a lot. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 17, 2017, 7:02pm UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/2 "2017-01-17T19:02:04Z")
> Should I use the logstash input filter? Logstash has inputs, outputs, and filters. Input filters do not exist. > Or perhaps filebeat? That's often a good option. > I do not see any documentation that would allow me to send files from a Ruby script to logstash. There are several Logstash input plugins that, so to speak, are Ruby-compatible. You could e.g. ship the logs via TCP or UDP. ---
**Author:** ![sjc07](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@sjc07](https://discuss.elastic.co/u/sjc07)\ **Post date:** [January 17, 2017, 8:39pm UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/3 "2017-01-17T20:39:19Z")
Apologies I meant file input plugins. Do you have any suggestion of the best approach for this and an example please? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 17, 2017, 8:47pm UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/4 "2017-01-17T20:47:24Z")
If you're looking for a way to ship log files, why not use Filebeat? Why do you need to detect the file type with a Ruby script? ---
**Author:** ![sjc07](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@sjc07](https://discuss.elastic.co/u/sjc07)\ **Post date:** [January 17, 2017, 8:51pm UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/5 "2017-01-17T20:51:58Z")
I use Ruby to do some additional work as well so the file will either go to logstash or somewhere else in theory. I have the following code to send data to logstash. Would filebeat work the same? host = 'localhost' port = 5400 ``` size = 1024 * 1024 * 10 TCPSocket.open(host, port) do |socket| File.open(curPath, 'rb') do |file| while chunk = file.read(size) socket.write(chunk) end end end ``` Many thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 18, 2017, 6:34am UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/6 "2017-01-18T06:34:54Z")
> Would filebeat work the same? Filebeat reads lines from files and ships them somewhere, e.g. to Logstash. Or what are you asking? ---
**Author:** ![sjc07](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@sjc07](https://discuss.elastic.co/u/sjc07)\ **Post date:** [January 18, 2017, 7:36am UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/7 "2017-01-18T07:36:00Z")
Thanks. I have a socket in Ruby. Ruby is doing some things, one of which is to establish the different log formats. The next steps are to send this to logstash, however different formats of logs require different conf files and filters. My challenge is how to manage this. I understand in the conf I can run if else statements based on a path for example but because I am sending raw file data via TCP I cannot do that. I guess what I need is a way to send a message with the log file data and say 'IIS, DATABLOCK' and then 'IIS' message is sent to the conf file and used in the filter. Does that make sense? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 18, 2017, 7:39am UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/8 "2017-01-18T07:39:15Z")
Just send the raw message wrapped in JSON and use the json codec on the receiving end. Then you can supply any tags or fields to Logstash in addition to the message itself. But I still don't understand why you can't do the log detection in Logstash. That would make things so much easier. ---
**Author:** ![sjc07](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@sjc07](https://discuss.elastic.co/u/sjc07)\ **Post date:** [January 18, 2017, 8:16am UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/9 "2017-01-18T08:16:19Z")
Right now I have the different formats going on different ports and a series of statements in the config file. Perhaps a bad way of doing it. In response to your questions - I am very new to this so do not know how I would wrap the message in JSON. Nor do I know how to use logstash for format detection. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 18, 2017, 8:21am UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/10 "2017-01-18T08:21:01Z")
> In response to your questions - I am very new to this so do not know how I would wrap the message in JSON. Instead of sending ``` line from the logfile ``` send ``` {"message": "line from the logfile", "some field": "some value"} ``` and use the extra field(s) in your Logstash configuration. > Nor do I know how to use logstash for format detection. And without more details it's impossible for us to help out. ---
**Author:** ![sjc07](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@sjc07](https://discuss.elastic.co/u/sjc07)\ **Post date:** [January 18, 2017, 9:17am UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/11 "2017-01-18T09:17:13Z")
Thanks for the reply. I am a little lost. I have summarised what I am trying to do, can you tell me what else you need please? Also, the examples you provide. What are you suggesting please? I don't follow. Thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 18, 2017, 9:50am UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/12 "2017-01-18T09:50:46Z")
I'm sorry, but I don't time to explain this at the level you seem to need to understand. Maybe someone else has time to get into details. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 15, 2017, 9:50am UTC](https://discuss.elastic.co/t/sending-files-to-logstash-with-ruby/71882/13 "2017-02-15T09:50:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.