Sending files to logstash with Ruby

Elastic Stack Logstash
1

I am working on some code to send a number of files from a series of folders to logstash to be filtered and then added to elasticsearch. I have some Ruby code that identifies the log file type and I then want to send it to logstash with the correct filter and let it handle the file. I am struggling to figure out the best way to do this.

Should I use the logstash input filter? Or perhaps filebeat? I do not see any documentation that would allow me to send files from a Ruby script to logstash. Any suggestions of examples would be very welcome.

Thanks a lot.

2

Should I use the logstash input filter?

Logstash has inputs, outputs, and filters. Input filters do not exist.

Or perhaps filebeat?

That's often a good option.

I do not see any documentation that would allow me to send files from a Ruby script to logstash.

There are several Logstash input plugins that, so to speak, are Ruby-compatible. You could e.g. ship the logs via TCP or UDP.

3

Apologies I meant file input plugins.

Do you have any suggestion of the best approach for this and an example please?

4

If you're looking for a way to ship log files, why not use Filebeat? Why do you need to detect the file type with a Ruby script?

5

I use Ruby to do some additional work as well so the file will either go to logstash or somewhere else in theory. I have the following code to send data to logstash. Would filebeat work the same?

host = 'localhost'
port = 5400

      size = 1024 * 1024 * 10

      TCPSocket.open(host, port) do |socket|
        File.open(curPath, 'rb') do |file|
          while chunk = file.read(size)
            socket.write(chunk)
          end
        end
      end

Many thanks

6

Would filebeat work the same?

Filebeat reads lines from files and ships them somewhere, e.g. to Logstash. Or what are you asking?

7

Thanks. I have a socket in Ruby. Ruby is doing some things, one of which is to establish the different log formats. The next steps are to send this to logstash, however different formats of logs require different conf files and filters.

My challenge is how to manage this. I understand in the conf I can run if else statements based on a path for example but because I am sending raw file data via TCP I cannot do that. I guess what I need is a way to send a message with the log file data and say 'IIS, DATABLOCK' and then 'IIS' message is sent to the conf file and used in the filter.

Does that make sense?

8

Just send the raw message wrapped in JSON and use the json codec on the receiving end. Then you can supply any tags or fields to Logstash in addition to the message itself.

But I still don't understand why you can't do the log detection in Logstash. That would make things so much easier.

9

Right now I have the different formats going on different ports and a series of statements in the config file. Perhaps a bad way of doing it.

In response to your questions - I am very new to this so do not know how I would wrap the message in JSON. Nor do I know how to use logstash for format detection.

10

In response to your questions - I am very new to this so do not know how I would wrap the message in JSON.

Instead of sending

line from the logfile

send

{"message": "line from the logfile", "some field": "some value"}

and use the extra field(s) in your Logstash configuration.

Nor do I know how to use logstash for format detection.

And without more details it's impossible for us to help out.

11

Thanks for the reply.

I am a little lost. I have summarised what I am trying to do, can you tell me what else you need please?

Also, the examples you provide. What are you suggesting please? I don't follow.

Thanks

12

I'm sorry, but I don't time to explain this at the level you seem to need to understand. Maybe someone else has time to get into details.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.