Logstash Configuration - filepath from remote machine

Elastic Stack Logstash
1

Hi,

My logstash configuration file contents is shown below:

input {
file {
path => ["D:/Sample/*"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "Testing-%{+YYYY.MM.dd}"
}
stdout {}
}

I have some log files in this path "D:\Sample\Logs*.log" which is in remote server \10...*.
And it needs credentials like some username and password.

So I want to upload those files from my machine.
Can anyone please tell me how should I configure input plugin, so that logstash should take logs from remote server and upload it into kibana.

2

I think installing filebeat on remote server and send logs to logstash is better solution for your case.

3

Hi @arkady_renko,

Can you please tell me how can i achieve that. I have installed filebeat on remote machine.
How to send logs from filebeat to logstash?

4

Hi @Sunillinus,

Please follow the given steps to transfer logs to elastic search:-

  1. Refer to the link:- Logstash(5.4.3) fails to transfer logs even when logstash starts as a service
  2. Use the below logstash.conf file for sending logs from logstash to elasticsearch
    input {
    file {
    path => [ "/var/log/.log", "/var/log/messages", "/var/log/syslog","/var/log/.log" ] #mention path of any file
    tags=> [""] #mention the tag here e.g. tags=> ["sunil"]
    }
    }

output {
elasticsearch {
hosts => "10.0.x.x" #mention the elasticsearch ip address here
manage_template => false
index => "Testing-%{+YYYY.MM.dd}"
}
}
3) Save the file and exit.
4) Start the logstash

Note: Here, logs are transferred to elasticsearch without using filebeat. As, logs can be transferred to elasticsearch by simply using logstash only.

@Sunillinus let me know if you face any issues

Thanks
Nikhil Kapoor

5

Hi @nikhil.k

Whatever you suggested is working for me. But my question is how to get the logs which are available in remote machine (some different server).

How to get filepath which is available in different machine?

6

Set logstash as output for filebeat installed on remove server.
https://www.elastic.co/guide/en/beats/filebeat/current/config-filebeat-logstash.html

7

Hi @Sunillinus,

In this case you need to install filebeat on the remote machine.

Steps to install filebeat:-

  1. curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-x86_64.rpm #please change the version if needed, as i was not knowing your logstash version
  2. sudo yum install filebeat-6.2.4-x86_64.rpm
  3. Make changes in the vim /etc/filebeat/filebeat.yml file as follows:-

a)

  • input_type: log

    # Paths that should be crawled and fetched. Glob based paths.
    paths:

    • /var/log/xyz.log #path of the logs that you want to transfer from remote machine to logstash

b)
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["10.0.x.x:5044"] # ip address of the logstash

  1. Save and exit the file
  2. firewall-cmd --permanent --zone=public --add-port=5044/tcp
  3. systemctl enable filebeat.service
  4. systemctl start filebeat.service
  5. systemctl status filebeat.service filebeat should have been started

Now you need to configure the logstash in order to recieve the logs from filebeat

  1. Open logstash.conf and enter the below given configuration
    input{
    beats {
    port => 5044
    }
    }

    output {
    file {
    path => "/filepath"
    }
    }

  2. Save and exit the file

  3. firewall-cmd --permanent --zone=public --add-port=5044/tcp

  4. Start logstash

Here, "xyz.log" from filebeat(remote server) will be transferred to logstash in a file(of given path in logstash.conf)

@Sunillinus let me know if you face any issue.

Regards
Nikhil Kapoor

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.