As most friends in our forum, I am also a newbie about our ELK stack. Here I get the Logstash 2.4.0 environment, then I receive the request: we want to collect contents of the file /home/yhh/VMState.3 and empty the file periodically.(This file is periodically refreshed by shell scripts in the server). After discussion, I decided to the plugin logstash-input-exec and logstash-output-tcp. Then I wrote the logstash-2.4.0/etc/test.conf as following:
input {
exec {
type => "vmstate"
command => "cat /home/yhh/VMState.3 ;> /home/yhh/VMState.3"
interval => 60
}
}
output {
if [type] == "vmstate" {
tcp {
host => "172.28.217.113"
port => 10988
codec => json
reconnect_interval => 30
}
}
}
I ran the command "/usr/local/src/logstash-2.4.0/bin/logstash -f /usr/local/src/logstash-2.4.0/etc/test.conf" to load the conf file, soon the TCP connection established. But after a while, the connection colsed as if there was something wrong.
netstat -antup |grep 10988
tcp 1 0 ::ffff:172.28.217.192:49828 ::ffff:172.28.217.113:10988 CLOSE_WAIT 22669/java
Below the command "/usr/local/src/logstash-2.4.0/bin/logstash -f /usr/local/src/logstash-2.4.0/etc/test.conf", there were some output warnings of which explanations seemed to be Ruby relevant. I was not familiar with Ruby, so it was difficult for me to read and understand, any one can give me a hand? Thanks very much
BTW, if I want to use trigger mode instead of periodic mode, which means that in the plugin logstash-input-exec only supports to execute the command periodically with option "interval => 60". In fact, the contents of file /home/yhh/VMState.3 are dynamically changing but non-periodically, sometimes some events may trigger the change of contents. Are there any plugins or options to catch that moment and trigger the operation to collect the changed contents and send to the remote endpoint though TCP or other ways?
This is my first topic, hope for your understanding!!
Please format your log entry as preformatted text using the </> toolbar button. Right now the interesting part of the log is being hidden since it's interpreted as HTML.
Side note: If you're starting out and presumably have no legacy to worry about, why start with Logstash 2.4 instead of 5.2?
You mean that my input text contents such as "Hello World" should be transformed to the format "" before send them out by plugins logstash-output-tcp, Is that right? I will try this.
The reason why I use Logstash 2.4 instead of 5.2 is that when I deployed ELK on our customer's site, version 2.4 is the newest one, ELK Stack did not change itself name to Elastic Stack that time. And what we can do now is just deploying in the environment based on 2.4 while customers do not allow us to change the current structure.
BTW, I met some problems in plugin logstash-output-tcp. When I used logstash-input-exec and logstash-output-tcp as above conf in my current situation. I found that once the contents (Length of the contents each time is not fixed size.) that need to be sent by TCP protocol exceeds the length of one TCP message, it will be divided into two messages. But on the remote receiver side, after they received these two TCP messages, they do not know the total length of the contents, so they are not able to merge these two messages to collect the whole contents. Is it possible for us to call some plugin or function in Logstash side to add one flag in the TCP messages which is used to point out the length of contents?
You mean that my input text contents such as "Hello World" should be transformed to the format "" before send them out by plugins logstash-output-tcp, Is that right?
Sorry for my misunderstanding, you mean I should preformat the log on our website, right?
My environment is that we want to send the contents to the remote server through plugin logstash-output-tcp. The remote server is just a customized app server without logstash's deployment on it. So I think in fact it is not logstash-to-logstash communication. The remote server is just able to receive the TCP messages on its specified port 10988. The remote server hopes to receive the TCP messages with the tag that specifies the length of contents. So that it is able to merge several messages together to restore the whole contents when length of the contents exceeds the length of one TCP message.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
