I currently have a logstash server, Elasticsearch server, and kibana server. I am able to send logstash local data using stdin from LS to my ES instance and I can interact with Kibana, but now I am trying to use a windows server 2012 r2 machine to send a log file to Logstash indexer.
Do I need logstash-forwarder (as a shipper)?
Do I need Redis (a broker)?
What would my logstash config file look like?
Do I need logstash-forwarder (as a shipper)?
No, but you could use it.
Do I need Redis (a broker)?
No, but you could use it.
What would my logstash config file look like?
You'll have to decide how to transport the messages off of the machine. As mentioned you could indeed have Logstash ship the messages to a Redis broker (using the redis output) and you'd have your central Logstash instance use a redis input to pull those messages.
You could also use the lumberjack protocol to ship the messages directly to the central Logstash instance. Then you'd use a lumberjack output on the Windows box and a lumberjack input on your central instance. The infrastructure for this is slightly less complicated and you could easily replace Logstash on the Windows machine with logstash-forwarder since it support the lumberjack protocol (and only that protocol).
Magnus,
Thanks for the answers! This I believe will help a ton.
Yes, I plan to move forward with lumberjack output/input. For now.
Best,
So I have downloaded and tried to install logstash-forwarder on my server 2012 box, but nothing seems to happen and I check processes and I don't see it running...
What should happen when trying to install logstash-forwarder on windows server machine?
There is no installation of logstash-forwarder. It's just an .exe file that you drop in any directory.
How are you invoking it?
I have the .exe on a windows server and I am trying to use LSF to ship a log file to my central input LS VM.
Besides having the .exe on the windows machine I want to ship files from, what else do I need?
Do I still need a output config file?
What do you mean by "invoke?"
Do you mean launching with cmd or powershell?
Besides having the .exe on the windows machine I want to ship files from, what else do I need?
You need a configuration file. See the README file.
What do you mean by "invoke?"
Do you mean launching with cmd or powershell?
Yes. Invoke, run, start, launch.
What would you suggest to run the LSF .exe? I was just trying to double click it...
What would you suggest to run the LSF .exe? I was just trying to double click it...
Please read the documentation. The second line of the Configuring section shows you to start LSF and a couple of lines down there's a configuration file example.
I believe I have the config file done... There are a few things I know are wrong (ie: the "type" in the files)
I am not surer how to access this config with the .exe...
am I on the right track?
#logstash-forwarder
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
{
"network": {
"servers": [ "DNS server:5000" ]
"timeout": 15,
"ssl ca":
},
"files": [
{
"paths": [
"user/appprograms/businesslayer.log"
],
"fields": { "type": "?" } #I am unsure of what type this is
}
]
}
It looks like you're mixing Logstash and LSF configurations. The "input" section doesn't belong in an LSF file.
Otherwise it looks fairly reasonable. The type is basically any string that describes what kind of message it is.
Once again, the reader file shows how to start LSF and pass the configuration file path as an argument.
Okay I used the following argument to run LFS via powershell
.\logstash-forwarder_windows_386.exe -config .\logstash-forwarder.conf
I'm haivng issues with the SSL certs, which I show in the message below my LSF configuration.
Also, here is my LSF config file but it is having SSL issues.
{
"network": {
"servers": [ "private_ip:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},
"files": [
{
"paths": [
"SEUM.BusinessLayer.log" ],
"fields": { "type": "syslog" }
}
]
}
after I run this I get an error saying:
"
setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
Failure reading CA certificate: open /etc/pki/tls/certs/logstash-forwarder.crt: The system cannot find the path specified.
The documentation contains an example of how you can create key and certificate files using OpenSSL. You can get that program from the cygwin distribution and there are possibly other Windows ports of OpenSSL. I'm sure there are other native Windows programs for generating X.509 certificates. It's a standardized format.
Magnus,
I have the following LSF config. I am getting a Failed message cause it can't "open /certs/lsfcert.crt: the system cannot find the path specified"
I am not sure how to specify the file path. The LSF.exe and LSF.conf are in the same directory and the the SSL folder is in the same directory as well.
Ideas?
{
"network": {
"servers": [ "x.x.x.x:xxxx" ],
"timeout": 15,
"ssl certificate": "/certs/lsfcert.crt",
"ssl key": "certs/lsfkey.key",
"ssl ca": "/certs/lsfcert.crt"
},
"files": [
{
"paths":
[ "SEUM.BusinessLayer.log" ],
"fields": { "type": "syslog" }
}
]
}
Relative paths might work but I'd try an absolute path ("c:/foo/bar/lsfcert.crt") first.
On my central logstash I have the following configs. Am I on the right track?
lsf-input.conf
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
=========================================
esearch-output.conf
output {
elasticsearch { host => elasticsearch-server:5601 }
stdout { codec => rubydebug }
}
Yes, except that I don't think you have ES running on port 5601. That's the Kibana port. By default ES's HTTP interface runs on port 9200. However, by default the elasticsearch output doesn't use HTTP so you'd have to specify port 9300. I suggest you just drop the port altogether. As long as you stick to the defaults you don't have to specify it explicitly.
After launching Elasticsearch server, Logstash central, and trying to connect LSF on windows server 2012.
I get the following error on windows shipper:
2015/09/14 16:02:16.444155 Read error looking for ack: WSARecv tcp x.x.x.x
2015/09/14 16:02:16.445154 Setting trusted CA from file: C:/Users/uxxxxxxx/Des
2015/09/14 16:02:16.447156 Connecting to [x.x.x.x]:5000 (x.x.x.x.)
Any ideas?
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.