Logs to logstash config file

Elastic Stack Logstash
1

I have a simple log file consisting of some information...how do convert it to a logstash-configuration file for indexing in elasticsearch and visualizing in kibana?
I am confused. please help!

2

I'm shipping some logs from server 1 to server 2 using logstash-shipper and receiving the message on redis...

the config file i'm using is this:

input {
file {
path => "/path/to/my/logfile.log"
type => "apache"
}
}

filter {
if [type] == "apache" {
grok {
pattern => "%{COMBINEDAPACHELOG}"
}
}
}

output {
redis { host => "myserverIP" data_type => "list" key => "logstash" }
}

when i type
java -Xmx256m -jar logstash-1.3.2-flatjar.jar agent -f /etc/logstash/sys-log-test.conf
on the command like...i get the following message and nothing else:

Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, ple ase let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-mil estones {:level=>:warn}
You are using a deprecated config setting "pattern" set in grok. Deprecated settings will continue to wor k, but are scheduled for removal from logstash in the future. You should use this instead: match => { "me ssage" => "your pattern here" } If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"pattern", :plugin=><LogStash::Filters::Grok --->, :level=>:warn}
Using milestone 2 output plugin 'redis'. This plugin should be stable, but if you see strange behavior, p lease let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-m ilestones {:level=>:warn}

What am i doing wrong? Please help!

3

It looks like it wants you to use the newer grok filter. Try changing this:

grok {
pattern => "%{COMBINEDAPACHELOG}"
}

to this:

grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
}
4

Tried it still doesn't work..

When i open up redis-cli..i type ping..it returns pong which works correctly..

When i send logs from logstash shipper to redis...it is suppose to display a message like
[83019] 02 Jul 12:51:02 - Accepted 127.0.0.1:58312
[83019] 02 Jul 12:51:06 - Client closed connection
[83019] 02 Jul 12:51:06 - DB 0: 1 keys (0 volatile) in 4 slots HT.

but i am not seeing this message. logstash shipper is not able to connect to redis-cli...

whats happening?

5

please help! i have been stuck with problem for long time...

6

the error i am getting is this:

Failed to send event to redis {:event=>#<LogStash::Event:0x482a0e9e @cancelled=false, @data={"message"=>"Hi", "@version"=>"1", "@timestamp"=>"2016-03-18T17:23:14.588Z", "host"=>"meshka"}>, :identity=>"default", :exception=>#<Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (ECONNREFUSED)>, :backtrace=>["file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/redis/client.rb:278:in establish_connection'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/redis/client.rb:69:inconnect'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/redis/client.rb:292:in ensure_connected'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/redis/client.rb:179:inprocess'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/redis/client.rb:258:in logging'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/redis/client.rb:178:inprocess'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/redis/client.rb:84:in call'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/redis.rb:970:inrpush'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/redis.rb:36:in synchronize'", "jar:file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/META-INF/jruby.home/lib/ruby/1.9/monitor.rb:211:inmon_synchronize'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/redis.rb:36:in synchronize'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/redis.rb:969:inrpush'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/logstash/outputs/redis.rb:163:in receive'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/logstash/outputs/base.rb:86:inhandle'", "(eval):21:in initialize'", "org/jruby/RubyProc.java:271:incall'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/logstash/pipeline.rb:259:in output'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/logstash/pipeline.rb:218:inoutputworker'", "file:/opt/Oracle/logstash_ol/logstash-1.3.2-flatjar.jar!/logstash/pipeline.rb:145:in `start_outputs'"], :level=>:warn}

7

Problem solved. All i did i was restart logstash and redis.