Problems with Log4j Config

Hi all,

I'm fairly new at Elasticsearch so please don't throw things at me for
asking silly questions. As the newb I got my instance up with minimal
fuss with the following.

redis
nginx
elasticsearch
logstash

I currently have a syslog entry already in my logstash.conf file and I'm
ready to add one for my Log4J stuff. Here are the contents of my original
logstash.conf.

input {
redis {
host => "My IP goes here"
data_type => "list"
type => "redis-input"
key => "logstash"
}
syslog {
type => syslog
port => 5514
}
}

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp}
%{SYSLOGHOST:syslog_hostname}
%{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?:
%{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

output {
stdout { }
elasticsearch {
cluster => "logstash"
#below entry is because I haven't upgraded yet and there is a known bug
with older versions and changing the ports
protocol => "http"
}
}

The above works great and I'm able to forward syslog events with no
problems.

Here is my log4j.config.

input {
redis {
host => "My IP goes here"
data_type => "list"
type => "redis-input"
key => "logstash"
}
log4j {
mode => "server"
port => 9500
type => eslogs
}
}
output {
stdout { }
elasticsearch {
type => eslogs
cluster => "logstash"
protocol => "http"
}
}

And on my log4j server vmware server I have this:

log4j.appender.vcloud.system.syslog=org.apache.log4j.net.SyslogAppender
log4j.appender.vcloud.system.syslog.syslogHost=My IP goes here:9500
log4j.appender.vcloud.system.syslog.facility=LOCAL1
log4j.appender.vcloud.system.syslog.layout=com.vmware.vcloud.logging.CustomPatternLayout
log4j.appender.vcloud.system.syslog.layout.ConversionPattern=%d{ISO8601} |
%-8.8p | %-25.50t | %-30.50c{1} | %m | %x%n
log4j.appender.vcloud.system.syslog.threshold=INFO

Now this is the part that I don't understanding and doesn't work right.

  1. I need to add a new log4j.config as noted here
    http://www.elasticsearch.org/blog/logging-elasticsearch-events-with-logstash-and-elasticsearch/,
    but do I need to add my redis server into every config file I create?

  2. Also I'm getting no results coming from my log4j servers. Zero
    results. I did try the --configtest and fixed some of the problems in my
    config but still missing something. No error in the
    logstash.err/.log/.stdout files and iptables is off.

What am I doing wrong here?

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/020acf0b-72ce-4d4d-8209-668c8747cf74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

The log4j syslog appender uses UDP which does not seem to work in your case.

I do not recommend using UDP because it is not reliable.

Check log4j2 for async logging and TCP for syslog.

http://logging.apache.org/log4j/2.x/manual/appenders.html#SyslogAppender

Jörg

On Mon, Sep 29, 2014 at 11:51 PM, Tallguy tallguy117@gmail.com wrote:

Hi all,

I'm fairly new at Elasticsearch so please don't throw things at me for
asking silly questions. As the newb I got my instance up with minimal
fuss with the following.

redis
nginx
elasticsearch
logstash

I currently have a syslog entry already in my logstash.conf file and I'm
ready to add one for my Log4J stuff. Here are the contents of my original
logstash.conf.

input {
redis {
host => "My IP goes here"
data_type => "list"
type => "redis-input"
key => "logstash"
}
syslog {
type => syslog
port => 5514
}
}

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp}
%{SYSLOGHOST:syslog_hostname}
%{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?:
%{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

output {
stdout { }
elasticsearch {
cluster => "logstash"
#below entry is because I haven't upgraded yet and there is a known bug
with older versions and changing the ports
protocol => "http"
}
}

The above works great and I'm able to forward syslog events with no
problems.

Here is my log4j.config.

input {
redis {
host => "My IP goes here"
data_type => "list"
type => "redis-input"
key => "logstash"
}
log4j {
mode => "server"
port => 9500
type => eslogs
}
}
output {
stdout { }
elasticsearch {
type => eslogs
cluster => "logstash"
protocol => "http"
}
}

And on my log4j server vmware server I have this:

log4j.appender.vcloud.system.syslog=org.apache.log4j.net.SyslogAppender
log4j.appender.vcloud.system.syslog.syslogHost=My IP goes here:9500
log4j.appender.vcloud.system.syslog.facility=LOCAL1

log4j.appender.vcloud.system.syslog.layout=com.vmware.vcloud.logging.CustomPatternLayout
log4j.appender.vcloud.system.syslog.layout.ConversionPattern=%d{ISO8601} |
%-8.8p | %-25.50t | %-30.50c{1} | %m | %x%n
log4j.appender.vcloud.system.syslog.threshold=INFO

Now this is the part that I don't understanding and doesn't work right.

  1. I need to add a new log4j.config as noted here
    http://www.elasticsearch.org/blog/logging-elasticsearch-events-with-logstash-and-elasticsearch/,
    but do I need to add my redis server into every config file I create?

  2. Also I'm getting no results coming from my log4j servers. Zero
    results. I did try the --configtest and fixed some of the problems in my
    config but still missing something. No error in the
    logstash.err/.log/.stdout files and iptables is off.

What am I doing wrong here?

Thanks,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/020acf0b-72ce-4d4d-8209-668c8747cf74%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/020acf0b-72ce-4d4d-8209-668c8747cf74%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEk1dvrsN6TCOp5fPmEnNCraHEuuDJaXaY10KvMb%3DeEEw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.