Use elasticsearch with syslog

hy
I try to use elasticsearch with my syslog server for that i creat a
facility with syslog.
I have a multiple question

  1. my configuration in logging.yml is good ? or i make a mistake
  2. if you have already implemented elasticsearch with syslog, have i done a
    mistake in my rsyslog.conf ?
    thanks for your answer

in host 1
elasticsearch/logging.yml

43 rootLogger: DEBUG, console, file, syslog,
44 syslog:
45 type: syslog
46 syslogHost: 10.110.0.60:515
47 facility: local3
48 layout:
49 type: pattern
50 conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

in my* /etc/rsyslog.conf*
local3. @syslog01*

in my syslog server
113 if $hostname contains 'bo' and $syslogfacility-text == 'local3' then
-/data/log/esDev.log

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Nobody use elasticsearch with syslog .
Nobody know ?

Le mardi 30 avril 2013 18:10:27 UTC+2, olivier morel a écrit :

hy
I try to use elasticsearch with my syslog server for that i creat a
facility with syslog.
I have a multiple question

  1. my configuration in logging.yml is good ? or i make a mistake
  2. if you have already implemented elasticsearch with syslog, have i done
    a mistake in my rsyslog.conf ?
    thanks for your answer

in host 1
elasticsearch/logging.yml

43 rootLogger: DEBUG, console, file, syslog,
44 syslog:
45 type: syslog
46 syslogHost: 10.110.0.60:515
47 facility: local3
48 layout:
49 type: pattern
50 conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

in my* /etc/rsyslog.conf*
local3. @syslog01*

in my syslog server
113 if $hostname contains 'bo' and $syslogfacility-text == 'local3' then
-/data/log/esDev.log

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I' not sure what your issue is. Do you receive log messages or not?

Just a few hints, before I try to reproduce syslog logging:

  • the log4j syslog appender is udp only (just fyi, this means, it s not
    reliable and may lose log messages)
  • the standard port for logging by UDP/TCP is 514 (not 515)
  • the log port must be open in firewall settings

Jörg

Am 02.05.13 10:32, schrieb olivier morel:

Nobody use elasticsearch with syslog .
Nobody know ?

Le mardi 30 avril 2013 18:10:27 UTC+2, olivier morel a écrit :

hy
 I try to use elasticsearch with my syslog server for that i creat
a facility with syslog.
I have a multiple question
1) my configuration in logging.yml is good ? or i make a mistake
2) if you have already implemented elasticsearch with syslog, have
i done a mistake in my rsyslog.conf ?
thanks for your answer

in host 1
*elasticsearch/logging.yml*


 43 rootLogger: DEBUG, console, file, syslog,
 44 syslog:
 45     type: syslog
 46     syslogHost: 10.110.0.60:515 <http://10.110.0.60:515>
 47     facility: local3
 48     layout:
 49       type: pattern
 50       conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"


in my*/etc/rsyslog.conf*
/local3.*   @syslog01/



in my syslog server
113 if $hostname contains 'bo' and $syslogfacility-text ==
'local3'  then -/data/log/esDev.log

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Adding more info (for Redhat Linux):

By default, rsyslog does not listen at 514 for UDP. Enable it in
/etc/rsyslog.conf by removing the comment marks

Provides UDP syslog reception

$ModLoad imudp
$UDPServerRun 514

and add a line

local3.* /var/log/elasticsearch.log

on the receiving Host . Restart rsyslog by /etc/init.d/rsyslogd
restart

Add this to /etc/sysconfig/iptables for each ES host

-A RH-Firewall-1-INPUT –s -m udp -p udp --dport 514 -j ACCEPT

restart firewall on by /etc/init.d/iptables restart

Then, add something like this to $ES_HOME/config/logging.yml

rootLogger: INFO, console, file, syslog
[...]
appender:
syslog:
type: syslog
syslogHost: :514
facility: local3
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

and restart ES on each . You should see messages in
/var/log/elasticsearch.log on

This is UDP and IPv4 only. Optionally, you may find how to use TCP
and/or IPv6. The setup is very similar.

Jörg

Am 02.05.13 12:20, schrieb Jörg Prante:

I' not sure what your issue is. Do you receive log messages or not?

Just a few hints, before I try to reproduce syslog logging:

  • the log4j syslog appender is udp only (just fyi, this means, it s
    not reliable and may lose log messages)
  • the standard port for logging by UDP/TCP is 514 (not 515)
  • the log port must be open in firewall settings

Jörg

Am 02.05.13 10:32, schrieb olivier morel:

Nobody use elasticsearch with syslog .
Nobody know ?

Le mardi 30 avril 2013 18:10:27 UTC+2, olivier morel a écrit :

hy
 I try to use elasticsearch with my syslog server for that i creat
a facility with syslog.
I have a multiple question
1) my configuration in logging.yml is good ? or i make a mistake
2) if you have already implemented elasticsearch with syslog, have
i done a mistake in my rsyslog.conf ?
thanks for your answer

in host 1
*elasticsearch/logging.yml*


 43 rootLogger: DEBUG, console, file, syslog,
 44 syslog:
 45     type: syslog
 46     syslogHost: 10.110.0.60:515 <http://10.110.0.60:515>
 47     facility: local3
 48     layout:
 49       type: pattern
 50       conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"


in my*/etc/rsyslog.conf*
/local3.*   @syslog01/



in my syslog server
113 if $hostname contains 'bo' and $syslogfacility-text ==
'local3'  then -/data/log/esDev.log

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I try what you write

i write on my //etc/elasticsearch/logging.yml /

rootLogger: DEBUG, console, file, syslog **
syslog:
** type: syslog
*
** syslogHost: 10.110.0.60:514**
** facility: local3**
** layout:**
** type: pattern**
** #conversionPattern: "ES: %-5p | [%t] | %c | %m%n"**
** conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" **
*

in my /rsyslog.conf /
local3. @syslog01*

In my syslog server i have write in the /rsyslog.conf /

provides UDP syslog reception

$ModLoad imudp*
**$UDPServerRun 514 *

**if $hostname contains 'bo' and $syslogfacility-text == 'local3' then
-/mnt/log/esDev.log *

But i get nothing in my esDev.log :frowning:

if i try to send a meesage by the port 514 with netcat it's working

|echo| |"hellow world !!"| || ncat -u @syslog01 514

i get it in my /var/log/syslog

|

Le 02/05/2013 12:47, Jörg Prante a écrit :

Adding more info (for Redhat Linux):

By default, rsyslog does not listen at 514 for UDP. Enable it in
/etc/rsyslog.conf by removing the comment marks

Provides UDP syslog reception

$ModLoad imudp
$UDPServerRun 514

and add a line

local3.* /var/log/elasticsearch.log

on the receiving Host . Restart rsyslog by
/etc/init.d/rsyslogd restart

Add this to /etc/sysconfig/iptables for each ES host

-A RH-Firewall-1-INPUT –s -m udp -p udp --dport 514 -j ACCEPT

restart firewall on by /etc/init.d/iptables restart

Then, add something like this to $ES_HOME/config/logging.yml

rootLogger: INFO, console, file, syslog
[...]
appender:
syslog:
type: syslog
syslogHost: :514
facility: local3
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

and restart ES on each . You should see messages in
/var/log/elasticsearch.log on

This is UDP and IPv4 only. Optionally, you may find how to use TCP
and/or IPv6. The setup is very similar.

Jörg

Am 02.05.13 12:20, schrieb Jörg Prante:

I' not sure what your issue is. Do you receive log messages or not?

Just a few hints, before I try to reproduce syslog logging:

  • the log4j syslog appender is udp only (just fyi, this means, it s
    not reliable and may lose log messages)
  • the standard port for logging by UDP/TCP is 514 (not 515)
  • the log port must be open in firewall settings

Jörg

Am 02.05.13 10:32, schrieb olivier morel:

Nobody use elasticsearch with syslog .
Nobody know ?

Le mardi 30 avril 2013 18:10:27 UTC+2, olivier morel a écrit :

hy
 I try to use elasticsearch with my syslog server for that i creat
a facility with syslog.
I have a multiple question
1) my configuration in logging.yml is good ? or i make a mistake
2) if you have already implemented elasticsearch with syslog, have
i done a mistake in my rsyslog.conf ?
thanks for your answer

in host 1
*elasticsearch/logging.yml*


 43 rootLogger: DEBUG, console, file, syslog,
 44 syslog:
 45     type: syslog
 46     syslogHost: 10.110.0.60:515 <http://10.110.0.60:515>
 47     facility: local3
 48     layout:
 49       type: pattern
 50       conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"


in my*/etc/rsyslog.conf*
/local3.*   @syslog01/



in my syslog server
113 if $hostname contains 'bo' and $syslogfacility-text ==
'local3'  then -/data/log/esDev.log

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You added this under the "appender:" line?

appender:
console: ...
file: ...
syslog: ...

Jörg

Am 02.05.13 17:03, schrieb gmail:

syslog:*
** type: syslog**
** syslogHost: 10.110.0.60:514**
** facility: local3**
** layout:**
** type: pattern**
** #conversionPattern: "ES: %-5p | [%t] | %c | %m%n"**
** conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" **
*

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

As you can see the block is just below the appender
appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

syslog:
type: syslog
syslogHost: 10.110.X.X:514
facility: local3
layout:
type: pattern
#conversionPattern: "ES: %-5p | [%t] | %c | %m%n"
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

Le 02/05/2013 17:10, Jörg Prante a écrit :

You added this under the "appender:" line?

appender:
console: ...
file: ...
syslog: ...

Jörg

Am 02.05.13 17:03, schrieb gmail:

syslog:*
** type: syslog**
** syslogHost: 10.110.0.60:514**
** facility: local3**
** layout:**
** type: pattern**
** #conversionPattern: "ES: %-5p | [%t] | %c | %m%n"**
** conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" **
*

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Please note, YAML is indentation sensitive. You have to indent the
syslog appender definition to get recognized correctly as an appender.

Jörg

Am 02.05.13 17:15, schrieb gmail:

As you can see the block is just below the appender

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You were right is juste a problem with my indentation i put here all of
my configuration for helping somebody .

in /etc/elasticsearch/logging.yml

rootLogger: DEBUG, console, file, syslog

29 appender:
30 console:
31 type: console
32 layout:
33 type: consolePattern
34 conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
35
36 syslog:
37 type: syslog
38 syslogHost: 10.110.X.X:514
39 facility: local3
40 layout:
41 type: pattern
42 #conversionPattern: "ES: %-5p | [%t] | %c | %m%n"
43 conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
44

in /etc/rsyslog.conf

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support (previously done
by rklogd)
#$ModLoad immark # provides --MARK-- message capability

provides UDP syslog reception

#$ModLoad imudp
#$UDPServerRun 514
local3.* @syslog01

In my syslog server (i use the facility of rsyslog )

if $hostname contains 'bo' and $syslogfacility-text == 'local3' then
-/mnt/log/esDev.log

If you have a problem watch the indentation because YAML is sensitive.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.