LS not ingesting syslog

Elastic Stack Logstash
1

Hi,

I've installed this on a docker container, this is the current docker-compose.yaml:

version: '2'
services:
  elasticsearch:
    image: elasticsearch-img:6.3.2
    container_name: elasticsearch-container
    volumes:
      - /data/elasticsearch-1/:/usr/share/elasticsearch/data
    ports: 
      - 9200:9200 #Elasticsearch HTTP
      - 9300:9300 #Elasticsearch TCP transport
    network_mode: bridge
    restart: always
    environment:
      # - cluster.name=docker-cluster
      # - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    
  logstash:
    image: logstash-img:6.3.2
    container_name: logstash-container
    ports:
      - 5000:5000 #logstash TCP input
      - 514:5140  #listent to syslog on 514(host) map it to 5140(container) 541 is reserved and needs root
      - 514:5140/udp  #listent to syslog on 514(host) map it to 5140(container) 541 is reserved and needs root
    restart: always #restarts on reboot
    environment:
      - "LS_JAVA_OPTS=-Xms8g -Xmx8g"
      - "SYNLITE_SYSLOG_TEMPLATE_PATH=/usr/share/logstash/syslog/templates"
      - "SYNLITE_SYSLOG_GROK_PATTERNS_DIR=/usr/share/logstash/syslog/patterns"
      - "SYNLITE_SYSLOG_RESOLVE_IP2HOST=true"
      - "SYNLITE_SYSLOG_NAMESERVER=8.8.8.8"
      - "SYNLITE_SYSLOG_ES_HOSTS=elasticsearch:9200"
      # - "SYNLITE_SYSLOG_ES_USER=elastic"
      # - "SYNLITE_SYSLOG_ES_PASSWORD=changeme"
      - "SYNLITE_SYSLOG_TCP_HOST=0.0.0.0"
      - "SYNLITE_SYSLOG_TCP_PORT=514"
      - "SYNLITE_SYSLOG_UDP_HOST=0.0.0.0"
      - "SYNLITE_SYSLOG_UDP_PORT=514"
      - "SYNLITE_SYSLOG_MSG_TIMESTAMP=true"
      - "SYNLITE_SYSLOG_TZ=UTC"
    network_mode: bridge 
    links:
    - elasticsearch
    depends_on:
    - elasticsearch

Initially, port mapping of 514:514 made docker crib stating that permission was denied. I'm guessing this is because it's a port < 1000 and hence is previlaged. I've mapped 514:5140 within the container.

My /etc/rsyslog.conf looks like below:

...
...
# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
...
...

I'm able to see syslog being written on to /var/log/syslog . It works when I do something like: logger -s " This is a test "

However, I do not see anyting being picked up by LS/ES. What am I missing?

Thanks

2

Can you post your logstash config otherwise we cannot help.

3

Oh, sorry. Yes. It's exactly as how it is here - I believe this is more to do with my rsyslog coniguration on the machine rather than ELK itself

4

Still need to see the logstash config

5

I've put up the correct link in the previous reply.

Here it is below:

input {
  # Listen on TCP/514 for syslog messages.
  tcp {
    id => "tcp_syslog"
    host => "${SYNLITE_SYSLOG_TCP_HOST:0.0.0.0}"
    port => "${SYNLITE_SYSLOG_TCP_PORT:514}"
    type => "syslog"
  }

  # Listen on UDP/514 for syslog messages.
  udp {
    id => "udp_syslog"
    host => "${SYNLITE_SYSLOG_UDP_HOST:0.0.0.0}"
    port => "${SYNLITE_SYSLOG_UDP_PORT:514}"
    type => "syslog"
  }
}

<<a whole lot of filters>>

output {
  elasticsearch {
    id => "output_elasticsearch"
    hosts => [ "${SYNLITE_SYSLOG_ES_HOSTS:127.0.0.1:9200}" ]
    user => "${SYNLITE_SYSLOG_ES_USER:elastic}"
    password => "${SYNLITE_SYSLOG_ES_PASSWORD:changeme}"
    index => "syslog-%{+YYYY.MM.dd}"
    template => "${SYNLITE_SYSLOG_TEMPLATE_PATH:/etc/logstash/synesis_lite_syslog/templates}/synesis_lite_syslog.template.json"
    template_name => "synesis_lite_syslog"
    template_overwrite => "true"
  }
}
6

I'm unsure about the use of variables in the config, have you tried putting in IP Addresses to test first?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.