# Сбор syslog с помощью ELK

**URL:** <https://discuss.elastic.co/t/syslog-elk/279993>\
**Category:** Вопросы на русском языке\
**Created:** [July 29, 2021, 1:38pm UTC](https://discuss.elastic.co/t/syslog-elk/279993 "2021-07-29T13:38:31Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Savai](https://avatars.discourse-cdn.com/v4/letter/s/9f8e36/32.png) [@Savai](https://discuss.elastic.co/u/Savai)\
**Post date:** [July 29, 2021, 1:38pm UTC](https://discuss.elastic.co/t/syslog-elk/279993/1 "2021-07-29T13:38:31Z")

</div>

Здравствуйте. Не подскажете, в чем может быть проблема?  
Запускаю ELK стек через docker-compose и пытаюсь получить syslog через logstash, однако индекс в elastic не создается и при запуске logstash выдается предупреждение.

```auto
"lgs01 | [2021-07-29T13:37:30,507][ERROR][logstash.outputs.elasticsearch][main] Failed to
 install template {:message=>"Got response code '403' contacting Elasticsearch at URL 
'https://es01:9200/_template/logstash'", 
:exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, 
:backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-
elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client
/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0
/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client
/pool.rb:306:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0
/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client
/pool.rb:293:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0
/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client
/pool.rb:382:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0
/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client
/pool.rb:292:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0
/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client
/pool.rb:300:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-
output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:379:in `exists?'", 
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-
java/lib/logstash/outputs/elasticsearch/http_client.rb:384:in `template_exists?'", "/usr/share
/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash
/outputs/elasticsearch/http_client.rb:80:in `template_install'", "/usr/share/logstash/vendor
/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs
/elasticsearch/template_manager.rb:29:in `install'", "/usr/share/logstash/vendor/bundle/jruby
/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch
/template_manager.rb:17:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0
/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch.rb:496:in 
`install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-
elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch.rb:309:in `finish_register'", 
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-
java/lib/logstash/outputs/elasticsearch.rb:279:in `block in register'", "/usr/share/logstash/vendor
/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/plugin_mixins
/elasticsearch/common.rb:145:in `block in after_successful_connection'"]}
"

```

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [July 29, 2021, 11:20pm UTC](https://discuss.elastic.co/t/syslog-elk/279993/2 "2021-07-29T23:20:42Z")

</div>

Здравствуйте! Хотелось бы услышать о Вашей проблеме поподробнее. Пока могу только указать на [Syslog input plugin | Logstash Reference [7.13] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html) если интересует именно ELK, если пойдет [BELK](https://www.elastic.co/what-is/elk-stack), то можно еще взглянуть на [Syslog input | Filebeat Reference [7.13] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-syslog.html)

---

<div class="post-metadata">

**Author:** ![Savai](https://avatars.discourse-cdn.com/v4/letter/s/9f8e36/32.png) [@Savai](https://discuss.elastic.co/u/Savai)\
**Post date:** [July 30, 2021, 4:54am UTC](https://discuss.elastic.co/t/syslog-elk/279993/3 "2021-07-30T04:54:25Z")

</div>

Конфигурация pipeline

```auto
input {
  udp {
    port => 514
    type => "syslog"
  }
}

output {
  elasticsearch
          {
           hosts => ["https://es01:9200"]
           user => "logstash_system"
           password => "logstashpasswd"
           ssl_certificate_verification => false
           index => "syslog-%{+YYYY.MM.dd}"
          }
}

```

---

<div class="post-metadata">

**Author:** ![Savai](https://avatars.discourse-cdn.com/v4/letter/s/9f8e36/32.png) [@Savai](https://discuss.elastic.co/u/Savai)\
**Post date:** [July 30, 2021, 5:11am UTC](https://discuss.elastic.co/t/syslog-elk/279993/4 "2021-07-30T05:11:39Z")

</div>

А еще, не подскажете- в интерфейсе Elastic-a есть пункт создания pipeline для logstash- можно ли создавать pipelines только там, без использования файлов, и если да- то как заставить это работать?

 ![изображение](https://us1.discourse-cdn.com/elastic/original/3X/9/0/90687e077d0076bb5d67c3d9be308537d5c0a9da.png)

---

<div class="post-metadata">

**Author:** ![Savai](https://avatars.discourse-cdn.com/v4/letter/s/9f8e36/32.png) [@Savai](https://discuss.elastic.co/u/Savai)\
**Post date:** [July 30, 2021, 11:54am UTC](https://discuss.elastic.co/t/syslog-elk/279993/5 "2021-07-30T11:54:41Z")

</div>

Зарегистрировал пользователя logstash\_internal, как написанно в документации, теперь ошибок доступа при запуске logstash нет, но индекс все равно не создается. Что я делаю не так?

```auto
input {
  udp {
    port => 514
    type => "syslog"
  }
}

output {
  elasticsearch
          {
           hosts => ["https://es01:9200"]
           user => logstash_internal
           password => logstas_internal_password
           ssl_certificate_verification => false
           index => "syslog-%{+YYYY.MM.dd}"
          }
}

```

---

<div class="post-metadata">

**Author:** ![Savai](https://avatars.discourse-cdn.com/v4/letter/s/9f8e36/32.png) [@Savai](https://discuss.elastic.co/u/Savai)\
**Post date:** [July 30, 2021, 2:00pm UTC](https://discuss.elastic.co/t/syslog-elk/279993/6 "2021-07-30T14:00:21Z")

</div>

docker-compose для запуска ELK

```auto
version: '2.2'

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    ports:
      - 9200:9200
    networks:
      - elastic

    healthcheck:
      test: curl --cacert $CERTS_DIR/ca/ca.crt -s https://localhost:9200 >/dev/null; if [[$$? == 52]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es02/es02.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es02/es02.crt
      - xpack.security.transport.ssl.enabled=true
 - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es02/es02.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es02/es02.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es03/es03.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es03/es03.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es03/es03.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es03/es03.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    networks:
      - elastic
  kib01:
    image: docker.elastic.co/kibana/kibana:${VERSION}
    container_name: kib01
    depends_on: {"es01": {"condition": "service_healthy"}}
    ports:
      - 5601:5601
    environment:
      SERVERNAME: localhost
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: kibana_system_password
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: $CERTS_DIR/kib01/kib01.key
 SERVER_SSL_CERTIFICATE: $CERTS_DIR/kib01/kib01.crt
    volumes:
      - certs:$CERTS_DIR
    networks:
      - elastic
  lgs01:
     image: docker.elastic.co/logstash/logstash:${VERSION}
     container_name: lgs01
     depends_on: {"es01": {"condition": "service_healthy"}}
     ports:
       - 5000:5000
       - 5044:5044
     environment:
       - xpack.security.enabled=true
       - xpack.monitoring.enabled=true
       - xpack.monitoring.elasticsearch.hosts=["https://es01:9200"]
       - xpack.monitoring.elasticsearch.username=logstash_system
       - xpack.monitoring.elasticsearch.PASSWORD=logstash_system_password
       - xpack.monitoring.elasticsearch.ssl.certificate_authority=$CERTS_DIR/ca/ca.crt
       - xpack.monitoring.elasticsearch.ssl.verification_mode=certificate
       - xpack.management.enabled:=true
       - xpack.management.elasticsearch.hosts:="https://es01:9200/"
       - xpack.management.elasticsearch.username:=logstash_admin
       - xpack.management.elasticsearch.password:=logstash_admin_password
       - xpack.management.logstash.poll_interval:=5s
       - "LS_OPTS= --config.reload.automatic"
       - "LS_JAVA_OPTS= -Xms1024m -Xmx1024m"
     volumes:
       - certs:$CERTS_DIR
       #- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
       #- ./logstash/pipeline:/usr/share/logstash/pipeline:ro
     networks:
       - elastic
volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local
  certs:
    driver: local

networks:
  elastic:
    driver: bridge

```

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [July 30, 2021, 4:07pm UTC](https://discuss.elastic.co/t/syslog-elk/279993/7 "2021-07-30T16:07:44Z")

</div>

Не понятно, кто у вас сообщения в syslog посылает и как он до logstash добирается, если у вас порт 514 в докере не открыт.

---

<div class="post-metadata">

**Author:** ![Savai](https://avatars.discourse-cdn.com/v4/letter/s/9f8e36/32.png) [@Savai](https://discuss.elastic.co/u/Savai)\
**Post date:** [August 2, 2021, 6:46am UTC](https://discuss.elastic.co/t/syslog-elk/279993/8 "2021-08-02T06:46:55Z")

</div>

Syslog шлются на 514 порт сервера где установлен ELK  
добавил в docker-compose строчку

```auto
 - 192.168.1.15:514:5044

```

ошибок при запуске нет, но индекс все так же не создется.  
пробовал создавать pipelines через интерфейс kibana- тоже не работает

Результат вывода команды docker container port lgs01

```auto
5044/tcp -> 0.0.0.0:5044
5044/tcp -> :::5044
5044/tcp -> 192.168.1.15:514
5000/tcp -> 0.0.0.0:5000
5000/tcp -> :::5000

```

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [August 2, 2021, 3:36pm UTC](https://discuss.elastic.co/t/syslog-elk/279993/9 "2021-08-02T15:36:56Z")

</div>

```auto
input {
  udp {
    port => 514
    type => "syslog"
  }
}

```

Судя по этой конфигурации, ваш Logstash внутри контейнера ожидает syslog трафик на порту 514, а не 5044 и по UDP, а не по TCP.

---

<div class="post-metadata">

**Author:** ![Savai](https://avatars.discourse-cdn.com/v4/letter/s/9f8e36/32.png) [@Savai](https://discuss.elastic.co/u/Savai)\
**Post date:** [August 3, 2021, 5:58am UTC](https://discuss.elastic.co/t/syslog-elk/279993/10 "2021-08-03T05:58:14Z")

</div>

Syslog на сервер отправляется по udp, поэтому для udp и настраивал.  
Указал порт 5044- все равно индек не создается и данные не собираются.

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [August 3, 2021, 3:44pm UTC](https://discuss.elastic.co/t/syslog-elk/279993/11 "2021-08-03T15:44:40Z")

</div>

У вас маппинг порта в докере настроен по TCP а посылаете и слушаете вы по UDP. Посмотрите [документацию](https://docs.docker.com/config/containers/container-networking/#published-ports) докера.

---

<div class="post-metadata">

**Author:** ![Savai](https://avatars.discourse-cdn.com/v4/letter/s/9f8e36/32.png) [@Savai](https://discuss.elastic.co/u/Savai)\
**Post date:** [August 5, 2021, 11:31am UTC](https://discuss.elastic.co/t/syslog-elk/279993/12 "2021-08-05T11:31:56Z")

</div>

Спасибо за подсказки! С настройкой pipeline через файл все заботало.  
А не подскажете- в интерфейсе Kibana есть создания logstash pipeline - копию туда рабочий pipeline из файла- а он не работает, в какую сторону смотреть?

 ![изображение](https://us1.discourse-cdn.com/elastic/original/3X/3/5/3573e6b3841d9c0407546f2c4aa363b6c2d3e2b6.png)

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [August 5, 2021, 10:50pm UTC](https://discuss.elastic.co/t/syslog-elk/279993/13 "2021-08-05T22:50:27Z")

</div>

```auto
       - xpack.management.enabled:=true
       - xpack.management.elasticsearch.hosts:="https://es01:9200/"
       - xpack.management.elasticsearch.username:=logstash_admin
       - xpack.management.elasticsearch.password:=logstash_admin_password
       - xpack.management.logstash.poll_interval:=5s

```

Двоеточие перед `=` попробуйте убрать.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 2, 2021, 10:50pm UTC](https://discuss.elastic.co/t/syslog-elk/279993/14 "2021-09-02T22:50:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.