# Filebeat ошибка - "Cannot index event publisher.Event"

**URL:** <https://discuss.elastic.co/t/filebeat-cannot-index-event-publisher-event/205843>\
**Category:** Вопросы на русском языке\
**Tags:** docker\
**Created:** [October 30, 2019, 12:01pm UTC](https://discuss.elastic.co/t/filebeat-cannot-index-event-publisher-event/205843 "2019-10-30T12:01:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![IwanHSKY](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iwanhsky/32/56594_2.png) [@IwanHSKY](https://discuss.elastic.co/u/IwanHSKY)\
**Post date:** [October 30, 2019, 12:01pm UTC](https://discuss.elastic.co/t/filebeat-cannot-index-event-publisher-event/205843/1 "2019-10-30T12:01:45Z")

</div>

Всем привет, использую ELK Stack запущенный в Docker'e.  
Пытаюсь отправить логи из Java приложения запущенного в Docker'e.  
Тестировал 7.4.0, и 7.4.1 версии.

- Отправляю логи через filebeat сразу в ES.

Конфиг filebeat'a

> **Filebeat.yml**
>
> ```auto
> # Modules configuration ==
> filebeat.modules:
> 
> #------------------------------- System Module -------------------------------
> - module: system
> # Syslog
> syslog:
> enabled: true
> var.paths: ["/var/log/syslog*", "/var/log/messages*"]
> auth:
> enabled: true
> var.paths: ["/var/log/auth.log*", "/var/log/secure*"]
> 
> #------------------------------- Auditd Module -------------------------------
> # Does not look like Auditd is supported in Alpine linux:
> # https://github.com/linuxkit/linuxkit/issues/52
> - module: auditd
> log:
> enabled: false
> 
> # Template 
> setup.template.enabled: true
> setup.template.name: "filebeat"
> setup.template.pattern: "filebeat-${INDEX_NAME:default}-%{[beat.version]}-*"
> 
> # Filebeat inputs 
> 
> filebeat.inputs:
> - type: log
> enabled: true
> paths: ['/var/log/nsi-backend/nsi-backend.log']
> json.message_key: message
> json.keys_under_root: true
> json.add_error_key: true
> 
> # Elasticsearch output
> output.elasticsearch:
> hosts: ["${ELASTICSEARCH_HOST:http://172.22.130.64:9200}"]
> index: "filebeat-${INDEX_NAME:default}-%{[beat.version]}-%{+yyyy.MM.dd}"
> username: elastic
> password: ${ELASTIC_PASSWORD}
> 
> monitoring:
> enabled: true
> elasticsearch:
> 
> # Dashboards
> setup.dashboards:
> enabled: true
> index: "filebeat-${INDEX_NAME:default}-*"
> setup.kibana:
> host: "${KIBANA_HOST:http://172.22.130.64:5601}"
> username: elastic
> password: ${ELASTIC_PASSWORD}
> 
> ```

Java логи пишутся через log4j в JSON формате

> **Пример JSON log и log4j.properties**
>
> [Пример лога у приложения в JSON](https://pastebin.com/MCWmkkqG)
> 
> ```auto
> # Root logger option
> log4j.rootLogger=INFO, json
> # JSON log
> log4j.appender.json=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.json.File=/var/log/nsi-backend/nsi-backend.log
> log4j.appender.json.DatePattern=.yyyy-MM-dd
> log4j.appender.json.layout=net.logstash.log4j.JSONEventLayoutV1
> 
> ```

С данным конфигом получаю ошибку:  
Ошибка, в отправке данных в индекс - `elasticsearch/client.go:535 Cannot index event publisher.Event`:

> **Полный лог в debug'e Filebeat'a**
>
> [Filebeat debug log](https://pastebin.com/wC6jMgwn)

> **Сама ошибка**
>
> `2019-10-29T11:40:42.629Z WARN elasticsearch/client.go:535 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf66272db0a2e777, ext:48849912601, loc:(*time.Location)(0x4de6580)}, Meta:common.MapStr(nil), Fields:common.MapStr{"@version":1, "agent":common.MapStr{"ephemeral_id":"8bc91806-bd18-4ab3-bfa3-a9a987e94b78", "hostname":"nsi-ci", "id":"a5a65b6e-7343-4af6-84d5-618332ab615b", "type":"filebeat", "version":"7.4.1"}, "class":"my.self.nsi.logging.Slf4JLoggingAPI", "ecs":common.MapStr{"version":"1.1.0"}, "file":"Slf4JLoggingAPI.java", "host":common.MapStr{"name":"nsi-ci"}, "input":common.MapStr{"type":"log"}, "level":"INFO", "line_number":"89", "log":common.MapStr{"file":common.MapStr{"path":"/var/log/nsi-backend/nsi-backend.log"}, "offset":2831236}, "logger_name":"my.self.nsi.base.events.EventTransactionAPI", "mdc":common.MapStr{"isFatal":"false", "username":""}, "message":"[Start] Get table", "method":"lambda$info$6", "source_host":"8dff15b81656", "thread_name":"main"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000c9a1a0), Source:"/var/log/nsi-backend/nsi-backend.log", Offset:2831606, Timestamp:time.Time{wall:0xbf66272dac59c927, ext:48778011857, loc:(*time.Location)(0x4de6580)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x4000e, Device:0xfd00}}, TimeSeries:false}, Flags:0x1}`
> 
> ` (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [file] tried to parse field [file] as object, but found a concrete value"}`

Протестировал следующее, выставил `false` у `json.keys_under_root`  
**`json.keys_under_root: false`**  
Логи стали отправляться, но не все, stacktrace из примера не отправляется, в кибане его нет.

> **Пример JSON лога из Kiban'ы**
>
> ```auto
> {
> "_index": "filebeat-7.4.1-2019.10.29-000001",
> "_type": "_doc",
> "_id": "VBa9F24Buro5YZJ3FUYe",
> "_version": 1,
> "_score": null,
> "_source": {
> "@timestamp": "2019-10-29T13:38:15.303Z",
> "log": {
> "offset": 2641843,
> "file": {
> "path": "/var/log/nsi-backend/nsi-backend.log"
> }
> },
> "json": {
> "@version": 1,
> "mdc": {
> "isFatal": "false",
> "username": ""
> },
> "source_host": "8dff15b81656",
> "method": "lambda$info$6",
> "class": "my.self.nsi.logging.Slf4JLoggingAPI",
> "file": "Slf4JLoggingAPI.java",
> "level": "INFO",
> "thread_name": "main",
> "message": "[Calling handlers after operation] Get table",
> "@timestamp": "2019-10-29T11:24:56.427Z",
> "line_number": "89",
> "logger_name": "my.self.nsi.base.events.EventTransactionAPI"
> },
> "input": {
> "type": "log"
> },
> "ecs": {
> "version": "1.1.0"
> },
> "host": {
> "name": "nsi-ci"
> },
> "agent": {
> "hostname": "nsi-ci",
> "id": "5616a0a9-b31b-420e-848c-4aaa6182e522",
> "version": "7.4.1",
> "type": "filebeat",
> "ephemeral_id": "cf25cfe8-4ab2-48e8-9461-0bb3eda9d821"
> }
> },
> "fields": {
> "suricata.eve.timestamp": [
> "2019-10-29T13:38:15.303Z"
> ],
> "@timestamp": [
> "2019-10-29T13:38:15.303Z"
> ]
> },
> "sort": [
> 1572356295303
> ]
> }
> 
> ```

Также протестировал `multiline`, логи стали писаться как желаемо весь кусок лога в одну запись в кибане. Но логи опять же не все.

> **filebeat.yml и JSON пример с multiline**
>
> ```auto
> filebeat.inputs:
> - type: log
> enabled: true
> paths: ['/var/log/nsi-backend/nsi-backend.log']
> json.message_key: message
> json.keys_under_root: false
> json.add_error_key: true
> multiline.pattern: '^{'
> multiline.negate: true
> multiline.match: after
> 
> ```
> 
> [Пример JSON лога](https://pastebin.com/gcbP08Qh)

Подскажите пожалуйста, что я делаю не так?  
Не знаю уже в какую строну копать, дабы логи были нормально читаемые в Kiban'e...

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [October 30, 2019, 2:34pm UTC](https://discuss.elastic.co/t/filebeat-cannot-index-event-publisher-event/205843/2 "2019-10-30T14:34:29Z")

</div>

На прошлой неделе это подробно обсуждали - [Ошибка : Could not index event to Elasticsearch](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/203630)

---

<div class="post-metadata">

**Author:** ![IwanHSKY](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iwanhsky/32/56594_2.png) [@IwanHSKY](https://discuss.elastic.co/u/IwanHSKY)\
**Post date:** [October 31, 2019, 6:03am UTC](https://discuss.elastic.co/t/filebeat-cannot-index-event-publisher-event/205843/3 "2019-10-31T06:03:23Z")

</div>

Игорь, спасибо за ссылку.  
Да я понимал, что поле file не такое как в шаблоне..  
Но решил проблему подругому.

Как в доке:

> **filebeat**
>
> ```auto
> filebeat.inputs:
> - type: log
> enabled: true
> paths: ['/var/log/nsi-backend/nsi-backend.log']
> json.keys_under_root: true
> 
> processors:
> - decode_json_fields:
> fields: ["exception"]
>       
> processors:
> - decode_json_fields:
> fields: ["stacktrace"]
>     
> output.logstash:
> hosts: ["172.22.130.64:5044"]
> 
> ```

А у logstash конфиг обычный:

> **logstash**
>
> ```auto
> input {
> beats {
> port => "5044"
> }
> }
>   
> output {
>     
> # Sending properly parsed log events to elasticsearch
> elasticsearch {
> hosts => ["http://172.22.130.64:9200"]
> user => "logstash_system"
> password => "${ELASTIC_PASSWORD}"
> index => "logstash-%{+yyyy-MM-dd}"
> }
> }
> 
> ```

Но у меня теперь есть баг похое с кабиной - [Видео пруф](https://imgur.com/VUNsqZH)  
Browser Firefox, в Chrome такой проблемы нет..

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [October 31, 2019, 3:01pm UTC](https://discuss.elastic.co/t/filebeat-cannot-index-event-publisher-event/205843/4 "2019-10-31T15:01:48Z")

</div>

> [@IwanHSKY](#):
>
> Но у меня теперь есть баг похое с кабиной

Поищите в [Issues · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues) и если там ничего подобного не упомянуто, откройте, пожалуйста, ишью.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 28, 2019, 3:01pm UTC](https://discuss.elastic.co/t/filebeat-cannot-index-event-publisher-event/205843/5 "2019-11-28T15:01:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.