# Filebeat - "Cannot index event publisher"
**URL:** https://discuss.elastic.co/t/filebeat-cannot-index-event-publisher/205793
**Category:** Beats
**Tags:** docker, filebeat
**Created:** [October 30, 2019, 5:31am UTC](https://discuss.elastic.co/t/filebeat-cannot-index-event-publisher/205793 "2019-10-30T05:31:39Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@IwanHSKY](https://discuss.elastic.co/u/IwanHSKY)
#### Post date: [October 30, 2019, 5:31am UTC](https://discuss.elastic.co/t/filebeat-cannot-index-event-publisher/205793/1 "2019-10-30T05:31:40Z")
Hello everyone, I use ELK Stack in Docker.
I try send logs from Java app in Docker.
I Tested 7.4.0, and 7.4.1 versions.
- Send logs with filebeat to ES;
Config Filebeat for ES
> **Filebeat.yml**
>
> ```auto
> # Modules configuration ==
> filebeat.modules:
>
> #------------------------------- System Module -------------------------------
> - module: system
> # Syslog
> syslog:
> enabled: true
> var.paths: ["/var/log/syslog*", "/var/log/messages*"]
> auth:
> enabled: true
> var.paths: ["/var/log/auth.log*", "/var/log/secure*"]
>
> #------------------------------- Auditd Module -------------------------------
> # Does not look like Auditd is supported in Alpine linux:
> # https://github.com/linuxkit/linuxkit/issues/52
> - module: auditd
> log:
> enabled: false
>
> # Template
> setup.template.enabled: true
> setup.template.name: "filebeat"
> setup.template.pattern: "filebeat-${INDEX_NAME:default}-%{[beat.version]}-*"
>
> # Filebeat inputs
>
> filebeat.inputs:
> - type: log
> enabled: true
> paths: ['/var/log/nsi-backend/nsi-backend.log']
> json.message_key: message
> json.keys_under_root: true
> json.add_error_key: true
>
> # Elasticsearch output
> output.elasticsearch:
> hosts: ["${ELASTICSEARCH_HOST:http://172.22.130.64:9200}"]
> index: "filebeat-${INDEX_NAME:default}-%{[beat.version]}-%{+yyyy.MM.dd}"
> username: elastic
> password: ${ELASTIC_PASSWORD}
>
> monitoring:
> enabled: true
> elasticsearch:
>
> # Dashboards
> setup.dashboards:
> enabled: true
> index: "filebeat-${INDEX_NAME:default}-*"
> setup.kibana:
> host: "${KIBANA_HOST:http://172.22.130.64:5601}"
> username: elastic
> password: ${ELASTIC_PASSWORD}
>
> ```
Java logs output with log4j
> **JSON log and log4j.properties**
>
> [Example JSON log](https://pastebin.com/MCWmkkqG)
>
> ```auto
> # Root logger option
> log4j.rootLogger=INFO, json
> # JSON log
> log4j.appender.json=org.apache.log4j.DailyRollingFileAppender
> log4j.appender.json.File=/var/log/nsi-backend/nsi-backend.log
> log4j.appender.json.DatePattern=.yyyy-MM-dd
> log4j.appender.json.layout=net.logstash.log4j.JSONEventLayoutV1
>
> ```
With the first config I get the following errors:
> **Filebeat debug log**
>
> [Filebeat debug log](https://pastebin.com/wC6jMgwn)
Error is - `elasticsearch/client.go:535 Cannot index event publisher.Event`:
> **Error**
>
> `2019-10-29T11:40:42.629Z WARN elasticsearch/client.go:535 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf66272db0a2e777, ext:48849912601, loc:(*time.Location)(0x4de6580)}, Meta:common.MapStr(nil), Fields:common.MapStr{"@version":1, "agent":common.MapStr{"ephemeral_id":"8bc91806-bd18-4ab3-bfa3-a9a987e94b78", "hostname":"nsi-ci", "id":"a5a65b6e-7343-4af6-84d5-618332ab615b", "type":"filebeat", "version":"7.4.1"}, "class":"my.self.nsi.logging.Slf4JLoggingAPI", "ecs":common.MapStr{"version":"1.1.0"}, "file":"Slf4JLoggingAPI.java", "host":common.MapStr{"name":"nsi-ci"}, "input":common.MapStr{"type":"log"}, "level":"INFO", "line_number":"89", "log":common.MapStr{"file":common.MapStr{"path":"/var/log/nsi-backend/nsi-backend.log"}, "offset":2831236}, "logger_name":"my.self.nsi.base.events.EventTransactionAPI", "mdc":common.MapStr{"isFatal":"false", "username":""}, "message":"[Start] Get table", "method":"lambda$info$6", "source_host":"8dff15b81656", "thread_name":"main"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000c9a1a0), Source:"/var/log/nsi-backend/nsi-backend.log", Offset:2831606, Timestamp:time.Time{wall:0xbf66272dac59c927, ext:48778011857, loc:(*time.Location)(0x4de6580)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x4000e, Device:0xfd00}}, TimeSeries:false}, Flags:0x1}`
>
> ` (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [file] tried to parse field [file] as object, but found a concrete value"}`
If set to false - `json.keys_under_root: false`
Logs sends, but not all, why stacktrace not sends?
> **Example JSON log from Kibana**
>
> ```auto
> {
> "_index": "filebeat-7.4.1-2019.10.29-000001",
> "_type": "_doc",
> "_id": "VBa9F24Buro5YZJ3FUYe",
> "_version": 1,
> "_score": null,
> "_source": {
> "@timestamp": "2019-10-29T13:38:15.303Z",
> "log": {
> "offset": 2641843,
> "file": {
> "path": "/var/log/nsi-backend/nsi-backend.log"
> }
> },
> "json": {
> "@version": 1,
> "mdc": {
> "isFatal": "false",
> "username": ""
> },
> "source_host": "8dff15b81656",
> "method": "lambda$info$6",
> "class": "my.self.nsi.logging.Slf4JLoggingAPI",
> "file": "Slf4JLoggingAPI.java",
> "level": "INFO",
> "thread_name": "main",
> "message": "[Calling handlers after operation] Get table",
> "@timestamp": "2019-10-29T11:24:56.427Z",
> "line_number": "89",
> "logger_name": "my.self.nsi.base.events.EventTransactionAPI"
> },
> "input": {
> "type": "log"
> },
> "ecs": {
> "version": "1.1.0"
> },
> "host": {
> "name": "nsi-ci"
> },
> "agent": {
> "hostname": "nsi-ci",
> "id": "5616a0a9-b31b-420e-848c-4aaa6182e522",
> "version": "7.4.1",
> "type": "filebeat",
> "ephemeral_id": "cf25cfe8-4ab2-48e8-9461-0bb3eda9d821"
> }
> },
> "fields": {
> "suricata.eve.timestamp": [
> "2019-10-29T13:38:15.303Z"
> ],
> "@timestamp": [
> "2019-10-29T13:38:15.303Z"
> ]
> },
> "sort": [
> 1572356295303
> ]
> }
>
> ```
And tested multiline:
> **filebeat.yml and JSON example**
>
> ```auto
> filebeat.inputs:
> - type: log
> enabled: true
> paths: ['/var/log/nsi-backend/nsi-backend.log']
> json.message_key: message
> json.keys_under_root: false
> json.add_error_key: true
> multiline.pattern: '^{'
> multiline.negate: true
> multiline.match: after
>
> ```
>
> [Example JSON log](https://pastebin.com/gcbP08Qh)
Logs sends, again not all..
Any help and ideas please..
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 27, 2019, 5:31am UTC](https://discuss.elastic.co/t/filebeat-cannot-index-event-publisher/205793/2 "2019-11-27T05:31:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.