Filebeat - "Cannot index event publisher"

[IwanHSKY (Iwan)]

Hello everyone, I use ELK Stack in Docker.
I try send logs from Java app in Docker.
I Tested 7.4.0, and 7.4.1 versions.

• Send logs with filebeat to ES;

Config Filebeat for ES

Filebeat.yml

#  Modules configuration ==
filebeat.modules:

#------------------------------- System Module -------------------------------
- module: system
  # Syslog
  syslog:
    enabled: true
    var.paths: ["/var/log/syslog*", "/var/log/messages*"]
  auth:
    enabled: true
    var.paths: ["/var/log/auth.log*", "/var/log/secure*"]

#------------------------------- Auditd Module -------------------------------
# Does not look like Auditd is supported in Alpine linux:
# https://github.com/linuxkit/linuxkit/issues/52
- module: auditd
  log:
    enabled: false

# Template 
setup.template.enabled: true
setup.template.name: "filebeat"
setup.template.pattern: "filebeat-${INDEX_NAME:default}-%{[beat.version]}-*"

# Filebeat inputs 

filebeat.inputs:
- type: log
  enabled: true
  paths: ['/var/log/nsi-backend/nsi-backend.log']
  json.message_key: message
  json.keys_under_root: true
  json.add_error_key: true

# Elasticsearch output
output.elasticsearch:
  hosts: ["${ELASTICSEARCH_HOST:http://172.22.130.64:9200}"]
  index: "filebeat-${INDEX_NAME:default}-%{[beat.version]}-%{+yyyy.MM.dd}"
  username: elastic
  password: ${ELASTIC_PASSWORD}

monitoring:
  enabled: true
  elasticsearch:

# Dashboards
setup.dashboards:
  enabled: true
  index: "filebeat-${INDEX_NAME:default}-*"
setup.kibana:
  host: "${KIBANA_HOST:http://172.22.130.64:5601}"
  username: elastic
  password: ${ELASTIC_PASSWORD}

Java logs output with log4j

JSON log and log4j.properties

Example JSON log

# Root logger option
log4j.rootLogger=INFO, json
# JSON log
log4j.appender.json=org.apache.log4j.DailyRollingFileAppender
log4j.appender.json.File=/var/log/nsi-backend/nsi-backend.log
log4j.appender.json.DatePattern=.yyyy-MM-dd
log4j.appender.json.layout=net.logstash.log4j.JSONEventLayoutV1

With the first config I get the following errors:

Filebeat debug log

Filebeat debug log

Error is - elasticsearch/client.go:535 Cannot index event publisher.Event:

Error

2019-10-29T11:40:42.629Z WARN elasticsearch/client.go:535 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf66272db0a2e777, ext:48849912601, loc:(*time.Location)(0x4de6580)}, Meta:common.MapStr(nil), Fields:common.MapStr{"@version":1, "agent":common.MapStr{"ephemeral_id":"8bc91806-bd18-4ab3-bfa3-a9a987e94b78", "hostname":"nsi-ci", "id":"a5a65b6e-7343-4af6-84d5-618332ab615b", "type":"filebeat", "version":"7.4.1"}, "class":"my.self.nsi.logging.Slf4JLoggingAPI", "ecs":common.MapStr{"version":"1.1.0"}, "file":"Slf4JLoggingAPI.java", "host":common.MapStr{"name":"nsi-ci"}, "input":common.MapStr{"type":"log"}, "level":"INFO", "line_number":"89", "log":common.MapStr{"file":common.MapStr{"path":"/var/log/nsi-backend/nsi-backend.log"}, "offset":2831236}, "logger_name":"my.self.nsi.base.events.EventTransactionAPI", "mdc":common.MapStr{"isFatal":"false", "username":""}, "message":"[Start] Get table", "method":"lambda$info$6", "source_host":"8dff15b81656", "thread_name":"main"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000c9a1a0), Source:"/var/log/nsi-backend/nsi-backend.log", Offset:2831606, Timestamp:time.Time{wall:0xbf66272dac59c927, ext:48778011857, loc:(*time.Location)(0x4de6580)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x4000e, Device:0xfd00}}, TimeSeries:false}, Flags:0x1}

(status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [file] tried to parse field [file] as object, but found a concrete value"}

If set to false - json.keys_under_root: false
Logs sends, but not all, why stacktrace not sends?

Example JSON log from Kibana

{
  "_index": "filebeat-7.4.1-2019.10.29-000001",
  "_type": "_doc",
  "_id": "VBa9F24Buro5YZJ3FUYe",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-10-29T13:38:15.303Z",
    "log": {
      "offset": 2641843,
      "file": {
        "path": "/var/log/nsi-backend/nsi-backend.log"
      }
    },
    "json": {
      "@version": 1,
      "mdc": {
        "isFatal": "false",
        "username": ""
      },
      "source_host": "8dff15b81656",
      "method": "lambda$info$6",
      "class": "my.self.nsi.logging.Slf4JLoggingAPI",
      "file": "Slf4JLoggingAPI.java",
      "level": "INFO",
      "thread_name": "main",
      "message": "[Calling handlers after operation] Get table",
      "@timestamp": "2019-10-29T11:24:56.427Z",
      "line_number": "89",
      "logger_name": "my.self.nsi.base.events.EventTransactionAPI"
    },
    "input": {
      "type": "log"
    },
    "ecs": {
      "version": "1.1.0"
    },
    "host": {
      "name": "nsi-ci"
    },
    "agent": {
      "hostname": "nsi-ci",
      "id": "5616a0a9-b31b-420e-848c-4aaa6182e522",
      "version": "7.4.1",
      "type": "filebeat",
      "ephemeral_id": "cf25cfe8-4ab2-48e8-9461-0bb3eda9d821"
    }
  },
  "fields": {
    "suricata.eve.timestamp": [
      "2019-10-29T13:38:15.303Z"
    ],
    "@timestamp": [
      "2019-10-29T13:38:15.303Z"
    ]
  },
  "sort": [
    1572356295303
  ]
}

And tested multiline:

filebeat.yml and JSON example

filebeat.inputs:
- type: log
  enabled: true
  paths: ['/var/log/nsi-backend/nsi-backend.log']
  json.message_key: message
  json.keys_under_root: false
  json.add_error_key: true
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match: after

Example JSON log

Logs sends, again not all..

Any help and ideas please..