Filebeat json parser is trying to parse excluded lines

filebeat log is filled with errors "Error decoding JSON: invalid character"

filebeat.yml

---
filebeat.inputs:
  - type: filestream
    id: 0
    paths:
      - '/data/mixed_json_plain.log'
    exclude_lines: [ "^PLAIN" ]
    parsers:
      - ndjson:
          add_error_key: true
          keys_under_root: true
          overwrite_keys: true
          message_key: message
          expand_keys: true

output.file:
  path: /data/filebeat_output
  rotate_every_kb: 100000
  rotate_on_startup: false
  codec.json:
    pretty: true

log sample file mixed_json_plain.log

{"id":1,"message":"lorem ipsum"}
PLAIN first line
PLAIN second line
{"id":2,"message":"dolor sit amet"}
{"id":3,"message":"consectetur adipiscing elit"}

To reproduce errors

docker run --rm -v $PWD:/data -v $PWD/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:8.2.0

When run with -e -d '*' it is visible on debug level, the line is exluded correctly.

Drop line as it does match one of the exclude patterns%sPLAIN first line
1 Like

@kvch can you have a look at this one please?

Could you please share the debug logs?

Hello, I already deleted my test scenario, but it took me like 2 minutes to recreate that from the info I provided earlier. See hidden part for full log. I run the container with -e -d '*'

The interesting part is

{"log.level":"error","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"parser_json","log.origin":{"file.name":"readjson/json.go","file.line":74},"message":"Error decoding JSON: invalid character 'P' looking for beginning of value","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":351},"message":"Drop line as it does match one of the exclude patterns%sPLAIN first line","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
filebeat_err_log_debug.json
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.971Z","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.971Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":743},"message":"Beat metadata path: /usr/share/filebeat/data/meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.972Z","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: fb7cd886-0353-41e6-a0de-e9345ccaed39","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.972Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":117},"message":"Loading syscall filter","service.name":"filebeat","seccomp_filter":{"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","clone3","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rseq","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"fb7cd886-0353-41e6-a0de-e9345ccaed39"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1072},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"045da3a1bb89944373c33332c18ca99ef6192df2","libbeat":"8.2.0","time":"2022-04-19T23:31:06.000Z","version":"8.2.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1075},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":12,"version":"go1.17.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1079},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-28T09:41:57Z","containerized":true,"name":"eec859e77752","ip":["127.0.0.1/8","192.168.123.2/24"],"kernel_version":"5.17.9-200.fc35.x86_64","mac":["02:42:c0:a8:7b:02"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/filebeat","exe":"/usr/share/filebeat/filebeat","name":"filebeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-05-31T15:20:47.680Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.974Z","log.origin":{"file.name":"instance/beat.go","file.line":325},"message":"Setup Beat: filebeat; Version: 8.2.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":353},"message":"Initializing output plugins","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"rotator","log.origin":{"file.name":"file/rotator.go","file.line":181},"message":"Initialized file rotator","service.name":"filebeat","rotator":{"filename":"","max_size_bytes":102400000,"max_backups":7,"permissions":"-rw-------","ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"file","log.origin":{"file.name":"fileout/file.go","file.line":102},"message":"Initialized file output. path=/data/filebeat_output/filebeat max_size_bytes=102400000 max_backups=7 permissions=-rw-------","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":98},"message":"start pipeline event consumer","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: eec859e77752","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/queue_reader.go","file.line":48},"message":"pipeline event consumer queue reader: start","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-31T15:20:47.974Z","log.origin":{"file.name":"beater/filebeat.go","file.line":166},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.975Z","log.origin":{"file.name":"instance/beat.go","file.line":505},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.975Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":289},"message":"isFile(/usr/share/filebeat/data/registry) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.975Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":289},"message":"isFile() -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.975Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":282},"message":"isDir(/usr/share/filebeat/data/registry/filebeat) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.975Z","log.logger":"registrar","log.origin":{"file.name":"registrar/migrate.go","file.line":84},"message":"Registry type '' found","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.975Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":289},"message":"isFile(.bak) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-31T15:20:47.976Z","log.origin":{"file.name":"beater/filebeat.go","file.line":283},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":82},"message":"registering encoder 'application/json': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":85},"message":"registering encoder 'application/x-www-form-urlencoded': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":91},"message":"registering decoder 'application/json': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":94},"message":"registering decoder 'application/x-ndjson': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":97},"message":"registering decoder 'text/csv': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.exclude_lines.0 filebeat.inputs.0.id filebeat.inputs.0.parsers.0.ndjson.add_error_key filebeat.inputs.0.parsers.0.ndjson.expand_keys filebeat.inputs.0.parsers.0.ndjson.keys_under_root filebeat.inputs.0.parsers.0.ndjson.message_key filebeat.inputs.0.parsers.0.ndjson.overwrite_keys filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":140},"message":"Starting Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":281},"message":"recursive glob enabled","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 16219457660125822758)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":111},"message":"Input filestream starting","service.name":"filebeat","id":"0","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":131},"message":"Starting prospector","service.name":"filebeat","id":"0","prospector":"file_prospector","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":138},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":208},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":177},"message":"A new file /data/mixed_json_plain.log has been found","service.name":"filebeat","id":"0","prospector":"file_prospector","operation":"create","source_name":"native::7639481-54","os_id":"7639481-54","new_path":"/data/mixed_json_plain.log","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":145},"message":"Starting harvester for file","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":174},"message":"newLogFileReader with config.MaxBytes:10485760","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"detect_null_bytes","log.origin":{"file.name":"debug/debug.go","file.line":95},"message":"Starting debug reader with a buffer size of 16384 and max failures of 100","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":209},"message":"Publish event: {\n  \"@timestamp\": \"2022-05-31T15:20:47.977Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.2.0\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"log\": {\n    \"offset\": 0,\n    \"file\": {\n      \"path\": \"/data/mixed_json_plain.log\"\n    }\n  },\n  \"id\": 1,\n  \"message\": \"lorem ipsum\",\n  \"input\": {\n    \"type\": \"filestream\"\n  },\n  \"host\": {\n    \"name\": \"eec859e77752\"\n  },\n  \"agent\": {\n    \"id\": \"fb7cd886-0353-41e6-a0de-e9345ccaed39\",\n    \"name\": \"eec859e77752\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.2.0\",\n    \"ephemeral_id\": \"519d82fa-20b1-46c3-b8f6-3f32a956b132\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"parser_json","log.origin":{"file.name":"readjson/json.go","file.line":74},"message":"Error decoding JSON: invalid character 'P' looking for beginning of value","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":351},"message":"Drop line as it does match one of the exclude patterns%sPLAIN first line","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"parser_json","log.origin":{"file.name":"readjson/json.go","file.line":74},"message":"Error decoding JSON: invalid character 'P' looking for beginning of value","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":351},"message":"Drop line as it does match one of the exclude patterns%sPLAIN second line","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":209},"message":"Publish event: {\n  \"@timestamp\": \"2022-05-31T15:20:47.977Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.2.0\"\n  },\n  \"id\": 2,\n  \"message\": \"dolor sit amet\",\n  \"log\": {\n    \"offset\": 68,\n    \"file\": {\n      \"path\": \"/data/mixed_json_plain.log\"\n    }\n  },\n  \"input\": {\n    \"type\": \"filestream\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"host\": {\n    \"name\": \"eec859e77752\"\n  },\n  \"agent\": {\n    \"id\": \"fb7cd886-0353-41e6-a0de-e9345ccaed39\",\n    \"name\": \"eec859e77752\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.2.0\",\n    \"ephemeral_id\": \"519d82fa-20b1-46c3-b8f6-3f32a956b132\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":209},"message":"Publish event: {\n  \"@timestamp\": \"2022-05-31T15:20:47.977Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.2.0\"\n  },\n  \"host\": {\n    \"name\": \"eec859e77752\"\n  },\n  \"log\": {\n    \"offset\": 104,\n    \"file\": {\n      \"path\": \"/data/mixed_json_plain.log\"\n    }\n  },\n  \"id\": 3,\n  \"message\": \"consectetur adipiscing elit\",\n  \"input\": {\n    \"type\": \"filestream\"\n  },\n  \"agent\": {\n    \"version\": \"8.2.0\",\n    \"ephemeral_id\": \"519d82fa-20b1-46c3-b8f6-3f32a956b132\",\n    \"id\": \"fb7cd886-0353-41e6-a0de-e9345ccaed39\",\n    \"name\": \"eec859e77752\",\n    \"type\": \"filebeat\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /data/mixed_json_plain.log; Backoff now.","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:48.985Z","log.logger":"publisher","log.origin":{"file.name":"memqueue/ackloop.go","file.line":160},"message":"ackloop: receive ack [0: 0, 3]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:48.985Z","log.logger":"publisher","log.origin":{"file.name":"memqueue/eventloop.go","file.line":535},"message":"broker ACK events: count=3, start-seq=1, end-seq=3\n","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:48.986Z","log.logger":"acker","log.origin":{"file.name":"beater/acker.go","file.line":64},"message":"stateless ack","service.name":"filebeat","count":3,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:48.986Z","log.logger":"publisher","log.origin":{"file.name":"memqueue/ackloop.go","file.line":128},"message":"ackloop: return ack to broker loop:3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:48.986Z","log.logger":"publisher","log.origin":{"file.name":"memqueue/ackloop.go","file.line":131},"message":"ackloop:  done send ack","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:49.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /data/mixed_json_plain.log; Backoff now.","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:53.978Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /data/mixed_json_plain.log; Backoff now.","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:20:57.978Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":138},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:57.978Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":208},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:21:01.983Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /data/mixed_json_plain.log; Backoff now.","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-31T15:21:07.977Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":138},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:21:07.978Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":208},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}

Indeed, could you please open an issue on Github? Sign in to GitHub · GitHub

Here you are @kvch

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.