# Filebeat json parser is trying to parse excluded lines

**URL:** <https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [May 13, 2022, 11:52am UTC](https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674 "2022-05-13T11:52:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![dosmanak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dosmanak/32/105639_2.png) [@dosmanak](https://discuss.elastic.co/u/dosmanak)\
**Post date:** [May 13, 2022, 11:52am UTC](https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674/1 "2022-05-13T11:52:06Z")

</div>

filebeat log is filled with errors "Error decoding JSON: invalid character"

filebeat.yml

```auto
---
filebeat.inputs:
  - type: filestream
    id: 0
    paths:
      - '/data/mixed_json_plain.log'
    exclude_lines: ["^PLAIN"]
    parsers:
      - ndjson:
          add_error_key: true
          keys_under_root: true
          overwrite_keys: true
          message_key: message
          expand_keys: true

output.file:
  path: /data/filebeat_output
  rotate_every_kb: 100000
  rotate_on_startup: false
  codec.json:
    pretty: true

```

log sample file mixed\_json\_plain.log

```auto
{"id":1,"message":"lorem ipsum"}
PLAIN first line
PLAIN second line
{"id":2,"message":"dolor sit amet"}
{"id":3,"message":"consectetur adipiscing elit"}

```

To reproduce errors

```auto
docker run --rm -v $PWD:/data -v $PWD/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:8.2.0

```

When run with `-e -d '*'` it is visible on debug level, the line is exluded correctly.

```auto
Drop line as it does match one of the exclude patterns%sPLAIN first line

```

---

<div class="post-metadata">

**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\
**Post date:** [May 26, 2022, 5:34am UTC](https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674/2 "2022-05-26T05:34:15Z")

</div>

@kvch can you have a look at this one please?

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [May 26, 2022, 3:58pm UTC](https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674/3 "2022-05-26T15:58:51Z")

</div>

Could you please share the debug logs?

---

<div class="post-metadata">

**Author:** ![dosmanak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dosmanak/32/105639_2.png) [@dosmanak](https://discuss.elastic.co/u/dosmanak)\
**Post date:** [May 31, 2022, 3:24pm UTC](https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674/4 "2022-05-31T15:24:42Z")

</div>

Hello, I already deleted my test scenario, but it took me like 2 minutes to recreate that from the info I provided earlier. See hidden part for full log. I run the container with `-e -d '*'`

The interesting part is

```auto
{"log.level":"error","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"parser_json","log.origin":{"file.name":"readjson/json.go","file.line":74},"message":"Error decoding JSON: invalid character 'P' looking for beginning of value","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":351},"message":"Drop line as it does match one of the exclude patterns%sPLAIN first line","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}

```

> **filebeat\_err\_log\_debug.json**
>
> ```auto
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.971Z","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.971Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":743},"message":"Beat metadata path: /usr/share/filebeat/data/meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.972Z","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: fb7cd886-0353-41e6-a0de-e9345ccaed39","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.972Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":117},"message":"Loading syscall filter","service.name":"filebeat","seccomp_filter":{"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","clone3","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rseq","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}},"ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"fb7cd886-0353-41e6-a0de-e9345ccaed39"},"ecs.version":"1.6.0"}}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1072},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"045da3a1bb89944373c33332c18ca99ef6192df2","libbeat":"8.2.0","time":"2022-04-19T23:31:06.000Z","version":"8.2.0"},"ecs.version":"1.6.0"}}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1075},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":12,"version":"go1.17.8"},"ecs.version":"1.6.0"}}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1079},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-28T09:41:57Z","containerized":true,"name":"eec859e77752","ip":["127.0.0.1/8","192.168.123.2/24"],"kernel_version":"5.17.9-200.fc35.x86_64","mac":["02:42:c0:a8:7b:02"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.973Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/filebeat","exe":"/usr/share/filebeat/filebeat","name":"filebeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-05-31T15:20:47.680Z"},"ecs.version":"1.6.0"}}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.974Z","log.origin":{"file.name":"instance/beat.go","file.line":325},"message":"Setup Beat: filebeat; Version: 8.2.0","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":353},"message":"Initializing output plugins","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"rotator","log.origin":{"file.name":"file/rotator.go","file.line":181},"message":"Initialized file rotator","service.name":"filebeat","rotator":{"filename":"","max_size_bytes":102400000,"max_backups":7,"permissions":"-rw-------","ecs.version":"1.6.0"}}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"file","log.origin":{"file.name":"fileout/file.go","file.line":102},"message":"Initialized file output. path=/data/filebeat_output/filebeat max_size_bytes=102400000 max_backups=7 permissions=-rw-------","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":98},"message":"start pipeline event consumer","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: eec859e77752","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/queue_reader.go","file.line":48},"message":"pipeline event consumer queue reader: start","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"warn","@timestamp":"2022-05-31T15:20:47.974Z","log.origin":{"file.name":"beater/filebeat.go","file.line":166},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.974Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.975Z","log.origin":{"file.name":"instance/beat.go","file.line":505},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.975Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":289},"message":"isFile(/usr/share/filebeat/data/registry) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.975Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":289},"message":"isFile() -> false","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.975Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":282},"message":"isDir(/usr/share/filebeat/data/registry/filebeat) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.975Z","log.logger":"registrar","log.origin":{"file.name":"registrar/migrate.go","file.line":84},"message":"Registry type '' found","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.975Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":289},"message":"isFile(.bak) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"warn","@timestamp":"2022-05-31T15:20:47.976Z","log.origin":{"file.name":"beater/filebeat.go","file.line":283},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:append","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:delete","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:set","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:append","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:delete","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:set","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:append","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:delete","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:set","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":82},"message":"registering encoder 'application/json': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":85},"message":"registering encoder 'application/x-www-form-urlencoded': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":91},"message":"registering decoder 'application/json': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":94},"message":"registering decoder 'application/x-ndjson': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":97},"message":"registering decoder 'text/csv': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.exclude_lines.0 filebeat.inputs.0.id filebeat.inputs.0.parsers.0.ndjson.add_error_key filebeat.inputs.0.parsers.0.ndjson.expand_keys filebeat.inputs.0.parsers.0.ndjson.keys_under_root filebeat.inputs.0.parsers.0.ndjson.message_key filebeat.inputs.0.parsers.0.ndjson.overwrite_keys filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":140},"message":"Starting Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":281},"message":"recursive glob enabled","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 16219457660125822758)","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":111},"message":"Input filestream starting","service.name":"filebeat","id":"0","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":131},"message":"Starting prospector","service.name":"filebeat","id":"0","prospector":"file_prospector","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":138},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.976Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":208},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":177},"message":"A new file /data/mixed_json_plain.log has been found","service.name":"filebeat","id":"0","prospector":"file_prospector","operation":"create","source_name":"native::7639481-54","os_id":"7639481-54","new_path":"/data/mixed_json_plain.log","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":145},"message":"Starting harvester for file","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":174},"message":"newLogFileReader with config.MaxBytes:10485760","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"detect_null_bytes","log.origin":{"file.name":"debug/debug.go","file.line":95},"message":"Starting debug reader with a buffer size of 16384 and max failures of 100","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":209},"message":"Publish event: {\n \"@timestamp\": \"2022-05-31T15:20:47.977Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.2.0\"\n },\n \"ecs\": {\n \"version\": \"8.0.0\"\n },\n \"log\": {\n \"offset\": 0,\n \"file\": {\n \"path\": \"/data/mixed_json_plain.log\"\n }\n },\n \"id\": 1,\n \"message\": \"lorem ipsum\",\n \"input\": {\n \"type\": \"filestream\"\n },\n \"host\": {\n \"name\": \"eec859e77752\"\n },\n \"agent\": {\n \"id\": \"fb7cd886-0353-41e6-a0de-e9345ccaed39\",\n \"name\": \"eec859e77752\",\n \"type\": \"filebeat\",\n \"version\": \"8.2.0\",\n \"ephemeral_id\": \"519d82fa-20b1-46c3-b8f6-3f32a956b132\"\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"error","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"parser_json","log.origin":{"file.name":"readjson/json.go","file.line":74},"message":"Error decoding JSON: invalid character 'P' looking for beginning of value","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":351},"message":"Drop line as it does match one of the exclude patterns%sPLAIN first line","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
> {"log.level":"error","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"parser_json","log.origin":{"file.name":"readjson/json.go","file.line":74},"message":"Error decoding JSON: invalid character 'P' looking for beginning of value","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":351},"message":"Drop line as it does match one of the exclude patterns%sPLAIN second line","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":209},"message":"Publish event: {\n \"@timestamp\": \"2022-05-31T15:20:47.977Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.2.0\"\n },\n \"id\": 2,\n \"message\": \"dolor sit amet\",\n \"log\": {\n \"offset\": 68,\n \"file\": {\n \"path\": \"/data/mixed_json_plain.log\"\n }\n },\n \"input\": {\n \"type\": \"filestream\"\n },\n \"ecs\": {\n \"version\": \"8.0.0\"\n },\n \"host\": {\n \"name\": \"eec859e77752\"\n },\n \"agent\": {\n \"id\": \"fb7cd886-0353-41e6-a0de-e9345ccaed39\",\n \"name\": \"eec859e77752\",\n \"type\": \"filebeat\",\n \"version\": \"8.2.0\",\n \"ephemeral_id\": \"519d82fa-20b1-46c3-b8f6-3f32a956b132\"\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":209},"message":"Publish event: {\n \"@timestamp\": \"2022-05-31T15:20:47.977Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.2.0\"\n },\n \"host\": {\n \"name\": \"eec859e77752\"\n },\n \"log\": {\n \"offset\": 104,\n \"file\": {\n \"path\": \"/data/mixed_json_plain.log\"\n }\n },\n \"id\": 3,\n \"message\": \"consectetur adipiscing elit\",\n \"input\": {\n \"type\": \"filestream\"\n },\n \"agent\": {\n \"version\": \"8.2.0\",\n \"ephemeral_id\": \"519d82fa-20b1-46c3-b8f6-3f32a956b132\",\n \"id\": \"fb7cd886-0353-41e6-a0de-e9345ccaed39\",\n \"name\": \"eec859e77752\",\n \"type\": \"filebeat\"\n },\n \"ecs\": {\n \"version\": \"8.0.0\"\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /data/mixed_json_plain.log; Backoff now.","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:48.985Z","log.logger":"publisher","log.origin":{"file.name":"memqueue/ackloop.go","file.line":160},"message":"ackloop: receive ack [0: 0, 3]","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:48.985Z","log.logger":"publisher","log.origin":{"file.name":"memqueue/eventloop.go","file.line":535},"message":"broker ACK events: count=3, start-seq=1, end-seq=3\n","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:48.986Z","log.logger":"acker","log.origin":{"file.name":"beater/acker.go","file.line":64},"message":"stateless ack","service.name":"filebeat","count":3,"ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:48.986Z","log.logger":"publisher","log.origin":{"file.name":"memqueue/ackloop.go","file.line":128},"message":"ackloop: return ack to broker loop:3","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:48.986Z","log.logger":"publisher","log.origin":{"file.name":"memqueue/ackloop.go","file.line":131},"message":"ackloop: done send ack","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:49.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /data/mixed_json_plain.log; Backoff now.","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:53.978Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /data/mixed_json_plain.log; Backoff now.","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:20:57.978Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":138},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:57.978Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":208},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:21:01.983Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: /data/mixed_json_plain.log; Backoff now.","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2022-05-31T15:21:07.977Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":138},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:21:07.978Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":208},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
> 
> ```

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [May 31, 2022, 4:57pm UTC](https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674/5 "2022-05-31T16:57:53Z")

</div>

Indeed, could you please open an issue on Github? [Sign in to GitHub · GitHub](https://github.com/elastic/beats/issues/new)

---

<div class="post-metadata">

**Author:** ![dosmanak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dosmanak/32/105639_2.png) [@dosmanak](https://discuss.elastic.co/u/dosmanak)\
**Post date:** [June 1, 2022, 8:56am UTC](https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674/6 "2022-06-01T08:56:07Z")

</div>

Here you are @kvch

> <https://github.com/elastic/beats/issues/31794>
>
> \- Version: 8.2.0
> \- Operating System: Irrelevant / Docker on Linux
> \- Discuss Fo…rum URL: https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674
> 
> filebeat log is filled with errors "Error decoding JSON: invalid character"
> 
> \## Steps to Reproduce: 
> \`filebeat.yml\`
> \`\`\`
> \---
> filebeat.inputs:
> - type: filestream
> id: 0
> paths:
> - '/data/mixed\_json\_plain.log'
> exclude\_lines: \["^PLAIN" \]
> parsers:
> - ndjson:
> add\_error\_key: true
> keys\_under\_root: true
> overwrite\_keys: true
> message\_key: message
> expand\_keys: true
> 
> output.file:
> path: /data/filebeat\_output
> rotate\_every\_kb: 100000
> rotate\_on\_startup: false
> codec.json:
> pretty: true
> \`\`\`
> log sample file \`mixed\_json\_plain.log\`
> \`\`\`
> {"id":1,"message":"lorem ipsum"}
> PLAIN first line
> PLAIN second line
> {"id":2,"message":"dolor sit amet"}
> {"id":3,"message":"consectetur adipiscing elit"}
> \`\`\`
> To reproduce errors
> \`\`\`
> docker run --rm -v $PWD:/data -v $PWD/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:8.2.0
> \`\`\`
> When run with -e -d '\*' it is visible on debug level, the line is exluded correctly.
> \> Drop line as it does match one of the exclude patterns%sPLAIN first line
> 
> Full debug log is in linked \[Discuss Forum URL\](https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674
> )
> Interresting lines are
> \`\`\`
> {"log.level":"error","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"parser\_json","log.origin":{"file.name":"readjson/json.go","file.line":74},"message":"Error decoding JSON: invalid character 'P' looking for beginning of value","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":351},"message":"Drop line as it does match one of the exclude patterns%sPLAIN first line","service.name":"filebeat","id":"0","source\_file":"filestream::0::native::7639481-54","path":"/data/mixed\_json\_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}
> \`\`\`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 29, 2022, 10:56am UTC](https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674/7 "2022-06-29T10:56:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.