Failed to expand fields: cannot expand "field" found conflicting key

[nobeerhere (Lukas)]

hi there

I am using ECS logging for Java and so far it worked fine. I now have the issue that the log structure changed a bit (field trace.id & transation.id are new) and i am having a conflicting key value. This also makes sense to me but i am cant figure out how i can fix this issue.

JSON-Log:

{
	"@timestamp": "2023-03-03T00:34:57.520Z",
	"log.level": "INFO",
	"message": "some message",
	"ecs.version": "1.2.0",
	"service.name": "srv-core",
	"event.dataset": "srv-core",
	"process.thread.name": "http-nio-14002-exec-4",
	"log.logger": "some logger",
	"path": "some path",
	"trace": "izlev4",
	"trace.id": "4e57fa5f3d47e9b54f8a17d1b83d92cd",
	"transaction.id": "99cb6deeb2cf431a",
	"ip": "ip",
	"user": "email",
	"tenant": "test-ag1"
}

As far as i understand the issue we do have two rook fields "trace" which differ in their type.

Filebeat 8.3.3 config:

- type: filestream

  id: filestream

  enabled: true

  paths:
    - log.json

  parsers:
    - ndjson:
        target: ""
        overwrite_keys: true
        add_error_key: true
        expand_keys: true

i tried to drop the field trace, but this is somehow not working properly. What options do i have?

Thanks for helping out.

[nobeerhere (Lukas)]

In the meantime i have found the reason that changed the log structure. We also upgraded to APM Agent 1.35.0 and this agent created these two new fields.

If i downgrade to the older version the new fields disappear and the log ingesting is working again.

But maybe someone still has a hint on how i could fix my issue.

[stephenb (Stephen Brown)]

@nobeerhere

As far as i understand the issue we do have two rook fields "trace" which differ in their type.

Yup that is right this is in valid

	"trace": "izlev4",
	"trace.id": "4e57fa5f3d47e9b54f8a17d1b83d92cd",
	"transaction.id": "99cb6deeb2cf431a",

@nobeerhere

i tried to drop the field trace,

How? Show the code....

@nobeerhere

but this is somehow not working properly. What options do i have?

What is the error message... you probably have a mapping where trace is a keyword... so trace.id which is an object with a field will fail with a mapping error...

You may need to create a new index so the proper mapping can be applied.

Show the code... show the error perhaps we can help.