Testing Custom Log Ingestion Issue: ECS Field

[erikg]

Hello,

I am trying to test out parsing some new logs via Custom File stream Integration. I did an initial upload and then worked backwards by creating the ingest pipeline using the sample documents from the initial upload.

I had made a mistake with ecs fields. I had parsed out the fields for

system.network.in.bytes
system.network.out.bytes

as keywords:

I made the mistake of not converting them. So, I added a convert processor to change them into numeric fields.

The issue is my data view doesn't have a mapping conflict error. I assumed there would be one since it was first a keyword and now a numeric field. It appears as if the convert processor is not working or perhaps, I am not understanding how this works. The goal is to make the ecs fields as expected numeric fields.

[stephenb (Stephen Brown)]

Did you rollover the data stream? Otherwise the new template won't take effect.

Then why don't you share the mappings from the new backing index to check?

And you really shouldn't need a convert processor unless the data was oddly formatted you may just need the correct mapping...

Whether the field is quoted or not, shouldn't matter....

So if you want to share what the incoming data looks like, etc, perhaps we can help

[erikg]

Hey @stephenb thanks for the help,

Yeah it seems the fix was to rollover the data stream, then delete the old index and now we are good:

[stephenb (Stephen Brown)]

Cool! Yeah keep _rollover in the back of your mind, tricky part with metrics because of TSDS etc... the rollover takes about 30 mins to take affect (very long discussion, because how the documents are routed by time etc. )