Custom Logs - json - Why do we need to map fields?

[jjwallaby (Justin W)]

Hi,
json objects fields are meant to be dynamic.
Does 'Custom Logs' from the 'Elastic Agent'
have a facility to map json fields on the fly as entry comes in ?

If you do not have any mapping an error occurs
'could not create the map from the configuration: missing field accessing 'outputs'

Thanks

[stephenb (Stephen Brown)]

Hi @jjwallaby Welcome to the community.

Apologies, I'm not sure I'm following.

Can you provide a little more information please?

What version are you on?

Can you confirm you're trying to Use the custom logs integration with elastic agent?

Can you show the configuration?

Provide a couple samples of the logs?

Where exactly did you see that error?

And yes elastic will dynamically create a mapping for you but there could be conflicts if you have JSON that have conflicting structures.

Perhaps if you provide a little more context, we might be able to help a little bit

[jjwallaby (Justin W)]

My question was really
Do you need to do the mapping of json
when doing 'Custom Logs integration'

I was just trying to follow an example of the best practice for integration of Custom logs.
I did do mappings. But not sure if it is needed.
elasticsearch version:8.9.0
agent version: elastic-agent-8.9.0-linux-x86_64

[stephenb (Stephen Brown)]

@jjwallaby

I was just trying to follow an example of the best practice for integration of Custom logs.
I did do mappings. But not sure if it is needed.

@jjwallaby

Do you need to do the mapping of json
when doing 'Custom Logs integration'

Which example?

Not sure what you mean mapping json...

so in short Mappings are a best practice for scale, production, efficiency etc etc etc...

If you are just messing around they are not necessary.

The custom logs will do some good guesses for you with some under the covers magic ... So take a look ...

But again, if you're doing production, typically you create a mapping...

[jjwallaby (Justin W)]

Thanks stephen,

I see that if you do not map the json fields. the fields do not appear
in the search field list of building dashboard/visualization.

[jjwallaby (Justin W)]

They appear now after a couple of hours. Seems it does eventually build its data dictionary.

[stephenb (Stephen Brown)]

What version are you on?

Shift Reload of Kibana should be sufficient to see fields.

so if your logs are ndjson you need to tell the integration to parse the json
if you do not all the json will just end up in the message field..

There are 3 ways to do that (of course always more than one way)

1. ndjson parser
2. decode_json_fields processor
3. ingest pipeline

First 2 you can just set in the Integration Setup there used to be commented out sample but not any more

[jjwallaby (Justin W)]

The version is 8.9.0.
The fields do appear after an hour.
Much appreciated.
Thanks for answering.