Custom Logs - json - Why do we need to map fields?

Elastic Stack Elastic Agent
1

Hi,
json objects fields are meant to be dynamic.
Does 'Custom Logs' from the 'Elastic Agent'
have a facility to map json fields on the fly as entry comes in ?

If you do not have any mapping an error occurs
'could not create the map from the configuration: missing field accessing 'outputs'

Thanks

2

Hi @jjwallaby Welcome to the community.

Apologies, I'm not sure I'm following.

Can you provide a little more information please?

What version are you on?

Can you confirm you're trying to Use the custom logs integration with elastic agent?

Can you show the configuration?

Provide a couple samples of the logs?

Where exactly did you see that error?

And yes elastic will dynamically create a mapping for you but there could be conflicts if you have JSON that have conflicting structures.

Perhaps if you provide a little more context, we might be able to help a little bit

3

My question was really
Do you need to do the mapping of json
when doing 'Custom Logs integration'

mapping_optional
mapping_optional819×363 30.7 KB

I was just trying to follow an example of the best practice for integration of Custom logs.
I did do mappings. But not sure if it is needed.
elasticsearch version:8.9.0
agent version: elastic-agent-8.9.0-linux-x86_64

4

Which example?

Not sure what you mean mapping json...

so in short Mappings are a best practice for scale, production, efficiency etc etc etc...

If you are just messing around they are not necessary.

The custom logs will do some good guesses for you with some under the covers magic ... So take a look ...

But again, if you're doing production, typically you create a mapping...

5

Thanks stephen,

I see that if you do not map the json fields. the fields do not appear
in the search field list of building dashboard/visualization.

if_do_not_add_fields_in_mapping_the_dont_appear
if_do_not_add_fields_in_mapping_the_dont_appear302×1006 52.7 KB

6

They appear now after a couple of hours. Seems it does eventually build its data dictionary.

7

What version are you on?

Shift Reload of Kibana should be sufficient to see fields.

so if your logs are ndjson you need to tell the integration to parse the json
if you do not all the json will just end up in the message field..

There are 3 ways to do that (of course always more than one way)

  1. ndjson parser
  2. decode_json_fields processor
  3. ingest pipeline

First 2 you can just set in the Integration Setup there used to be commented out sample but not any more

8

The version is 8.9.0.
The fields do appear after an hour.
Much appreciated.
Thanks for answering.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.