Mapping conflict with Elastic Agent

Elastic Stack Elasticsearch
1

I am using the Fleet server with policies/integrations but I am running into some mapping conflicts that I am unsure how to resolve. For my "logs-*" index pattern I see I have a conflict of keyword,object on the "source" field.

The strange part is all of these keyword types are coming from the .ds-logs-elastic_agent.filebeat- indices. This makes me thing it is something to do with the IIS integration I have enabled because the two namespaces are one of the few that have an IIS integration.

How does someone go about correcting this mapping conflict? This is causing me to lose GEO data. I have a support ticket open but it has been going over for quite a while and I'm trying to find other sources of assistance to resolve this.

The mappings in conflict are:

.ds-logs-elastic_agent.filebeat-****-2022.01.07-000002
.ds-logs-elastic_agent.filebeat-****-2022.02.06-000004
.ds-logs-elastic_agent.filebeat-*********-2022.01.07-000003
.ds-logs-elastic_agent.filebeat-*********-2022.02.06-000006

and it shows the type is keyword when it should be object:

"source": {
"type": "keyword",
"ignore_above": 1024
},

2

Hi @ARDiver86, Filebeat sometimes logs using the source key with a string value that Elasticserach will understand as a keyword type instead of object. We have a GitHub issue about it: Source keyword in log files can lead to indexation failure. · Issue #30667 · elastic/beats · GitHub

However, it's not fully clear to me which GEO data you're loosing. Is it coming from the IIS integration?

Are there events failing to be indexed? Are they from Filebeat or the IIS integration?

1 Like
3

I have documents in other indexes where they have the source.ip column but no geo data. My assumption was the conflict was causing it not to look up the geo location because of this conflict.

This is also causing an issue where I cannot put in exceptions in the Alerts section for source.ip using lists. I should be able to make a list for IP Addresses and a put a rule in under alerts that states "if source.ip is in list " but the drop down does not populate my ip address list. We discovered that it doesnt populate the ip address list because source has a mapping conflict and is keyword.

Looking at some of the indices it says is in conflict they dont even have a source column in that index. So im confused as to why there is a conflict to begin with.

My ticket is 00865004

4

Thanks! I'll talk to the folks looking into this ticket. I have a hunch about the root cause. I'll write back once I have something confirmed.

1 Like
5

So I still have the issue with the mapping conflicts caused by the issue you mentioned but I managed to fix the IIS GEO issue.

It turns out the integration did not provide a pipeline so the IIS pipeline was not running on the documents being ingested. I think this may be a bug in a past version of the IIS integration because I completely removed the integration and installed it again and now the settings have a default_pipeline supplied. However, I thought Elastic did some sort of name matching to match a ingest pipeline based on the data stream name and/or index template name. That doesn't appear to be working or it isn't working because its also comparing the - which then it wouldn't of matched a pipeline.

6

I didn't quite get what you meant there, let me break down what I understood:

However, I thought Elastic did some sort of name matching to match a ingest pipeline based on the data stream name and/or index template name.

When adding a document to Elasticsearch, the request (single or bulk) can specify which ingest pipeline to use, that is how Filebeat uses ingest pipelines, when sending the events it tells ES which pipelines to use. This is all handled by Filebeat/Elastic-Agent, you should not need to worry about it.

Are you sending data to those indexes by means other than the Elastic-Agent/Filebeat/Beats?

That doesn't appear to be working or it isn't working because its also comparing the - which then it wouldn't of matched a pipeline.

What is being compared to what? I don't get that... What is the behaviour you are seeing and what did you expect to see?

7

According to the documentation: Ingest pipelines | Elasticsearch Guide [7.17] | Elastic

it states " Elasticsearch matches these templates to your Fleet data streams based on the stream’s naming scheme."

So what I was saying, is the ingest pipeline should be matched to the data stream just off the naming scheme. So if I have a template named "logs-msexchange.msgtracking" which created a "logs-msexchange.msgtracking-" I figured it would of matched the pipeline named "logs-msexchange.msgtracking". I'm assuming that it isn't matching due to the namespace not being in the name of the ingest pipeline.

8

First things first, this documentation you linked is regarding running Fleet and Elastic-Agent. When the Elastic-Agent is running under fleet it does a lot of work under the hood to get all the indexes, pipelines, etc working.

I believe you are confusing the templates this phrase refers too. Also the whole paragraph needs to be taken into consideration:

Fleet automatically adds ingest pipelines for its integrations. Fleet applies these pipelines using index templates that include pipeline index settings. Elasticsearch matches these templates to your Fleet data streams based on the stream’s naming scheme.

The "templates" mentioned in the bit you quoted are the index templates and they're matched to the "fleet data streams" (the data streams set up by Fleet and Elastic-Agent), it does not affect all data streams.

However, I thought Elastic did some sort of name matching to match a ingest pipeline based on the data stream name and/or index template name.

Ingest pipelines are not "auto-magically" matched by name to indexes, this configuration is explicit. When Elastic-Agent does its configuration of ingest pipelines, index templates and data streams, the observable result is that pattern matching. However if you just create ingest pipelines and indexes manually they will not be matched by name patterns.

That' one of the many advantages of using Elastic-Agent and integrations, they take care of all this heavy lifting for you.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.