Custom Filestream Integration - Adding mappings

Elastic Stack Elastic Agent
1

Hello,

I am having trouble with adding custom mapping to my data stream built from custom filestream input.
It appears that once the data comes in and build the data stream, the data stream is “managed”.
It doesn’t allow me the option to add any custom mappings?

image
image481×939 46.2 KB

ES/Kibana: 8.18.0
Custom Filestream: 1.2.0

2

Just to add,
I have used Custom Azure Input Integration before and was able to add my custom mappings, this is now pointing to an issue with custom filestream input.

For example:

image
image513×966 41.9 KB

This creates its own index template:
In which you can add your own component template, azure

image
image956×951 54.4 KB

I cannot edit the encompassing ‘logs’ index template in Custom Filestream Input, that would apply unnecessary component templates.

3

The logs-azure_logs.*@custom component template would the be place to put custom settings/mappings, not the azure template.

Any of the @custom templates and pipelines are not managed by the system and will persist your changes.

After making changes to the template to define mappings, you would need to rollover the data stream so it can make a new index with the updated mappings.

4

Hey @mmahacek thanks for that clarification, I will fix that for my azure setup.

My main issue is how do I add the custom settings/mappings to the Custom Filestream Integration? There seems to be no @custom component template, just the default logs

5

Hi @erikg

Here is how it works.... and agree it is inconsistent between the different packages... the azure input sets up template scaffolding the filestream does not I am no sure why.

That said currently Custom File Stream Input does not generate a template specific to the data_stream.dataset that you define and thus just uses the default logs-*

You can fix / address this in about 5 minutes .... but if you want to do it really quick here it is I keep this around handy.

Assumes your datastream is logs-mycustom.filestream-*

  • Put this in
  • Rollover the data stream
  • It will show up in the UI
  • It will use all the logs-* goodness and add your own.
  • Its basically a clone then edit of the logs template
PUT _index_template/logs-mycustom.filestream
{
  "version": 1,
  "priority": 200,  << MUST BE GREATER THAN 100 so it takes precendence 
  "template": {
    "mappings": {
      "_source": {
        "mode": "synthetic" 
      }
    },
    "settings": {
      "index": {
        "mode": "standard" <<< I set to logsdb if you want 
      }
    }
  },
  "index_patterns": [
    "logs-mycustom.filestream-*" <<< Your datastream name 
  ],
  "data_stream": {
    "hidden": false,
    "allow_custom_routing": false
  },
  "composed_of": [
    "logs@mappings",
    "logs@settings",
    "ecs@mappings",
    "logs-mycustom.filestream@custom",  << ADD THIS
    "logs@custom"
  ],
  "ignore_missing_component_templates": [
    "logs-mycustom.filestream@custom", << ADD THIS
    "logs@custom"
  ],
  "allow_auto_create": true,
  "_meta": {
    "description": "custom filestream logs template",
    "managed": false <<< TURN OF MANAGED 
  },
  "deprecated": false
}

You can then add your custom mappings and settings to
logs-mycustom.filestream@custom or
liga@custom

1 Like
6

The @custom component templates aren’t created automatically. If you create a new component template that has that exact name, it will line up on work. The index template creates references with a ignore_missing_component_templates flag to skip it as long as it doesn’t exist.