Renaming of auto-generated field names (created by Postgres connector)

Elastic Stack Elasticsearch
,
1

Dear Elastic team,
I am currently using an Elasticsearch docker cluster with a Postgres connector and Kibana using a basic license. I am able to synchronize my data but find the default field names that are being generated unfortunate since I am only synchronizing a single table. Appearently the following naming scheme is being use schema_table_field - but this leads to very long field names. I assume that I need to use a custom pipeline but in Kibana I see the following message:

Custom pipelines require a Platinum license or higher and are not available to Standard license self-managed deployments. You need to upgrade to use this feature.

Did you know that custom pipelines are available with a Standard Elastic Cloud license? Elastic Cloud gives you the flexibility to run where you want. Deploy our managed service on Google Cloud, Microsoft Azure, or Amazon Web Services, and we'll handle the maintenance and upkeep for you.

So my question is: Which options do I have to rename the field names?

Thanks in advance :sunflower:

2

Can you share exactly where you did saw this message? I'm using the basic license and a I can create custom ingest pipelines, so not sure where this is coming from.

3

Oh, sorry, my mistake.

I thought you were talking about Elastic Agent integrations, but this is related to Enterprise Search.

I just found where this message shows up.

I do not use Enterprise Search, but looking at it, it seems to use a managed ingest pipeline named ent-search-generic-ingestion, while it is not recommended you could try to add a pipeline processor in this managed ingest pipeline to call another pipeline with your renames.

I'm not sure this will work as it does not make much sense to require a License to use a custom ingest pipeline, but at the same time being able to edit the managed ingest pipeline to add processors.

Maybe someone from Elastic has a better explanation.

1 Like
4

Hey @andrej.peplinski Looking at the documentation, yes for index-level cusom pipelines for connectors basic license is not sufficient: Ingest pipelines in Search | Elasticsearch Guide [8.14] | Elastic

Perhaps you can create a new ingest pipeline, and directly attach it to the index to which connector syncs data, I think you can set the default pipeline for your index: Ingest pipelines | Elasticsearch Guide [8.15] | Elastic. It should work, but it might be a bit more of a hassle to manage.

Alternatively, if you are running postgresql connector as a self-managed connector, you can customize the connector source code to achieve the desired result. See: connectors/connectors/sources/postgresql.py at main · elastic/connectors · GitHub

Hope this helps!