Unable to sync only documents that match connector include rules

Elastic Stack Kibana
1

Hi there,

I am setting up a MySQL connector to synchronise a table with Elasticsearch. My use case is to synchronise only rows that have a field matching a given value (and exclude all the others).

I created one include rule as follows (rule with prio 0):

image
image1504×576 31.3 KB

The rule with prio 1 is a default and uneditable rule that includes everything else (a default also described in the documentation). As expected, my rule is useless since the default rule includes everything.

But how do I only include the rows with the post_type field set to product?

I tried to create a catch-all exclusion rule similar to the default inclusion rule. Validation fails however:

Sync rule 0a555db2-9cbd-4ed7-939a-8336def57d20 is invalid.

Basic rule 2 (id: '0a555...57d20') is semantically equal to Basic rule 3 (id: 'D...T').

Basic rule 2 (id: '0a555...57d20') uses a match all regexps ['.*', '(.*)'], which are not allowed.

The error saying that the rule is semantically equivalent to the default rule is wrong as it excludes everything else (and not includes).

How do I implement my use case? I also tried implementing this with the filtering API with no luck.

Since it is not possible to define a "not equal" rule and since the default catch-all rule cannot be changed, I do not see how to implement this. Inclusion rules do not seem very useful if this is the default anyway.

Is there a way to change this default rule to an exclude rule?

Thanks for your help,
Angelo

2

We probably need to re-evaluate if match-all rules should be allowed, if there's a preceeding rule.

As a temporary workaround, you could use a regex rule to match anything except product, and use "Exclude" for that rule.

Alternatively, you can add explicit "Exclude" rules for all other types of post_type values.

3

Facing the same, have we re-evaluate?

Temporary workaround won't be a scalable solution if we need to add multiple rules.

4

how does the inclusion works under basic sync rule engine?
There’s always a default rule suggests everything else included.

{
    field: '_',
    id: 'DEFAULT',
    order: rules.length,
    policy: 'include',
    rule: 'regex',
    value: '.*',
}

So If I select a shared_drive in inclusion, it still syncs all data, not just that shared drive.

image
image2574×802 111 KB

And if i try to play around the default policy, it never allow us to modify, adding additional policy returns the following error.

{
  "ids": [
      "DEFAULT"
  ],
  "messages": [
      "Basic rule 4 (id: 'D...T') is semantically equal to Basic rule 3 (id: '4be03...9f58d')."
  ]
},
{
  "ids": [
      "4be0393e-0beb-4f6b-8d47-bdbcfe09f58d"
  ],
  "messages": [
      "Basic rule 3 (id: '4be03...9f58d') is semantically equal to Basic rule 4 (id: 'D...T')."
  ]
}
5

Made a slight change in second last case to exclude everything. It's workaround but worked.

image
image1752×162 25.9 KB

6

You can also just change it to .+ or .*.* to make it work.