Merge different index in function two fields and paste eleement of fields

[Vivian (Robin)]

Hello, I have an index of this shape :

{
  "_index": "df_extrait",
  "_type": "_doc",
  "_id": "27685",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "_score": 1,
  "_source": {
    "source": "avatar",
    "target": "toutpublic",
    "confidence_Score": "-",
    "Ref": "10716926",
    "Databases": "OCS:1234",
    "Taxon": "Film",
    "Interaction_Method": "Dialogue",
    "ID_method": [ "M0004" ],
    "Source_film": "OCS"
  }
}

{
  "_index": "df_extrait",
  "_type": "_doc",
  "_id": "82335",
  "_version": 1,
  "_seq_no": 10,
  "_primary_term": 1,
  "_score": 1,
  "_source": {
    "Source_film": "TF1Studio",
    "source": "avatar",
    "target": "toutpublic",
    "Taxon": "Film",
    "Ref": [
      "20936779",
      "10716926",
      "12970366",
      "32203420"
    ],
    "ID_method": [
      "M0398",
      "M0428",
      "M0096",
      "M0019",
      "M0018",
      "M0059",
      "M0045",
      "M0004",
      "M0004",
      "M0007"
    ],
    "Interaction_Method": [
      "Projection interactive",
      "Animation",
      "Animation virtuelle",
      "Modelage",
      "Projection anime",
      "Fond vert",
      "Extraction paysage",
      "Dialogue",
      "Dialogue",
      "Fond rouge"
    ]
  }
}

I would like to reunite the different index according to the source and target fields and create a new index in this case acquiere only one index :
expected results in this cases :

{
  "_index": "new_index",
  "_type": "_doc",
  "_id": "82335",
  "_version": 1,
  "_seq_no": 10,
  "_primary_term": 1,
  "_score": 1,
  "_source": {
    "Source_film": ["OCS,"TF1Studio"]
    "source": "avatar",
    "target": "toutpublic",
    "Taxon": "Film",
    "Ref": [
      "20936779",
      "10716926",
      "12970366",
      "32203420"
    ],
    "ID_method": [
      "M0398",
      "M0428",
      "M0096",
      "M0019",
      "M0018",
      "M0059",
      "M0045",
      "M0004",
      "M0007"
    ],
    "Interaction_Method": [
      "Projection interactive",
      "Animation",
      "Animation virtuelle",
      "Modelage",
      "Projection anime",
      "Fond vert",
      "Extraction paysage",
      "Dialogue",
      "Fond rouge"
    ]
  }
}

I tried to do by aggregation or by reindexation but I fall back on my starting index, can you help me.
Thanks for you help,
Vivian Robin

[warkolm (Mark Walkom)]

Elasticsearch doesn't use columns or rows, so it'd be worth changing the way you think/talk about data structures when using it, so no one is confused

I can't see a target field in your documents though?

[Vivian (Robin)]

I have modified the target field (previously called "cible") and the post for a better understanding as a beginner in devtools and Elasticsearch, please excuse me for my bad vocabulary

[can.ozdemir]

I cannot understand the requirement clearly but, I think you want to merge two indexes using a field to match the documents. if thats the case you can check;

https://stackoverflow.com/q/69044453/15799164

[Vivian (Robin)]

That's the idea and to eliminate redundant information I am inspired by this post (Merging documents based on matched fields values , , I work on siren investigate, I managed to make my aggregation as below :

GET df_extrait/_search?size=0
{
"aggs": {
    "my composite": {
      "composite": {
        "sources": [
          {
            "source_principal": {
              "terms": {
                "field": "source.keyword"
              }
            }
          },
          {
            "target_principal": {
              "terms": {
                "field": "target.keyword"
              }
            }
          }
        ]
        
      }
    }
  }

When I try to create a new index in this way I get this error:

[warkolm (Mark Walkom)]

Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them

[Vivian (Robin)]

As a replacement for the image:

PUT _transform/data_log
{
  "source": {
    "index": "df_extrait"
  },
  "dest": {
    "index": "df_extrait_reduced"
  },
  "pivot": {
    "group_by": {
      "source_principal": {"terms": {"field": "source"}},
      "target_principal": {"terms": {"field": "target"}}
    },
    "aggs": {
    "my composite": {
      "composite": {
        "sources": [
          {
            "source_principal": {
              "terms": {
                "field": "source.keyword"
              }
            }
          },
          {
            "target_principal": {
              "terms": {
                "field": "target.keyword"
              }
            }
          }
        ]
      }
        
      }
    }
  }
}

Output:

{
  "error": "Incorrect HTTP method for uri [/_transform/data_log?pretty] and method [PUT], allowed: [POST]",
  "status": 405
}

[warkolm (Mark Walkom)]

@Vivian

PUT _transform/data_log

That doesn't match the error though, so try removing the ?pretty.

[Vivian (Robin)]

In my request there is no pretty PUT _transform/data_log is this due to the fact that siren investigate the dev-tools do not have _transform ?