Reindex Data Stream

[teamomni]

We have recently moved all of our Logstash indices to the current data stream, but we are getting errors using the same ingest pipeline to fix the variable types of the original data stream.

The reindex command given:

POST /_reindex?pretty
{
  "source": {
    "index": ".ds-logs-logs-logstash-datastream-2022.05.06-000001"
  },
  "dest": {
    "index": "logs-logs-logstash-datastream",
    "pipeline":"convert_geoip",
    "op_type": "create"
  }
}

The ingest pipeline used to fix the type to geoip: (was used for reindexing indices over)

{
  "convert_geoip" : {
    "processors" : [
      {
        "geoip" : {
          "field" : "source.ip"
        }
      },
      {
        "geoip" : {
          "field" : "destination.ip"
        }
      },
      {
        "geoip" : {
          "field" : "host.ip"
        }
      }
    ]
  }
}

and a snippet of the results:

    "failures" : [
      {
        "index" : "logs-logs-logstash-datastream",
        "id" : "CGVYvoAB0t9pHq-JeUk1",
        "cause" : {
          "type" : "illegal_argument_exception",
          "reason" : "field [ip] not present as part of path [source.ip]"
        },
        "status" : 400
      },
      {
        "index" : "logs-logs-logstash-datastream",
        "id" : "MWVYvoAB0t9pHq-JfUkr",
        "cause" : {
          "type" : "illegal_argument_exception",
          "reason" : "field [source] not present as part of path [source.ip]"
        },
        "status" : 400
      },
	  ...

We also teseted adding "conflicts": "proceed" since not all logs contain IPs, but had the same errors. Does reindexing a data stream work differently than an index? Is it that the pipeline does not apply the same way?

[Badger]

This is not a logstash question, you might want to move it to the Elasticsearch forum.

[teamomni]

Moved it to the Elasticsearch forum.