Filebeat and updating existing docs

Elastic Stack Beats
6

You only provided snippets of logs so it's very hard for me to tell / help.

You also didn't show me a sanitized version of what the publisher is actually publishing?

You show this ... But leave out all the interesting stuff :slight_smile:

DEBUG [processors] processing/processors.go:203 Publish event: {

I would remove the pipeline, run with -d "*"

Does it look correctly dissected?

Do you see the _id set?

Here is my sample... it works,
It creates a fingerprint
It assigns the _id and when written to elasticsearch uses that _id

- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - "/Users/sbrown/workspace/customers/acme/sample-data/catalina.out"

  processors:
    - fingerprint:
        fields: ["message"]
        target_field: "@metadata._id"
        method: "sha1"

And when I run
filebeat -e -d "*"

I see this in the below in the filebeat console... (you have to format it a bit) this is what is published to elasticsearch, it has the _id set.

{
	"@timestamp": "2022-12-10T16:24:41.400Z",
	"@metadata": {
		"beat": "filebeat",
		"type": "_doc",
		"version": "8.4.3",
		"_id": "6d084d6706c79dfb95ff164a9bf6abb0661ed5ee" <!---- Yup _id Set to fingerprint)
	},
	"message": "2022-11-09 01:26:17 INFO  Interesting Log Message",
	"input": {
		"type": "filestream"
	},
	"host": {
		"ip": [

So if you do not see this ... nothing down stream will work.

I suspect one of the processors it failing for you.