Pipeline/output.go:180 failed to publish events: temporary bulk send failure

Elastic Stack Beats
,
1

i have defined the following pipeline called "suzieq" and tested it against 2 documents

PUT _ingest/pipeline/suzieq
{
  "version": 2,
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{@timestamp} %{?ident} %{?suzieq.header}.%{suzieq.component}.%{suzieq.module}.%{+suzieq.service}.%{+suzieq.service} %{?ident} %{severity} %{?ident} %{message}",
        "append_separator": "-",
        "ignore_missing": true
      }
    }
  ],
  "on_failure": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{@timestamp} %{?ident} %{?suzieq.header}.%{suzieq.component}.%{suzieq.module} %{?ident} %{severity} %{?ident} %{message}"
      }
    }
  ]
}

i then added

output.elasticsearch.pipeline: "suzieq"

to filebeat.yml

when loading filebeat, no logs are being received in elastic and filebeat logs show the following

2022-03-11T12:51:10.128Z        INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://security-master:92
00)) established
2022-03-11T12:51:10.192Z        INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2022-03-11T12:51:10.192Z        INFO    [publisher]     pipeline/retry.go:223     done
2022-03-11T12:51:11.907Z        ERROR   [publisher_pipeline_output]     pipeline/output.go:180  failed to publish events: temporary bulk send failure
2022-03-11T12:51:11.907Z        INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://security-master:92
00))
2022-03-11T12:51:11.907Z        INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer

what am i doing wrong?

2

That might not be the most helpful error message to get started with :expressionless:

  1. Do you have anything in the Elasticsearch log? There might be a good hint hidden there.
  2. I assume this works if you disable the ingest pipeline in the Filebeat output, right?
3

Hi,

yes, removing the pipeline i can get everything working and logs are properly sent...

for context, this is the filebeat configuration

filebeat.inputs:
 - type: filestream
   enabled: true
   paths:
    - /tmp/*.log
 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ["https://security-master:9200"]
  #pipeline: "suzieq"
  protocol: "https"
  username: '${ELASTICSEARCH_USERNAME}'
  password: '${ELASTICSEARCH_PASSWORD}'
  ssl:
    enabled: true
    verification_mode: certificate
    certificate_authorities: ["/usr/share/filebeat/elastic-certificate.pem"]
    #certificate: "/usr/share/filebeat/elastic-certificate.crt"
    #key: "/usr/share/filebeat/elastic-certificate.pem" 

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded

let me check the logs on the Elasticsearch side, good hint, i didn't think about it before

4

so i have been checking the logs, but i don't see much in there, i pulled logs from all 3 the Elasticsearch nodes

5

Maybe simply message sent by filebeat is not dissected by ingest pipeline. Share full json message stored in index without ingest pipeline...

6

Hello Ylasir,

here are two messages , the same ones i used to validate the pipeline itself.

here is the first one:

{
  "_index": "filebeat-7.17.1-2022.03.11-000001",
  "_type": "_doc",
  "_id": "deR2h38Bwp_nE4SE2J0D",
  "_version": 1,
  "_score": 1,
  "_source": {
    "@timestamp": "2022-03-14T08:06:26.074Z",
    "message": "2022-03-14 08:06:25,009 - suzieq.poller.worker.nodes.node - ERROR - Unable to connect to F88_G1MSW10 for show bgp all neighbors due to Permission denied",
    "input": {
      "type": "filestream"
    },
    "ecs": {
      "version": "1.12.0"
    },
    "host": {
      "name": "sq-poller-0",
      "ip": [
        "10.244.1.115",
        "fe80::2009:a7ff:fe58:1fb8"
      ],
      "mac": [
        "22:09:a7:58:1f:b8"
      ],
      "hostname": "sq-poller-0",
      "architecture": "x86_64",
      "os": {
        "platform": "ubuntu",
        "version": "20.04.3 LTS (Focal Fossa)",
        "family": "debian",
        "name": "Ubuntu",
        "kernel": "5.4.0-1069-azure",
        "codename": "focal",
        "type": "linux"
      },
      "containerized": true
    },
    "agent": {
      "id": "fc2fa625-f84a-42f1-9e90-7caa86c64643",
      "name": "sq-poller-0",
      "type": "filebeat",
      "version": "7.17.1",
      "hostname": "sq-poller-0",
      "ephemeral_id": "2c74d2c5-170a-4404-99ed-edfe30a9e630"
    },
    "log": {
      "offset": 2921049,
      "file": {
        "path": "/tmp/sq-poller-0.log"
      }
    }
  },
  "fields": {
    "host.os.name.text": [
      "Ubuntu"
    ],
    "host.hostname": [
      "sq-poller-0"
    ],
    "host.mac": [
      "22:09:a7:58:1f:b8"
    ],
    "host.ip": [
      "10.244.1.115",
      "fe80::2009:a7ff:fe58:1fb8"
    ],
    "agent.type": [
      "filebeat"
    ],
    "host.os.version": [
      "20.04.3 LTS (Focal Fossa)"
    ],
    "host.os.kernel": [
      "5.4.0-1069-azure"
    ],
    "host.os.name": [
      "Ubuntu"
    ],
    "agent.name": [
      "sq-poller-0"
    ],
    "host.name": [
      "sq-poller-0"
    ],
    "host.os.type": [
      "linux"
    ],
    "host.os.codename": [
      "focal"
    ],
    "input.type": [
      "filestream"
    ],
    "log.offset": [
      2921049
    ],
    "agent.hostname": [
      "sq-poller-0"
    ],
    "message": [
      "2022-03-14 08:06:25,009 - suzieq.poller.worker.nodes.node - ERROR - Unable to connect to F88_G1MSW10 for show bgp all neighbors due to Permission denied"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "@timestamp": [
      "2022-03-14T08:06:26.074Z"
    ],
    "agent.id": [
      "fc2fa625-f84a-42f1-9e90-7caa86c64643"
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "host.containerized": [
      true
    ],
    "host.os.platform": [
      "ubuntu"
    ],
    "log.file.path": [
      "/tmp/sq-poller-0.log"
    ],
    "agent.ephemeral_id": [
      "2c74d2c5-170a-4404-99ed-edfe30a9e630"
    ],
    "agent.version": [
      "7.17.1"
    ],
    "host.os.family": [
      "debian"
    ]
  }
}

and here is the second

{
  "_index": "filebeat-7.17.1-2022.03.11-000001",
  "_type": "_doc",
  "_id": "F-R5h38Bwp_nE4SE9Z4D",
  "_version": 1,
  "_score": 1,
  "_source": {
    "@timestamp": "2022-03-14T08:09:50.106Z",
    "log": {
      "offset": 2950510,
      "file": {
        "path": "/tmp/sq-poller-0.log"
      }
    },
    "input": {
      "type": "filestream"
    },
    "ecs": {
      "version": "1.12.0"
    },
    "host": {
      "os": {
        "type": "linux",
        "platform": "ubuntu",
        "version": "20.04.3 LTS (Focal Fossa)",
        "family": "debian",
        "name": "Ubuntu",
        "kernel": "5.4.0-1069-azure",
        "codename": "focal"
      },
      "containerized": true,
      "ip": [
        "10.244.1.115",
        "fe80::2009:a7ff:fe58:1fb8"
      ],
      "mac": [
        "22:09:a7:58:1f:b8"
      ],
      "name": "sq-poller-0",
      "hostname": "sq-poller-0",
      "architecture": "x86_64"
    },
    "agent": {
      "type": "filebeat",
      "version": "7.17.1",
      "hostname": "sq-poller-0",
      "ephemeral_id": "2c74d2c5-170a-4404-99ed-edfe30a9e630",
      "id": "fc2fa625-f84a-42f1-9e90-7caa86c64643",
      "name": "sq-poller-0"
    },
    "message": "2022-03-14 08:09:48,772 - suzieq.poller.worker.nodes.node - ERROR - Unable to connect to F88_G1MSW10 for show ip interface due to Permission denied"
  },
  "fields": {
    "host.os.name.text": [
      "Ubuntu"
    ],
    "host.hostname": [
      "sq-poller-0"
    ],
    "host.mac": [
      "22:09:a7:58:1f:b8"
    ],
    "host.ip": [
      "10.244.1.115",
      "fe80::2009:a7ff:fe58:1fb8"
    ],
    "agent.type": [
      "filebeat"
    ],
    "host.os.version": [
      "20.04.3 LTS (Focal Fossa)"
    ],
    "host.os.kernel": [
      "5.4.0-1069-azure"
    ],
    "host.os.name": [
      "Ubuntu"
    ],
    "agent.name": [
      "sq-poller-0"
    ],
    "host.name": [
      "sq-poller-0"
    ],
    "host.os.type": [
      "linux"
    ],
    "host.os.codename": [
      "focal"
    ],
    "input.type": [
      "filestream"
    ],
    "log.offset": [
      2950510
    ],
    "agent.hostname": [
      "sq-poller-0"
    ],
    "message": [
      "2022-03-14 08:09:48,772 - suzieq.poller.worker - WARNING - log level WARNING"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "@timestamp": [
      "2022-03-14T08:09:50.106Z"
    ],
    "agent.id": [
      "fc2fa625-f84a-42f1-9e90-7caa86c64643"
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "host.containerized": [
      true
    ],
    "host.os.platform": [
      "ubuntu"
    ],
    "log.file.path": [
      "/tmp/sq-poller-0.log"
    ],
    "agent.ephemeral_id": [
      "2c74d2c5-170a-4404-99ed-edfe30a9e630"
    ],
    "agent.version": [
      "7.17.1"
    ],
    "host.os.family": [
      "debian"
    ]
  }
}
7

Making sure this doesn't get closed

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.