File Beat to Elasticsearch unable to publish Events

Hi Team,

Why I am Getting the below Error

2024-02-26T14:40:53.019+0700    ERROR   [elasticsearch] elasticsearch/client.go:226     failed to perform any bulk index operations: Post "https://x.x.x.x:9200/_bulk": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2024-02-26T14:40:53.019+0700    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2024-02-26T14:40:53.019+0700    INFO    [publisher]     pipeline/retry.go:223     done
2024-02-26T14:40:53.019+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://x.x.x.x:9200))
2024-02-26T14:40:53.019+0700    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2024-02-26T14:40:53.020+0700    INFO    [publisher]     pipeline/retry.go:223     done
2024-02-26T14:40:53.020+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://x.x.x.x:9200))
2024-02-26T14:40:53.020+0700    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2024-02-26T14:40:53.020+0700    INFO    [publisher]     pipeline/retry.go:223     done
2024-02-26T14:40:53.020+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://1x.x.x.x:9200))
2024-02-26T14:40:53.020+0700    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2024-02-26T14:40:53.020+0700    INFO    [publisher]     pipeline/retry.go:223     done
2024-02-26T14:40:53.031+0700    INFO    [esclientleg]   eslegclient/connection.go:282   Attempting to connect to Elasticsearch version 7.17.6
2024-02-26T14:40:53.033+0700    INFO    [esclientleg]   eslegclient/connection.go:282   Attempting to connect to Elasticsearch version 7.17.6
2024-02-26T14:40:53.035+0700    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2024-02-26T14:40:53.083+0700    INFO    [esclientleg]   eslegclient/connection.go:282   Attempting to connect to Elasticsearch version 7.17.6
2024-02-26T14:40:53.101+0700    INFO    [index-management.ilm]  ilm/std.go:170  ILM policy firewall_ilm exists already.
2024-02-26T14:40:53.152+0700    INFO    [index-management.ilm]  ilm/std.go:126  Index Alias smartfren-firewall exists already.
2024-02-26T14:40:53.191+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://x.x.x.x:9200)) established
2024-02-26T14:40:53.196+0700    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2024-02-26T14:40:53.260+0700    INFO    [index-management.ilm]  ilm/std.go:170  ILM policy firewall_ilm exists already.
2024-02-26T14:40:53.274+0700    INFO    [index-management.ilm]  ilm/std.go:126  Index Alias smartfren-firewall exists already.
2024-02-26T14:40:53.322+0700    ERROR   [publisher_pipeline_output]     pipeline/output.go:180  failed to publish events: Post "https://x.x.x.x:9200/_bulk": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2024-02-26T14:40:53.429+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://x.x.x.x:9200)) established
2024-02-26T14:40:53.433+0700    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2024-02-26T14:40:53.468+0700    INFO    [index-management.ilm]  ilm/std.go:170  ILM policy firewall_ilm exists already.
2024-02-26T14:40:53.486+0700    INFO    [index-management.ilm]  ilm/std.go:126  Index Alias smartfren-firewall exists already.
2024-02-26T14:40:53.548+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://x.x.x.x:9200)) established
2024-02-26T14:40:54.331+0700    ERROR   [publisher_pipeline_output]     pipeline/output.go:180  failed to publish events: Post "https://x.x.x.x:9200/_bulk": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2024-02-26T14:40:57.123+0700    INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":94743710,"time":{"ms":2324}},"total":{"ticks":517486550,"time":{"ms":14015},"value":517486550},"user":{"ticks":422742840,"time":{"ms":11691}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":201},"info":{"ephemeral_id":"b4cf11fd-f5b8-4b49-b6bf-74b2e9a3899b","uptime":{"ms":576750953},"version":"7.16.2"},"memstats":{"gc_next":2912234304,"memory_alloc":1528311144,"memory_total":79227890298288,"rss":5171474432},"runtime":{"goroutines":715}},"filebeat":{"events":{"added":67584,"done":67584},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":106944,"active":124163967,"batches":133,"total":149440},"read":{"bytes":28066588,"errors":67},"write":{"bytes":147279359}},"pipeline":{"clients":1,"events":{"active":668661,"published":67584,"retry":112768,"total":67584},"queue":{"acked":67584}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.43,"15":0.37,"5":0.43,"norm":{"1":0.0538,"15":0.0463,"5":0.0538}}}}}}

Because of Retry attempts losing Events data in Dashboard from File beat to Elasticsearch nodes, can anyone have any idea let me know .

Even after implementing the below configuration in filebeat.yml still, the error exists

  loadbalance: true
  timeout: 400
  backoff.init: 5
  backoff.max: 10

try running

filebeat test output

The error says that fielbeat can not connect with your elasticsearch there are many possible reasons try the test first

Then from filebeat server

curl -k -v -u elastic https://x.x.x.x:9200

Show the commands and results from both

Hi @stephenb,

Every time you will be there for Community! Glad to hear more from your Recommendations again.

As this is the Production Environment, I will be doing this and sharing the test output soon, but this issue is happening only once in 2 days fyi we are maintaining an ELK Cluster with 10 nodes.

Please find the Curl output here

[root@logstash]$ curl -k -v -u elastic https://x.x.x.x:9200
Enter host password for user 'elastic':
* About to connect() to x.x.x.x port 9200 (#0)
*   Trying x.x.x.x...
* Connected to x.x.x.x (x.x.x.x) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=instance
*       start date: Sep 15 15:02:08 2022 GMT
*       expire date: Sep 14 15:02:08 2025 GMT
*       common name: instance
*       issuer: CN=Elastic Certificate Tool Autogenerated CA
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Authorization: Basic ZWxhc3RpYzplbGFzdGljMTIz
> User-Agent: curl/7.29.0
> Host: x.x.x.x:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 548
<
{
  "name" : "elasticsearch07",
  "cluster_name" : "elk",
  "cluster_uuid" : "0SK6GtiGSFiuCeA1c_E5Gw",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host x.x.x.x left intact
[root@logstash]$ curl -k -v -u elastic https://x.x.x.x:9200
Enter host password for user 'elastic':
* About to connect() to x.x.x.x port 9200 (#0)
*   Trying x.x.x.x...
* Connected tox.x.x.x (x.x.x.x) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=instance
*       start date: Sep 15 15:02:08 2022 GMT
*       expire date: Sep 14 15:02:08 2025 GMT
*       common name: instance
*       issuer: CN=Elastic Certificate Tool Autogenerated CA
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Authorization: Basic ZWxhc3RpYzplbGFzdGljMTIz
> User-Agent: curl/7.29.0
> Host: x.x.x.x:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 548
<
{
  "name" : "elasticsearch09",
  "cluster_name" : "elk",
  "cluster_uuid" : "0SK6GtiGSFiuCeA1c_E5Gw",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host x.x.x.x left intact
[root@logstash]$ curl -k -v -u elastic https://x.x.x.x:9200
Enter host password for user 'elastic':
* About to connect() to x.x.x.x port 9200 (#0)
*   Trying x.x.x.x...
* Connected to x.x.x.x (x.x.x.x) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=instance
*       start date: Sep 15 15:02:08 2022 GMT
*       expire date: Sep 14 15:02:08 2025 GMT
*       common name: instance
*       issuer: CN=Elastic Certificate Tool Autogenerated CA
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Authorization: Basic ZWxhc3RpYzplbGFzdGljMTIz
> User-Agent: curl/7.29.0
> Host: x.x.x.x:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 548
<
{
  "name" : "elasticsearch02",
  "cluster_name" : "elk",
  "cluster_uuid" : "0SK6GtiGSFiuCeA1c_E5Gw",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host x.x.x.x left intact
[root@logstash ~]$ curl -k -v -u elastic https://x.x.x.x:9200
Enter host password for user 'elastic':
* About to connect() to x.x.x.x port 9200 (#0)
*   Trying x.x.x.x...
* Connected to x.x.x.x (x.x.x.x) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=instance
*       start date: Sep 15 15:02:08 2022 GMT
*       expire date: Sep 14 15:02:08 2025 GMT
*       common name: instance
*       issuer: CN=Elastic Certificate Tool Autogenerated CA
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Authorization: Basic ZWxhc3RpYzplbGFzdGljMTIz
> User-Agent: curl/7.29.0
> Host: x.x.x.x:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 548
<
{
  "name" : "elasticsearch10",
  "cluster_name" : "elk",
  "cluster_uuid" : "0SK6GtiGSFiuCeA1c_E5Gw",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host x.x.x.x left intact

Thanks for your time in sharing the info.

I am not sure what you mean..
a)is the the Error connecting only happening intermittently?
b) or all the time?

Please share a sample of the filebeat.yml output.elasticsearch section ... sanitized...

Hi @stephenb,

The error happening intermittently only

output.elasticsearch:
  # Array of hosts to connect to.
  hosts:  ["X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200"]
  # Protocol - either `http` (default) or `https`.
  protocol: "https"
  username: "elastic"
  password: "xxxxxxxx"
  ssl.verification_mode: none
   ssl.certificate_authorities: ["logstash-ca.crt"]
  ssl.certificate: "logstash.pem"
  ssl.key: "logstash-ca.key"
  worker: 28
  bulk_max_size: 1600


output.elasticsearch.index: "X.X.X.X"
setup.template.enabled: false
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "X.X.X.X"
setup.ilm.policy_name: "X.X.X.X"


Intermittent is usually some form of connectivity issue...

Those are hard to debug...

@stephenb Thanks for your Valuable time, Let me check once

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.