File Beat to Elasticsearch unable to publish Events

Mani_Manikanta · February 26, 2024, 7:44am

Hi Team,

Why I am Getting the below Error

2024-02-26T14:40:53.019+0700    ERROR   [elasticsearch] elasticsearch/client.go:226     failed to perform any bulk index operations: Post "https://x.x.x.x:9200/_bulk": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2024-02-26T14:40:53.019+0700    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2024-02-26T14:40:53.019+0700    INFO    [publisher]     pipeline/retry.go:223     done
2024-02-26T14:40:53.019+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://x.x.x.x:9200))
2024-02-26T14:40:53.019+0700    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2024-02-26T14:40:53.020+0700    INFO    [publisher]     pipeline/retry.go:223     done
2024-02-26T14:40:53.020+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://x.x.x.x:9200))
2024-02-26T14:40:53.020+0700    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2024-02-26T14:40:53.020+0700    INFO    [publisher]     pipeline/retry.go:223     done
2024-02-26T14:40:53.020+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://1x.x.x.x:9200))
2024-02-26T14:40:53.020+0700    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2024-02-26T14:40:53.020+0700    INFO    [publisher]     pipeline/retry.go:223     done
2024-02-26T14:40:53.031+0700    INFO    [esclientleg]   eslegclient/connection.go:282   Attempting to connect to Elasticsearch version 7.17.6
2024-02-26T14:40:53.033+0700    INFO    [esclientleg]   eslegclient/connection.go:282   Attempting to connect to Elasticsearch version 7.17.6
2024-02-26T14:40:53.035+0700    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2024-02-26T14:40:53.083+0700    INFO    [esclientleg]   eslegclient/connection.go:282   Attempting to connect to Elasticsearch version 7.17.6
2024-02-26T14:40:53.101+0700    INFO    [index-management.ilm]  ilm/std.go:170  ILM policy firewall_ilm exists already.
2024-02-26T14:40:53.152+0700    INFO    [index-management.ilm]  ilm/std.go:126  Index Alias smartfren-firewall exists already.
2024-02-26T14:40:53.191+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://x.x.x.x:9200)) established
2024-02-26T14:40:53.196+0700    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2024-02-26T14:40:53.260+0700    INFO    [index-management.ilm]  ilm/std.go:170  ILM policy firewall_ilm exists already.
2024-02-26T14:40:53.274+0700    INFO    [index-management.ilm]  ilm/std.go:126  Index Alias smartfren-firewall exists already.
2024-02-26T14:40:53.322+0700    ERROR   [publisher_pipeline_output]     pipeline/output.go:180  failed to publish events: Post "https://x.x.x.x:9200/_bulk": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2024-02-26T14:40:53.429+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://x.x.x.x:9200)) established
2024-02-26T14:40:53.433+0700    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2024-02-26T14:40:53.468+0700    INFO    [index-management.ilm]  ilm/std.go:170  ILM policy firewall_ilm exists already.
2024-02-26T14:40:53.486+0700    INFO    [index-management.ilm]  ilm/std.go:126  Index Alias smartfren-firewall exists already.
2024-02-26T14:40:53.548+0700    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://x.x.x.x:9200)) established
2024-02-26T14:40:54.331+0700    ERROR   [publisher_pipeline_output]     pipeline/output.go:180  failed to publish events: Post "https://x.x.x.x:9200/_bulk": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2024-02-26T14:40:57.123+0700    INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":94743710,"time":{"ms":2324}},"total":{"ticks":517486550,"time":{"ms":14015},"value":517486550},"user":{"ticks":422742840,"time":{"ms":11691}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":201},"info":{"ephemeral_id":"b4cf11fd-f5b8-4b49-b6bf-74b2e9a3899b","uptime":{"ms":576750953},"version":"7.16.2"},"memstats":{"gc_next":2912234304,"memory_alloc":1528311144,"memory_total":79227890298288,"rss":5171474432},"runtime":{"goroutines":715}},"filebeat":{"events":{"added":67584,"done":67584},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":106944,"active":124163967,"batches":133,"total":149440},"read":{"bytes":28066588,"errors":67},"write":{"bytes":147279359}},"pipeline":{"clients":1,"events":{"active":668661,"published":67584,"retry":112768,"total":67584},"queue":{"acked":67584}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.43,"15":0.37,"5":0.43,"norm":{"1":0.0538,"15":0.0463,"5":0.0538}}}}}}

Because of Retry attempts losing Events data in Dashboard from File beat to Elasticsearch nodes, can anyone have any idea let me know .

Mani_Manikanta · February 26, 2024, 2:03pm

Even after implementing the below configuration in filebeat.yml still, the error exists

  loadbalance: true
  timeout: 400
  backoff.init: 5
  backoff.max: 10

stephenb · February 26, 2024, 3:28pm

try running

filebeat test output

The error says that fielbeat can not connect with your elasticsearch there are many possible reasons try the test first

Then from filebeat server

curl -k -v -u elastic https://x.x.x.x:9200

Show the commands and results from both

Mani_Manikanta · February 27, 2024, 3:43am

Hi @stephenb,

Every time you will be there for Community! Glad to hear more from your Recommendations again.

As this is the Production Environment, I will be doing this and sharing the test output soon, but this issue is happening only once in 2 days fyi we are maintaining an ELK Cluster with 10 nodes.

Please find the Curl output here

[root@logstash]$ curl -k -v -u elastic https://x.x.x.x:9200
Enter host password for user 'elastic':
* About to connect() to x.x.x.x port 9200 (#0)
*   Trying x.x.x.x...
* Connected to x.x.x.x (x.x.x.x) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=instance
*       start date: Sep 15 15:02:08 2022 GMT
*       expire date: Sep 14 15:02:08 2025 GMT
*       common name: instance
*       issuer: CN=Elastic Certificate Tool Autogenerated CA
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Authorization: Basic ZWxhc3RpYzplbGFzdGljMTIz
> User-Agent: curl/7.29.0
> Host: x.x.x.x:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 548
<
{
  "name" : "elasticsearch07",
  "cluster_name" : "elk",
  "cluster_uuid" : "0SK6GtiGSFiuCeA1c_E5Gw",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host x.x.x.x left intact
[root@logstash]$ curl -k -v -u elastic https://x.x.x.x:9200
Enter host password for user 'elastic':
* About to connect() to x.x.x.x port 9200 (#0)
*   Trying x.x.x.x...
* Connected tox.x.x.x (x.x.x.x) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=instance
*       start date: Sep 15 15:02:08 2022 GMT
*       expire date: Sep 14 15:02:08 2025 GMT
*       common name: instance
*       issuer: CN=Elastic Certificate Tool Autogenerated CA
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Authorization: Basic ZWxhc3RpYzplbGFzdGljMTIz
> User-Agent: curl/7.29.0
> Host: x.x.x.x:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 548
<
{
  "name" : "elasticsearch09",
  "cluster_name" : "elk",
  "cluster_uuid" : "0SK6GtiGSFiuCeA1c_E5Gw",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host x.x.x.x left intact
[root@logstash]$ curl -k -v -u elastic https://x.x.x.x:9200
Enter host password for user 'elastic':
* About to connect() to x.x.x.x port 9200 (#0)
*   Trying x.x.x.x...
* Connected to x.x.x.x (x.x.x.x) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=instance
*       start date: Sep 15 15:02:08 2022 GMT
*       expire date: Sep 14 15:02:08 2025 GMT
*       common name: instance
*       issuer: CN=Elastic Certificate Tool Autogenerated CA
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Authorization: Basic ZWxhc3RpYzplbGFzdGljMTIz
> User-Agent: curl/7.29.0
> Host: x.x.x.x:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 548
<
{
  "name" : "elasticsearch02",
  "cluster_name" : "elk",
  "cluster_uuid" : "0SK6GtiGSFiuCeA1c_E5Gw",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host x.x.x.x left intact
[root@logstash ~]$ curl -k -v -u elastic https://x.x.x.x:9200
Enter host password for user 'elastic':
* About to connect() to x.x.x.x port 9200 (#0)
*   Trying x.x.x.x...
* Connected to x.x.x.x (x.x.x.x) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=instance
*       start date: Sep 15 15:02:08 2022 GMT
*       expire date: Sep 14 15:02:08 2025 GMT
*       common name: instance
*       issuer: CN=Elastic Certificate Tool Autogenerated CA
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Authorization: Basic ZWxhc3RpYzplbGFzdGljMTIz
> User-Agent: curl/7.29.0
> Host: x.x.x.x:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 548
<
{
  "name" : "elasticsearch10",
  "cluster_name" : "elk",
  "cluster_uuid" : "0SK6GtiGSFiuCeA1c_E5Gw",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host x.x.x.x left intact

Thanks for your time in sharing the info.

stephenb · February 27, 2024, 7:07am

I am not sure what you mean..
a)is the the Error connecting only happening intermittently?
b) or all the time?

Please share a sample of the filebeat.yml output.elasticsearch section ... sanitized...

Mani_Manikanta · February 27, 2024, 7:28am

Hi @stephenb,

The error happening intermittently only

output.elasticsearch:
  # Array of hosts to connect to.
  hosts:  ["X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200", "X.X.X.X:9200"]
  # Protocol - either `http` (default) or `https`.
  protocol: "https"
  username: "elastic"
  password: "xxxxxxxx"
  ssl.verification_mode: none
   ssl.certificate_authorities: ["logstash-ca.crt"]
  ssl.certificate: "logstash.pem"
  ssl.key: "logstash-ca.key"
  worker: 28
  bulk_max_size: 1600


output.elasticsearch.index: "X.X.X.X"
setup.template.enabled: false
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "X.X.X.X"
setup.ilm.policy_name: "X.X.X.X"

stephenb · February 27, 2024, 8:52am

Intermittent is usually some form of connectivity issue...

Those are hard to debug...

Mani_Manikanta · February 27, 2024, 10:11am

@stephenb Thanks for your Valuable time, Let me check once

system · March 26, 2024, 12:11pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.