Hi, i am newbie on Elasticsearch. I installed Elasticsearch and tried to send logs from fluentbit to elastic. I have script that generates 60 mb logs per minute. I read these logs with fluentbit and try to send them to elastic. But somehow it does not work. I tried same fluentbit configuration with other outputs and it works but it does not work with Elasticsearch. I can not see the logs on Kibana. I am using Elasticsearch and Kibana 7.15.1 version. Here is my Elasticsearch.yml:
# ======================== Elasticsearch Configuration =========================
# #
# # NOTE: Elasticsearch comes with reasonable defaults for most settings.
# # Before you set out to tweak and tune the configuration, make sure you
# # understand what are you trying to accomplish and the consequences.
# #
# # The primary way of configuring a node is via this file. This template lists
# # the most important settings you may want to configure for a production cluster.
# #
# # Please consult the documentation for further information on configuration options:
# # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
# #
# # ---------------------------------- Cluster -----------------------------------
# #
# # Use a descriptive name for your cluster:
# #
# #cluster.name: my-application
# #
# # ------------------------------------ Node ------------------------------------
# #
# # Use a descriptive name for the node:
# #
# #node.name: node-1
# #
# # Add custom attributes to the node:
# #
# #node.attr.rack: r1
# #
# # ----------------------------------- Paths ------------------------------------
# #
# # Path to directory where to store the data (separate multiple locations by comma):
# #
path.data: /var/lib/elasticsearch
# #
# # Path to log files:
# #
path.logs: /var/log/elasticsearch
# #
# # ----------------------------------- Memory -----------------------------------
# #
# # Lock the memory on startup:
# #
# #bootstrap.memory_lock: true
# #
# # Make sure that the heap size is set to about half the memory available
# # on the system and that the owner of the process is allowed to use this
# # limit.
# #
# # Elasticsearch performs poorly when the system is swapping the memory.
# #
# # ---------------------------------- Network -----------------------------------
# #
# # By default Elasticsearch is only accessible on localhost. Set a different
# # address here to expose this node on the network:
# #
network.host: XX.XXX.XXX.XXX
# #
# # By default Elasticsearch listens for HTTP traffic on the first free port it
# # finds starting at 9200. Set a specific HTTP port here:
# #
http.port: 9200
# #
# # For more information, consult the network module documentation.
# #
# # --------------------------------- Discovery ----------------------------------
# #
# # Pass an initial list of hosts to perform discovery when this node is started:
# # The default list of hosts is ["127.0.0.1", "[::1]"]
# #
discovery.seed_hosts: ["XX.XXX.XXX.XXX"]
# #
# # Bootstrap the cluster using an initial set of master-eligible nodes:
# #
# #cluster.initial_master_nodes: ["node-1", "node-2"]
# #
# # For more information, consult the discovery and cluster formation module documentation.
# #
# # ---------------------------------- Various -----------------------------------
# #
# # Require explicit names when deleting indices:
# #
# #action.destructive_requires_name: true
# xpack.security.enabled: true
discovery.type: single-node
Cluster health seems okay. Here is the result:
[root@elasticsearch ~]# curl -X GET 'XX.XXX.XXX.XXX:9200/_cluster/health?pretty'
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 11,
"active_shards" : 11,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 84.61538461538461
}
I tried to increase index refresh_interval value. I set it to 30 seconds. But result is same. Why Elasticsearch can not get the datas from fluentbit? Is it because it indexes slowly?
