Hi,
I am trying to setup Elsticsearch 8, Kibana and Filebeat 8 all on my local laptop, using this guide:
I've got ES and Kibana running in Docker containers and communicating just fine.
Next, I want to send events from Filebeat (which pulls them from a GCP PubSub topic) to ES - and I see the the Filebeat reads events form PubSub just fine, tries to push them into ES, but fails with errors:
{"log.level":"debug","@timestamp":"2022-10-20T17:06:49.805Z","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":210},"message":"Publish event: {\n \"@timestamp\": \"2022-10-20T17:06:48.466Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.4.3\",\n \"_id\": \"m_id_1020_1\"\n },\n \"event\": {\n \"id\": \"59279bf715-5952832523902946\",\n \"created\": \"2022-10-20T17:06:49.799Z\"\n },\n \"message\": {\n \"request_status\": \"500\",\n \"cid\": \"12345\",\n \"remote_ip\": \"165.155.130.139\",\n \"referer\": \"https://www.my.site2.com/\",\n \"ref_param\": \"https://www.nyt.com\",\n \"request_method\": \"POST\",\n \"response_size\": \"124\",\n \"activity_date\": \"2022-10-20\",\n \"user_agent\": \"Mozilla/5.0 (X11; CrOS aarch64 13421.102.0) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/86.0.4240.199 Safari/537.36\",\n \"event_timestamp_millis\": \"1666285498000\",\n \"request_size\": \"52\",\n \"latency\": \"1.3\",\n \"logstash_id\": \"m_id_1020_1\"\n },\n \"input\": {\n \"type\": \"gcp-pubsub\"\n },\n \"host\": {\n \"containerized\": true,\n \"ip\": [\n \"172.17.0.2\"\n ],\n \"mac\": [\n \"02:42:ac:11:00:02\"\n ],\n \"hostname\": \"d61a776a9c35\",\n \"architecture\": \"x86_64\",\n \"name\": \"d61a776a9c35\",\n \"os\": {\n \"kernel\": \"5.10.47-linuxkit\",\n \"codename\": \"focal\",\n \"type\": \"linux\",\n \"platform\": \"ubuntu\",\n \"version\": \"20.04.5 LTS (Focal Fossa)\",\n \"family\": \"debian\",\n \"name\": \"Ubuntu\"\n }\n },\n \"agent\": {\n \"version\": \"8.4.3\",\n \"ephemeral_id\": \"0ce2530a-8f5d-4710-ac31-2a64b5b31272\",\n \"id\": \"c5a8197e-6dc7-4c55-bc43-779ac473f1f1\",\n \"name\": \"d61a776a9c35\",\n \"type\": \"filebeat\"\n },\n \"ecs\": {\n \"version\": \"8.0.0\"\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-20T17:06:50.808Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(https://localhost:9200))","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-20T17:06:50.810Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":267},"message":"ES Ping(url=https://localhost:9200)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-10-20T17:06:50.813Z","log.logger":"esclientleg","log.origin":{"file.name":"transport/logging.go","file.line":38},"message":"Error dialing dial tcp 127.0.0.1:9200: connect: connection refused","service.name":"filebeat","network":"tcp","address":"localhost:9200","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-20T17:06:50.814Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":271},"message":"Ping request failed with: Get \"https://localhost:9200\": dial tcp 127.0.0.1:9200: connect: connection refused","service.name":"filebeat","ecs.version":"1.6.0"}
Here is my full filebeat.yml for reference:
queue.mem:
events: 4096
flush.min_events: 2048
flush.timeout: 1s
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: gcp-pubsub
enabled: true
project_id: ${PROJECT_ID}
topic: ${PUBSUB_INPUT_TOPIC}
subscription.name: ${SUBSCRIPTION_NAME}
fields_under_root: true
# ======================= Elasticsearch template setting =======================
setup.template.name: "ibc-parsed-logs"
setup.template.pattern: "ibc-parsed-logs-*"
setup.template.json.enabled: true
setup.template.json.path: "ibc_es_template.json"
setup.template.json.name: "ibc-parsed-logs-template"
setup.template.enabled: true
setup.ilm.enabled: false
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
output.console:
enabled: false
pretty: true
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
enabled: true
index: "ibc-parsed-logs"
parameters.pipeline: "geoip-info"
hosts: ${ES_HOSTS}
protocol: "https"
api_key: ${ES_API_KEY}
# ============================= X-Pack Monitoring ==============================
monitoring.enabled: true
monitoring.cluster_uuid: ${MON_CLUSTER_UUID}
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- decode_json_fields:
fields: ["message"]
add_error_key: true
document_id: "event_uuid"
# ================================== Logging ===================================
logging.metrics.enabled: true
logging.enabled: true
logging.level: debug
logging.to_files: true
logging.files:
path: /usr/share/filebeat/f_logs
name: filebeat
keepfiles: 10
permissions: 0640
logging.selectors: ["*"]
I run Filebeat in a Docker container as well, as following:
Dockerfile:
FROM docker.elastic.co/beats/filebeat:8.4.3
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
COPY ibc_es_template.json /usr/share/filebeat/ibc_es_template.json
USER root
RUN chmod +x filebeat.yml
I've built a Docker image with this Dockerfile: filebeat8-local-min:1.0
Docker container run command:
docker run -it --rm \
-v /Users/mpopova/.config/gcloud/application_default_credentials.json:/usr/share/filebeat/application_default_credentials.json \
-v "$(pwd)/f_logs:/usr/share/filebeat/f_logs" \
-e GOOGLE_APPLICATION_CREDENTIALS=/usr/share/filebeat/application_default_credentials.json \
-e PROJECT_ID=my-gcp-pr \
-e PUBSUB_INPUT_TOPIC=logs-for-es-marina \
-e SUBSCRIPTION_NAME=logs-for-es-marina-sub \
-e ES_HOSTS="https://localhost:9200" \
-e MON_CLUSTER_UUID="qQNb9W_xxx" \
-e ES_API_KEY="urks_xxx" \
-e DEBUG_LEVEL=info \
filebeat8-local-min:1.0
I tried a few variation of the ES_HOSTS variable I pass:
-- http://localhost:9200
-- http://0.0.0.0:9200
same result...
What am I missing?
Thank you!!
Marina