Hi Everyone, I am new to the ELK world and learning the stack. I deployed ELK v8.11 using Helm in Kubernetes. I am currently trying to integrate Mikrotik NetFlow logs with Elasticsearch. My current setup already has Elasticsearch, Kibana, Logstash, and Filebeat running and working as expected. However, according to the Kibana dashboard, NetFlow record integration works with Elastic Agent.
My Fleet Server and Elastic Agent are also configured. I added two policies in my Elastic Agent: System and NetFlow. This allows me to get system beats, and my Elastic Agent is listening on port 2055 for NetFlow from Mikrotik.
The problem is that I can see system beats in indexes and other graphs in agent dashboards, but I do not see NetFlow beats at all. I confirmed using Tcpdump that Mikrotik is sending the logs in IPFIX format to the Elastic Agent, but I do not see those logs in Elasticsearch; only the agent's system logs are coming in.
Here is my Elastic Agent inspect output:
// agent:
// download:
// source_uri: https://artifacts.elastic.co/downloads/
// monitoring:
// enabled: true
// logs: true
// metrics: true
// namespace: default
// use_output: default
//fleet:
// hosts:
// - https://192.x.1.x:8220
//id: 5860d4e0-1a0e-11ef-be1f-1f7892ff17ad
//inputs:
//- data_stream:
// namespace: mktikus
// id: netflow-netflow-595ab1bc-1aed-4ab3-8271-d8316f32fd7a
// meta:
// package:
// name: netflow
// version: 2.9.0
// name: mktik-pakistan-netflow-policy
// package_policy_id: 595ab1bc-1aed-4ab3-8271-d8316f32fd7a
// revision: 3
// streams:
// - data_stream:
// dataset: netflow.log
// type: logs
// detect_sequence_reset: true
// expiration_timeout: 30m
// host: localhost:2055
// id: netflow-netflow.log-595ab1bc-1aed-4ab3-8271-d8316f32fd7a
// internal_networks:
// - 192.x.x.0/24
// - 10.xx.x0.0/24
// max_message_size: 10KiB
// protocols:
// - v1
// - v5
// - v6
// - v7
// - v8
// - v9
// - ipfix
// publisher_pipeline.disable_host: true
// queue_size: 8192
// tags:
// - netflow
// - forwarded
// type: netflow
// use_output: default
//output_permissions:
// default:
// _elastic_agent_checks:
// cluster:
// - monitor
// _elastic_agent_monitoring:
// indices:
// - names:
// - logs-elastic_agent.apm_server-default
// privileges:
// - auto_configure
// - create_doc
// (cleaned up some config)
// - metrics-elastic_agent.packetbeat-default
// privileges:
// - auto_configure
// - create_doc
// 595ab1bc-1aed-4ab3-8271-d8316f32fd7a:
// indices:
// - names:
// - logs-netflow.log-mktikus
// privileges:
// - auto_configure
// - create_doc
//outputs:
// default:
// api_key: 9D3lrI8BxasdfsadfasdfdfasdfasdfeicrULRZGW4xBRb4qj2Q
// hosts:
// - https://elasticsearch-master:31888
// type: elasticsearch
//revision: 11
list of indices at elasticsearch
GET _cat/indices/?v
here is the out put all indices. you may notice there is no netflow indices except other
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .ds-metrics-system.filesystem-default-2024.05.25-000001 klPyP4vEQOyVK6VgZQeXqQ 1 1 10168 0 3.1mb 1.5mb
green open .ds-logs-system.auth-default-2024.05.25-000001 LuWtyAONTzqtmAZBS7Pmwg 1 1 2118 0 1.2mb 710.5kb
green open .ds-logs-elastic_agent.metricbeat-default-2024.05.25-000001 QOtiUaQaREeHwAIHGzIodA 1 1 69489 0 10.9mb 5.4mb
green open .ds-metrics-system.socket_summary-default-2024.05.25-000001 Tm_UXQ55RpqZTr6riKEdFw 1 1 16739 0 4.8mb 2.4mb
green open .ds-metrics-system.fsstat-default-2024.05.25-000001 V45rp-oXQ4WstTi2-rTCxg 1 1 2792 0 1mb 532.9kb
green open .ds-logs-elastic_agent-default-2024.05.25-000001 5uPRaCQDSxmaTAAwarP_9A 1 1 8783 0 1.8mb 993.8kb
green open lb-gateway-2024.05.23 CN31OO0BSa2xfwNc6_V8FA 1 1 2220 0 2.4mb 1.1mb
green open .ds-metrics-elastic_agent.elastic_agent-default-2024.05.25-000001 cCC8Ck8aTlu_rXn4B17EZA 1 1 58965 0 18.7mb 9.3mb
green open lb-gateway-2024.05.25 x45TKUGVRGO7WfwZGrjV3Q 1 1 2699 0 3.2mb 1.4mb
green open lb-gateway-2024.05.26 FAOO6zHlSW6h8MqLE7TnhA 1 1 1204 0 1.3mb 708.7kb
green open lb-gateway-2024.05.24 s21GRFi0T7Gq2B8FYg-S1Q 1 1 6110 0 6.4mb 3.2mb
green open .ds-logs-elastic_agent.fleet_server-default-2024.05.25-000001 CRlKMusjQlSLEvmom-gvCA 1 1 391 0 244.5kb 122.2kb
green open .ds-metrics-system.load-default-2024.05.25-000001 Gsz_N6wjQ4KrcJdrhmmqKw 1 1 16739 0 4mb 2mb
green open .ds-metrics-system.process.summary-default-2024.05.25-000001 uHZLIrPgRqSmVpDNZl92Jw 1 1 16739 0 3.8mb 1.9mb
green open .ds-logs-elastic_agent.filebeat-default-2024.05.25-000001 TorpjISqROKUCuh9AfXuWA 1 1 26607 0 7.7mb 3.8mb
green open metrics-endpoint.metadata_current_default x2L8KSnZRserkWR2gLeMlA 1 1 0 0 450b 225b
green open .ds-metrics-system.uptime-default-2024.05.25-000001 -qIUT5lxSjSvJHxzA-nTWg 1 1 16739 0 3.6mb 1.7mb
green open .ds-metrics-elastic_agent.fleet_server-default-2024.05.25-000001 LCOUL6_EQXigyRZqGzGRSQ 1 1 8445 0 1.9mb 969.8kb
green open .ds-metrics-system.cpu-default-2024.05.25-000001 AMXGgeXkRY-4udVGjVIMSA 1 1 16739 0 6mb 3.1mb
green open .ds-metrics-elastic_agent.filebeat-default-2024.05.25-000001 d8Ew8QREQN2BwHgPJf_qvw 1 1 33778 0 11.2mb 5.6mb
green open .ds-logs-system.syslog-default-2024.05.25-000001 3dC1r2sWSWSorihvrTQ4ng 1 1 12038 0 3.5mb 1.7mb
green open .ds-.fleet-actions-results-2024.05.24-000001 udTg28m7R9Kkbb37thJCEw 1 1 16 0 62kb 33kb
green open .ds-metrics-system.process-default-2024.05.25-000001 S1IVxkbLSlaEiuK0wOoDCQ 1 1 135121 0 216.7mb 106.9mb
green open .ds-metrics-system.memory-default-2024.05.25-000001 G5oQiTz5RMyqe7vc0B4_0w 1 1 16739 0 5.5mb 2.7mb
green open .ds-metrics-elastic_agent.metricbeat-default-2024.05.25-000001 VKk0wXiATT6avL-rmeEAZQ 1 1 33484 0 6.6mb 3.2mb
green open .ds-metrics-system.network-default-2024.05.25-000001 nLrB-RwtQAS2C3vBRMxhAA 1 1 58654 0 18.8mb 9.4mb
green open .monitoring-es-7-2024.05.25 CXvW_oBKRIaOToHy9iUSjw 1 1 114412 0 151.3mb 63.5mb
green open .monitoring-es-7-2024.05.26 lByB5qqpTfi6yOY_8Vn1YQ 1 1 182251 97448 466.8mb 230.2mb
green open .monitoring-kibana-7-2024.05.25 7p3I3_E0Sr6sNLaQQ3HB-A 1 1 6340 0 2.8mb 1.4mb
green open .ds-metrics-system.diskio-default-2024.05.25-000001 5DwcXPWNS3SjJI0ml1M2XA 1 1 431456 0 115.6mb 58.7mb
green open .monitoring-kibana-7-2024.05.26 Qm7Ah0aNQ2KaKUptzxlCeg 1 1 8 0 2.2mb 1.1mb
Thank you,
Myk