Hi
I try using F5 telemetry streaming with Security onion (siem) based on Elasticsearch.
For do this i use F5 BIG-IP integration.
Unfortunately its not working.
F5 :
Version : BIG-IP 15.1.10.4 Build 0.0.5 Point Release 4
Module :
- f5-appsvcs-3.53.0-7.noarch
- f5-telemetry-1.37.0-1.noarch
Déclaration Telemetry Streaming :
Send with Postman , post to https://F5IP/mgmt/shared/telemetry/declare
{
"class": "Telemetry",
"controls": {
"class":"Controls",
"logLevel": "error"
},
"My_System": {
"class": "Telemetry_System",
"systemPoller": {
"interval": 60
}
},
"My_Listener": {
"class": "Telemetry_Listener",
"port": 6514,
"trace": [
{
"type": "input"
},
{
"type": "output"
}
]
},
"My_Consumer": {
"class": "Telemetry_Consumer",
"type": "Generic_HTTP",
"host": "SOC_IP",
"allowSelfSignedCert": true,
"protocol": "http",
"trace": true,
"path": "/",
"method": "POST",
"port": "9570",
"headers": [
{
"name": "content-type",
"value": "application/json"
}
]
}
}
Result 200 ok
AS3 logging sources Déclaration :
Send with Postman , post to https://F5IP>/mgmt/shared/appsvcs/declare
{
"class": "ADC",
"schemaVersion": "3.10.0",
"remark": "Example depicting creation of BIG-IP module log profiles",
"Common": {
"class": "Tenant",
"Shared": {
"class": "Application",
"template": "shared",
"telemetry_local_rule": {
"remark": "Only required when TS is a local listener",
"class": "iRule",
"iRule": "when CLIENT_ACCEPTED {\n node 127.0.0.1 6514\n}"
},
"telemetry_local": {
"remark": "Only required when TS is a local listener",
"class": "Service_TCP",
"virtualAddresses": [
"255.255.255.254"
],
"virtualPort": 6514,
"iRules": [
"telemetry_local_rule"
]
},
"telemetry": {
"class": "Pool",
"members": [
{
"enable": true,
"serverAddresses": [
"255.255.255.254"
],
"servicePort": 6514
}
],
"monitors": [
{
"bigip": "/Common/tcp"
}
]
},
"telemetry_hsl": {
"class": "Log_Destination",
"type": "remote-high-speed-log",
"protocol": "tcp",
"pool": {
"use": "telemetry"
}
},
"telemetry_formatted": {
"class": "Log_Destination",
"type": "splunk",
"forwardTo": {
"use": "telemetry_hsl"
}
},
"telemetry_publisher": {
"class": "Log_Publisher",
"destinations": [
{
"use": "telemetry_formatted"
}
]
},
"telemetry_traffic_log_profile": {
"class": "Traffic_Log_Profile",
"requestSettings": {
"requestEnabled": true,
"requestProtocol": "mds-tcp",
"requestPool": {
"use": "telemetry"
},
"requestTemplate": "event_source=\"request_logging\",hostname=\"$BIGIP_HOSTNAME\",client_ip=\"$CLIENT_IP\",server_ip=\"$SERVER_IP\",http_method=\"$HTTP_METHOD\",http_uri=\"$HTTP_URI\",virtual_name=\"$VIRTUAL_NAME\",event_timestamp=\"$DATE_HTTP\""
},
"responseSettings": {
"responseEnabled": true,
"responseProtocol": "mds-tcp",
"responsePool": {
"use": "telemetry"
},
"responseTemplate": "event_source=\"response_logging\",hostname=\"$BIGIP_HOSTNAME\",client_ip=\"$CLIENT_IP\",server_ip=\"$SERVER_IP\",http_method=\"$HTTP_METHOD\",http_uri=\"$HTTP_URI\",virtual_name=\"$VIRTUAL_NAME\",event_timestamp=\"$DATE_HTTP\",http_statcode=\"$HTTP_STATCODE\",http_status=\"$HTTP_STATUS\",response_ms=\"$RESPONSE_MSECS\""
}
}
}
}
}
Result 200 ok
I add the logging Source Sytem log with the GUI => System => Logs => Configuration =>Remote logging => Modify the system syslog configuration by adding a destination 127.0.0.1 remote-port 6514
I config 4 vhosts with (not in the same partitions)=> Request Logging Profile => telemetry_traffic_log_profile
I check the log in :
[root]# tail -f /var/log/restnoded/restnoded.log
Wed, 12 Mar 2025 07:46:01 GMT - info: [telemetry.service.RESTAPIService] Request dd78e received: POST /shared/telemetry/declare
Wed, 12 Mar 2025 07:46:01 GMT - info: [telemetry] Global logLevel set to 'error'
Wed, 12 Mar 2025 07:46:01 GMT - severe: [telemetry.service.RuntimeConfigService.task] Task done!
Wed, 12 Mar 2025 07:46:06 GMT - finest: socket 2233 closed
With the trace at true for the listener and consumer i get the files below and à lot of events:
[root]# ls /var/tmp/telemetry/Telemetry_*
/var/tmp/telemetry/Telemetry_Consumer.f5telemetry_default::My_Consumer /var/tmp/telemetry/Telemetry_Listener.f5telemetry_default::My_Listener
If i try to check if my big-ip TS Event Listener is sending data to my consumer => Documentation https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/troubleshooting.html
Send with Postman , post to https://F5IP>/mgmt/shared/telemetry/eventListener/My_Listener
Body:
{
"message": "my debugging message"
}
Result => "code": 200,
log My_Listener
{
"data": {
"data": {
"data": "{\"message\":\"my debugging message\"}",
"telemetryEventCategory": "event",
"originalRawData": "{\"message\":\"my debugging message\"}"
},
"type": "event",
"sourceId": "cbf1e402-6f30-4aba-9526-3f005dba08d0",
"destinationIds": [
"663094f2-3480-4620-8bf4-b4f5cfca722f"
]
},
"timestamp": "2025-03-12T09:17:24.435Z"
}
log My_Consummer
{
"data": {
"allowSelfSignedCert": true,
"body": {
"data": "{\"message\":\"my debugging message\"}",
"telemetryEventCategory": "event",
"originalRawData": "{\"message\":\"my debugging message\"}"
},
"compressionType": "none",
"host": "10.250.130.33",
"fallbackHosts": [],
"headers": {
"content-type": "application/json"
},
"method": "POST",
"port": 9570,
"protocol": "http",
"uri": "/"
},
"timestamp": "2025-03-12T09:17:24.435Z"
}
In security onion (ELK 8.14.3) with distributed installation
the firewall is open for the source ip_F5 and the tcp port 9570
The module integration F5 Big IP (v1.17.0) is correctly ( i hope) configured :
- Collect F5 BIG-IP logs via HTTP Endpoint :
Integration name => f5_bigip-1
Namespace => default
Enable Collect F5 BIG-IP logs via HTTP Endpoint
Listen Address => 0.0.0.0
F5 BIG-IP logs via HTTP Endpoint listen port: 9570
url => /
tags => f5_bigip-log , forwarded
[siem@soc ~]$ sudo iptables -nvL | grep 9570
[sudo] password for siem:
2842 3125K ACCEPT tcp -- * * IP_F5 0.0.0.0/0 tcp dpt:9570
0 0 ACCEPT tcp -- * * IP_F5-HA 0.0.0.0/0 tcp dpt:9570
[siem@soc ~]$ sudo ss -tunlp | grep -E "9570"
tcp LISTEN 0 4096 *:9570 *:* users:(("agentbeat",pid=1903,fd=11))
in ssh on the F5 i force sending data to my nod => "originalRawData": "{\"message\":\"my debugging message\"}" and try to see if i recept him in my node
[siem@soc ~]$ sudo tcpdump -i ens192 tcp and host F5_IP and port 9570 -A | grep "debugging message"
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), snapshot length 262144 bytes
{"data":"{\"message\":\"my debugging message\"}","telemetryEventCategory":"event","originalRawData":"{\"message\":\"my debugging message\"}"}
70 packets captured
73 packets received by filter
0 packets dropped by kernel
When i try to go to Kibana - Discover - search with "F5" => Nothing
The F5 send the paquet , the node manager with elastic agent and the policy "so-grid-nodes_general" who carries the F5 integration receive the paquet .. but nothing come in Discover
i try to search the good log file F5 integration local or in the dockers , but i can't find it. It sounds like a bad configuration of the F5 module
If anyone has already encountered this problem or has an idea that would allow me to resolve the problem, I am interested.