Elastic Agent has not opened the port for Syslog to receive data. Fleet integration - filebeat module - Palo Alto firewall network (panw) - via Syslog.
Use Case:
I set up a fleet server, created a new policy and added an agent. I wanted this agent to work as a filebeat forwarder for the Palo Alto Network module/integration via syslog.
The agent is connected to Fleet and Elasticsearch and is working, but the logs are not displayed on the Fleet Agent - Logs tab. (Problem 1)
The monitoring logs of the agent are displayed, this can be seen on the data stream tab, module (panw - Palo Alto networks logs).
As I configured the integration, Filebeat should be listening on port 9001 for logs, but I cannot see this, the port is closed. (Problem 2)
elastic-agent inspect output -o default
[default] filebeat:
filebeat:
inputs:
- exclude_files:
- .gz$
id: logfile-system.auth-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: logs-system.auth-default
meta:
package:
name: system
version: 1.6.4
multiline:
match: after
pattern: ^\s
name: system-2
paths:
- /var/log/auth.log*
- /var/log/secure*
processors:
- add_locale: null
- add_fields:
fields:
dataset: system.auth
namespace: default
type: logs
target: data_stream
- add_fields:
fields:
dataset: system.auth
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
type: log
- exclude_files:
- .gz$
id: logfile-system.syslog-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: logs-system.syslog-default
meta:
package:
name: system
version: 1.6.4
multiline:
match: after
pattern: ^\s
name: system-2
paths:
- /var/log/messages*
- /var/log/syslog*
processors:
- add_locale: null
- add_fields:
fields:
dataset: system.syslog
namespace: default
type: logs
target: data_stream
- add_fields:
fields:
dataset: system.syslog
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
type: log
- fields:
_conf:
external_zones:
- untrust
internal_zones:
- trust
tz_offset: local
fields_under_root: true
id: syslog-panw.panos-837851ac-1859-4562-9d46-3706d9252590
index: logs-panw.panos-default
meta:
package:
name: panw
version: 1.3.2
name: panorama-integration
processors:
- add_locale: null
- add_fields:
fields:
dataset: panw.panos
namespace: default
type: logs
target: data_stream
- add_fields:
fields:
dataset: panw.panos
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
protocol:
udp:
host: 10.5.50.7:9001
publisher_pipeline.disable_host: true
revision: 1
tags:
- panw-panos
- forwarded
type: syslog
output:
elasticsearch:
api_key: cuTYun4BGwFPsNURfTGN:kDubZMOhQmKNhXlzzqIYdw
hosts:
- https://es01.ca.augustinum.org:9200
- https://es02.ca.augustinum.org:9200
- https://es03.ca.augustinum.org:9200
ssl:
certificate_authorities:
- /opt/Elastic/certs/ca.crt
---
[default] metricbeat:
metricbeat:
modules:
- cpu.metrics:
- percentages
- normalized_percentages
id: system/metrics-system.cpu-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: metrics-system.cpu-default
meta:
package:
name: system
version: 1.6.4
metricsets:
- cpu
module: system
name: system-2
period: 10s
processors:
- add_fields:
fields:
dataset: system.cpu
namespace: default
type: metrics
target: data_stream
- add_fields:
fields:
dataset: system.cpu
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
- id: system/metrics-system.filesystem-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: metrics-system.filesystem-default
meta:
package:
name: system
version: 1.6.4
metricsets:
- filesystem
module: system
name: system-2
period: 1m
processors:
- drop_event:
when:
regexp:
system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- add_fields:
fields:
dataset: system.filesystem
namespace: default
type: metrics
target: data_stream
- add_fields:
fields:
dataset: system.filesystem
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
- diskio.include_devices: null
id: system/metrics-system.diskio-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: metrics-system.diskio-default
meta:
package:
name: system
version: 1.6.4
metricsets:
- diskio
module: system
name: system-2
period: 10s
processors:
- add_fields:
fields:
dataset: system.diskio
namespace: default
type: metrics
target: data_stream
- add_fields:
fields:
dataset: system.diskio
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
- id: system/metrics-system.load-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: metrics-system.load-default
meta:
package:
name: system
version: 1.6.4
metricsets:
- load
module: system
name: system-2
period: 10s
processors:
- add_fields:
fields:
dataset: system.load
namespace: default
type: metrics
target: data_stream
- add_fields:
fields:
dataset: system.load
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
- id: system/metrics-system.memory-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: metrics-system.memory-default
meta:
package:
name: system
version: 1.6.4
metricsets:
- memory
module: system
name: system-2
period: 10s
processors:
- add_fields:
fields:
dataset: system.memory
namespace: default
type: metrics
target: data_stream
- add_fields:
fields:
dataset: system.memory
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
- id: system/metrics-system.fsstat-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: metrics-system.fsstat-default
meta:
package:
name: system
version: 1.6.4
metricsets:
- fsstat
module: system
name: system-2
period: 1m
processors:
- drop_event:
when:
regexp:
system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- add_fields:
fields:
dataset: system.fsstat
namespace: default
type: metrics
target: data_stream
- add_fields:
fields:
dataset: system.fsstat
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
- id: system/metrics-system.process.summary-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: metrics-system.process.summary-default
meta:
package:
name: system
version: 1.6.4
metricsets:
- process_summary
module: system
name: system-2
period: 10s
processors:
- add_fields:
fields:
dataset: system.process.summary
namespace: default
type: metrics
target: data_stream
- add_fields:
fields:
dataset: system.process.summary
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
- id: system/metrics-system.network-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: metrics-system.network-default
meta:
package:
name: system
version: 1.6.4
metricsets:
- network
module: system
name: system-2
network.interfaces: null
period: 10s
processors:
- add_fields:
fields:
dataset: system.network
namespace: default
type: metrics
target: data_stream
- add_fields:
fields:
dataset: system.network
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
- id: system/metrics-system.process-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: metrics-system.process-default
meta:
package:
name: system
version: 1.6.4
metricsets:
- process
module: system
name: system-2
period: 10s
process.cgroups.enabled: false
process.cmdline.cache.enabled: true
process.include_cpu_ticks: false
process.include_top_n.by_cpu: 5
process.include_top_n.by_memory: 5
processes:
- .*
processors:
- add_fields:
fields:
dataset: system.process
namespace: default
type: metrics
target: data_stream
- add_fields:
fields:
dataset: system.process
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
- id: system/metrics-system.socket_summary-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: metrics-system.socket_summary-default
meta:
package:
name: system
version: 1.6.4
metricsets:
- socket_summary
module: system
name: system-2
period: 10s
processors:
- add_fields:
fields:
dataset: system.socket_summary
namespace: default
type: metrics
target: data_stream
- add_fields:
fields:
dataset: system.socket_summary
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
- id: system/metrics-system.uptime-81e95ba6-a7a2-4bb7-8abf-901a7684490b
index: metrics-system.uptime-default
meta:
package:
name: system
version: 1.6.4
metricsets:
- uptime
module: system
name: system-2
period: 10s
processors:
- add_fields:
fields:
dataset: system.uptime
namespace: default
type: metrics
target: data_stream
- add_fields:
fields:
dataset: system.uptime
target: event
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
snapshot: false
version: 7.16.2
target: elastic_agent
- add_fields:
fields:
id: 78814aa0-1cd0-4b8d-9dbc-11f44480cd68
target: agent
revision: 1
output:
elasticsearch:
api_key: cuTYun4BGwFPsxxxxxxxxxxxxxxxxxxx
hosts:
- https://es01.xxxxxxxxxxxxxxxxxxx:9200
- https://es02.xxxxxxxxxxxxxxxxxxx:9200
- https://es03.xxxxxxxxxxxxxxxxxxx:9200
ssl:
certificate_authorities:
- /opt/Elastic/certs/ca.crt
---
[default] FLEET_MONITORING:
agent:
monitoring:
enabled: true
logs: true
metrics: true
namespace: default
use_output: default
monitoring_checksum: 5da0b2f629db8bd86c80e3add10d0e63
output:
elasticsearch:
api_key: cuTYun4BGxxxxxxxxxxxxxxxxxxxx
hosts:
- https://es01.xxxxxx:9200
- https://es02.xxxxxx:9200
- https://es03.xxxxxx:9200
ssl:
certificate_authorities:
- /opt/Elastic/certs/ca.crt
type: elasticsearch
programs:
- filebeat
- metricbeat
---
/opt/Elastic/Agent/data/elastic-agent-3c518f/install/filebeat-7.16.2-linux-x86_64/logs/filebeat
"host": {
"architecture": "x86_64",
"os": {
"codename": "focal",
"type": "linux",
"platform": "ubuntu",
"version": "20.04.3 LTS (Focal Fossa)",
"family": "debian",
"name": "Ubuntu",
"kernel": "5.4.0-96-generic"
},
"name": "aug-srv-fb001",
"id": "71518294922945a9b19620d480101c31",
"containerized": false,
"ip": [
"10.5.1.183",
"fe80::250:56ff:fe92:732"
],
"mac": [
"00:50:56:92:07:32"
],
"hostname": "aug-srv-fb001"
},
"input": {
"type": "filestream"
},
"event": {
"dataset": "elastic_agent.filebeat"
}
}
2022-02-02T18:40:07.473Z DEBUG [input.filestream] filestream/filestream.go:131 End of file reached: /opt/Elastic/Agent/data/elastic-agent-3c518f/logs/default/filebeat-json.log; Backoff now. {"id": "429D33EE9882AB68", "source": "filestream::.global::native::933237-64768", "path": "/opt/Elastic/Agent/data/elastic-agent-3c518f/logs/default/filebeat-json.log", "state-id": "native::933237-64768"}
2022-02-02T18:40:07.490Z DEBUG [elasticsearch] elasticsearch/client.go:232 PublishEvents: 4 events have been published to elasticsearch in 16.281599ms.
2022-02-02T18:40:07.490Z DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [12439: 0, 4]
2022-02-02T18:40:07.490Z DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=4, start-seq=7841, end-seq=7844
2022-02-02T18:40:07.490Z DEBUG [acker] beater/acker.go:64 stateless ack {"count": 4}
2022-02-02T18:40:07.490Z DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:4
2022-02-02T18:40:07.490Z DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2022-02-02T18:40:07.770Z INFO [file_watcher] filestream/fswatch.go:137 Start next scan
2022-02-02T18:40:07.770Z DEBUG [file_watcher] filestream/fswatch.go:204 Found 3 paths
2022-02-02T18:40:07.775Z INFO [file_watcher] filestream/fswatch.go:137 Start next scan
2022-02-02T18:40:07.775Z DEBUG [file_watcher] filestream/fswatch.go:204 Found 8 paths
2022-02-02T18:40:07.775Z DEBUG [input.filestream] filestream/prospector.go:164 File /opt/Elastic/Agent/data/elastic-agent-3c518f/logs/default/filebeat-json.log has been updated {"id": "429D33EE9882AB68", "prospector": "file_prospector", "operation": "write", "source_name": "native::933237-64768", "os_id": "933237-64768", "new_path": "/opt/Elastic/Agent/data/elastic-agent-3c518f/logs/default/filebeat-json.log", "old_path": "/opt/Elastic/Agent/data/elastic-agent-3c518f/logs/default/filebeat-json.log"}
2022-02-02T18:40:07.775Z DEBUG [input.filestream] input-logfile/harvester.go:145 Starting harvester for file {"id": "429D33EE9882AB68", "source": "filestream::.global::native::933237-64768"}
2022-02-02T18:40:07.775Z DEBUG [input.filestream] input-logfile/harvester.go:181 Stopped harvester for file {"id": "429D33EE9882AB68", "source": "filestream::.global::native::933237-64768"}
2022-02-02T18:40:07.775Z INFO [file_watcher] filestream/fswatch.go:137 Start next scan
2022-02-02T18:40:07.775Z DEBUG [file_watcher] filestream/fswatch.go:204 Found 8 paths
2022-02-02T18:40:07.775Z DEBUG [input.filestream] filestream/prospector.go:164 File /opt/Elastic/Agent/data/elastic-agent-3c518f/logs/default/metricbeat-json.log has been updated {"id": "A8E38CDDCA22F6F9", "prospector": "file_prospector", "operation": "write", "source_name": "native::933199-64768", "os_id": "933199-64768", "new_path": "/opt/Elastic/Agent/data/elastic-agent-3c518f/logs/default/metricbeat-json.log", "old_path": "/opt/Elastic/Agent/data/elastic-agent-3c518f/logs/default/metricbeat-json.log"}
2022-02-02T18:40:07.775Z DEBUG [input.filestream] input-logfile/harvester.go:145 Starting harvester for file {"id": "A8E38CDDCA22F6F9", "source": "filestream::.global::native::933199-64768"}
2022-02-02T18:40:07.775Z DEBUG [input.filestream] input-logfile/harvester.go:181 Stopped harvester for file {"id": "A8E38CDDCA22F6F9", "source": "filestream::.global::native::933199-64768"}
Configuration:
- Version: v7.16.2
- Operating System: Ubuntu
- Steps to Reproduce: Customer managed fleet server