Logstash 8.8.2 not sending filebeat 8.8.2 logs to eleasticsearch 8.8.2 database

Elastic Stack Logstash
,
1

Dear Team,

Our scenario is network devices send logs to filebeat. Filebeat send those logs to logstash and logstash send logs to elasticsearch database but we are receive below warning in logstash logs and we didn't receive logs over elasticsearch database.

Please check below log and guide us.

LOG:
[WARN ] 2023-07-15 10:56:07.673 [[main]>worker3] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-8.8.2-2023.07.15", :routing=>nil, :pipeline=>"filebeat-8.8.2-cef-log-pipeline"}, {"network"=>{"community_id"=>"1:9om9NcQdXluRGUfDOvfgFpU/Cho=", "direction"=>"inbound", "transport"=>"17"}, "fileset"=>{"name"=>"log"}, "agent"=>{"id"=>"a227c13a-73ad-4712-99ef-85f31649c7e7", "name"=>"filestash", "version"=>"8.8.2", "ephemeral_id"=>"5b1dbc27-634c-423a-b98d-0463be0a2fd7", "type"=>"filebeat"}, "message"=>"domain-udp", "service"=>{"type"=>"cef"}, "tags"=>["cef", "forwarded", "beats_input_codec_plain_applied"], "source"=>{"nat"=>{"ip"=>"10.10.10.2", "port"=>54430}, "ip"=>"192.168.1.5", "port"=>57878}, "log"=>{"source"=>{"address"=>"192.168.1.1:50451"}}, "input"=>{"type"=>"syslog"}, "observer"=>{"product"=>"VPN-1 & FireWall-1", "version"=>"Check Point", "vendor"=>"Check Point"}, "cef"=>{"name"=>"domain-udp", "extensions"=>{"parent_rule"=>"0", "sourcePort"=>57878, "version"=>"5", "outzone"=>"External", "deviceCustomString2Label"=>"Rule Name", "ifname"=>"eth5", "deviceReceiptTime"=>"2023-07-15T10:56:06.000Z", "inzone"=>"Internal", "origin"=>"172.16.1.1", "rule_action"=>"Accept", "loguid"=>"{0x64b27b46,0x1,0x1ed6d655,0xd797660c}", "transportProtocol"=>"17", "sourceTranslatedAddress"=>"10.10.10.2", "layer_uuid"=>"63b7fe60-76d2-4287-bca5-21af87337b0a", "sourceTranslatedPort"=>54430, "match_id"=>"1", "sourceAddress"=>"192.168.1.5", "destinationPort"=>53, "nat_rulenum"=>"0", "destinationTranslatedAddress"=>"0.0.0.0", "deviceAction"=>"Accept", "destinationTranslatedPort"=>0, "deviceDirection"=>0, "destinationAddress"=>"1.1.1.1", "originsicname"=>"cn=cp_mgmt,o=ISN..iqb794", "logid"=>"0", "rule_uid"=>"76fd22f2-efa8-4c81-a617-40201d3f5c4e", "deviceCustomString2"=>"Full access", "sequencenum"=>"9", "nat_addtnl_rulenum"=>"0", "layer_name"=>"Network", "conn_direction"=>"Outgoing"}, "version"=>"0", "device"=>{"event_class_id"=>"Log", "product"=>"VPN-1 & FireWall-1", "version"=>"Check Point", "vendor"=>"Check Point"}, "severity"=>"Unknown"}, "@timestamp"=>2023-07-15T10:56:06.000Z, "event"=>{"original"=>"CEF:0|Check Point|VPN-1 & FireWall-1|Check Point|Log|domain-udp|Unknown|act=Accept destinationTranslatedAddress=0.0.0.0 destinationTranslatedPort=0 deviceDirection=0 rt=1689418566000 sourceTranslatedAddress=10.10.10.2 sourceTranslatedPort=54430 spt=57878 dpt=53 cs2Label=Rule Name cs2=Full access layer_name=Network layer_uuid=63b7fe60-76d2-4287-bca5-21af87337b0a match_id=1 parent_rule=0 rule_action=Accept rule_uid=76fd22f2-efa8-4c81-a617-40201d3f5c4e conn_direction=Outgoing ifname=eth5 logid=0 loguid={0x64b27b46,0x1,0x1ed6d655,0xd797660c} origin=172.16.1.1 originsicname=cn\=cp_mgmt,o\=ISN..iqb794 sequencenum=9 version=5 dst=1.1.1.1 inzone=Internal nat_addtnl_rulenum=0 nat_rulenum=0 outzone=External proto=17 src=192.168.1.5\n", "dataset"=>"cef.log", "action"=>"Accept", "code"=>"Log", "module"=>"cef"}, "ecs"=>{"version"=>"1.12.0"}, "@version"=>"1", "destination"=>{"nat"=>{"ip"=>"0.0.0.0", "port"=>0}, "ip"=>"1.1.1.1", "port"=>53}}], :response=>{"index"=>{"_index"=>"filebeat-8.8.2-2023.07.15", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"pipeline with id [filebeat-8.8.2-cef-log-pipeline] does not exist"}}}}

2

What does your Logstash output looks like? This error message means that it is trying to use an ingest pipeline in Elasticsearch named filebeat-8.8.2-cef-log-pipeline, but this ingest pipeline does not exist.

3

Hi @Kiran_K Welcome to the community.

Perhaps take a look at this Logstash : Working with Filebeat Modules

In set 3 just run this command instead of the one the docs show, it will make sure all the assets are loaded (there may be dashboards etc) when you run setup filebeat must be pointing to Kibana and Elasticsearch

filebeat setup -e

After you run setup, switch filebeat back to point to logstash... also take a look at the longstash.conf on that doc pages that will ensure that the correct pipelines will get called.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.