Pipeline with id [packetbeat-8.15.2-routing] does not exist, dropping event!

jimB100 · October 2, 2024, 2:54pm

Hi, I'm testing out packetbeat to ship dns data to our self-hosted elastic server, I have tried version x64 windows 8.15.2 and 8.15.0 and neither version created the routing pipeline

{"log.level":"warn","@timestamp":"2024-10-02T15:38:09.327+0100","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus","file.name":"elasticsearch/client.go","file.line":489},"message":"Cannot index event '/// snipped packet data /// (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [packetbeat-8.15.2-routing] does not exist"}, dropping event!","service.name":"packetbeat","log.type":"event","ecs.version":"1.6.0"}

i do appear to have the -default and -geoip pipelines though
i have added every role going to the user account, so i'm assuming it's not a permissions issue as it does create the other pipelines

this is my first foray into beats and elastic, any idea what is going wrong here?

Cheers

leandrojmp · October 2, 2024, 3:37pm

Did you ran the packetbeat setup before to install the ingest pipelines and dashboards?

jimB100 · October 2, 2024, 3:58pm

hi, yes i ran the setup, didn't see any errors in the output
I do have some pipelines, just not the one it wants...

stephenb · October 2, 2024, 4:47pm

Hi @jimB100 Welcome to the community

How did you install apckbeat and how did you run setup? ... did you let it finish?

I just create a fresh Elastic / Kibana / Packetbeat 8.15.2

./packetbeat setup -e

Here are the all the ingest pipelines.... perhaps run setup again...including the routing one...

Can you rerun setup capture the output and share...

stephenb · October 2, 2024, 5:04pm

If you run

.\packetbeat setup --pipelines -e -d "*"

You should see all the pipelines get loaded , there will be some odd debug messages but that is ok...

If you run this again then it will show each is loaded...

Seem like it did not finish for you..

jimB100 · October 3, 2024, 7:28am

hi @stephenb i'll give the --pipelines switch a go and let you know how i get on, cheers!

jimB100 · October 3, 2024, 8:34am

no good, i'm afraid - still only got 2 pipelines...

here's the output from the command, i can't see anything obvious in it, but i don't really know what i'm looking for other than something saying error!

PS C:\Program Files\packetbeat> .\packetbeat setup --pipelines -e -d "*"
{"log.level":"info","@timestamp":"2024-10-03T09:12:28.553+0100","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure","file.name":"instance/beat.go","file.line":828},"message":"Home path: [C:\\Program Files\\packetbeat] Config path: [C:\\Program Files\\packetbeat] Data path: [C:\\Program Files\\packetbeat\\data] Logs path: [C:\\Program Files\\packetbeat\\logs]","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:28.553+0100","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).loadMeta","file.name":"instance/beat.go","file.line":950},"message":"Beat metadata path: C:\\Program Files\\packetbeat\\data\\meta.json","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-03T09:12:28.553+0100","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure","file.name":"instance/beat.go","file.line":836},"message":"Beat ID: 5364af0a-7ba5-4d3b-99a4-9277765b4a8a","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:28.573+0100","log.logger":"conditions","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/conditions.NewCondition","file.name":"conditions/conditions.go","file.line":98},"message":"New condition contains: map[]","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:28.574+0100","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors.New","file.name":"processors/processor.go","file.line":114},"message":"Generated new processors: drop_fields={\"Fields\":[\"host\"],\"RegexpFields\":[],\"IgnoreMissing\":false}","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:28.591+0100","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors.New","file.name":"processors/processor.go","file.line":114},"message":"Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]]","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:28.592+0100","log.logger":"docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/docker.NewClient","file.name":"docker/client.go","file.line":49},"message":"Docker client will negotiate the API version on the first request.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:28.593+0100","log.logger":"add_docker_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_docker_metadata.buildDockerMetadataProcessor","file.name":"add_docker_metadata/add_docker_metadata.go","file.line":89},"message":"add_docker_metadata: docker environment not detected: protocol not available","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:28.592+0100","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata","file.name":"add_cloud_metadata/providers.go","file.line":147},"message":"add_cloud_metadata: starting to fetch metadata, timeout=3s","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.594+0100","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata","file.name":"add_cloud_metadata/providers.go","file.line":193},"message":"add_cloud_metadata: timed-out waiting for all responses","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.595+0100","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata.func1","file.name":"add_cloud_metadata/providers.go","file.line":150},"message":"add_cloud_metadata: fetchMetadata ran for 3.0010304s","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.602+0100","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1","file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":100},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.604+0100","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors.New","file.name":"processors/processor.go","file.line":114},"message":"Generated new processors: if contains: map[] then drop_fields={\"Fields\":[\"host\"],\"RegexpFields\":[],\"IgnoreMissing\":false} else add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], add_cloud_metadata={}, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.parent.pid]], detect_mime_type=http.request.body.content->http.request.mime_type, detect_mime_type=http.response.body.content->http.response.mime_type","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.605+0100","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo","file.name":"instance/beat.go","file.line":1385},"message":"Beat info","service.name":"packetbeat","system_info":{"beat":{"path":{"config":"C:\\Program Files\\packetbeat","data":"C:\\Program Files\\packetbeat\\data","home":"C:\\Program Files\\packetbeat","logs":"C:\\Program Files\\packetbeat\\logs"},"type":"packetbeat","uuid":"5364af0a-7ba5-4d3b-99a4-9277765b4a8a"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.607+0100","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo","file.name":"instance/beat.go","file.line":1394},"message":"Build info","service.name":"packetbeat","system_info":{"build":{"commit":"26daf71e4ec87172523af7f0e916cba9f79dc0d0","libbeat":"8.15.2","time":"2024-09-19T09:33:49.000Z","version":"8.15.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.608+0100","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo","file.name":"instance/beat.go","file.line":1397},"message":"Go runtime info","service.name":"packetbeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":4,"version":"go1.22.6"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.619+0100","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo","file.name":"instance/beat.go","file.line":1403},"message":"Host info","service.name":"packetbeat","system_info":{"host":{"architecture":"x86_64","native_architecture":"x86_64","boot_time":"2024-10-02T15:28:19+01:00","name":"redacted-client-name","ip":["fe80::3fcc:dc1f:9e71:a1b3","10.31.4.53","::1","127.0.0.1"],"kernel_version":"10.0.19041.4780 (WinBuild.160101.0800)","mac":["00:50:56:b9:8f:3a"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"19045.4780"},"timezone":"BST","timezone_offset_sec":3600,"id":"a8712cd6-1132-43ed-bde6-73484e1b9ecb"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.620+0100","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo","file.name":"instance/beat.go","file.line":1432},"message":"Process info","service.name":"packetbeat","system_info":{"process":{"cwd":"C:\\Program Files\\packetbeat","exe":"C:\\Program Files\\packetbeat\\packetbeat.exe","name":"packetbeat.exe","pid":10212,"ppid":2400,"start_time":"2024-10-03T09:12:28.426+0100"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.621+0100","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater","file.name":"instance/beat.go","file.line":341},"message":"Setup Beat: packetbeat; Version: 8.15.2","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.632+0100","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater","file.name":"instance/beat.go","file.line":369},"message":"Initializing output plugins","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.648+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection","file.name":"eslegclient/connection.go","file.line":133},"message":"elasticsearch url: https://redacted-server-name:9200","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-10-03T09:12:31.649+0100","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig","file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.649+0100","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).run","file.name":"pipeline/consumer.go","file.line":110},"message":"start pipeline event consumer","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.649+0100","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*queueReader).run","file.name":"pipeline/queue_reader.go","file.line":49},"message":"pipeline event consumer queue reader: start","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.650+0100","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings","file.name":"pipeline/module.go","file.line":105},"message":"Beat name: redacted-client-name","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.653+0100","log.logger":"npcap","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/beater.installNpcap.func1","file.name":"beater/install_npcap.go","file.line":56},"message":"npcap version: Npcap version 1.79, based on libpcap version 1.10.4","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.656+0100","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/procs.(*ProcessesWatcher).init","file.name":"procs/procs.go","file.line":114},"message":"Process watcher disabled","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.663+0100","log.logger":"flows","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/flows.newFlowsWorker","file.name":"flows/worker.go","file.line":159},"message":"new flows worker. timeout=30s, period=10s, tick=10s, ticksTO=3, ticksP=1","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.663+0100","log.logger":"main","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/beater.setupSniffer","file.name":"beater/processor.go","file.line":234},"message":"Initializing protocol plugins","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.664+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: sip","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.665+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: mysql","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.665+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: nfs","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.666+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: amqp","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.666+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: dns","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.667+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: pgsql","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.668+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: tls","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.668+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: mongodb","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.669+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: thrift","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.669+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: dhcpv4","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.670+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: cassandra","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.670+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: http","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.671+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: memcache","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.671+0100","log.logger":"protos","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos.ProtocolsStruct.InitFiltered","file.name":"protos/protos.go","file.line":123},"message":"registered protocol plugin: redis","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.675+0100","log.logger":"memcache","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos/memcache.(*memcache).init","file.name":"memcache/memcache.go","file.line":157},"message":"init memcache plugin","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.679+0100","log.logger":"memcache","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos/memcache.(*memcache).setFromConfig","file.name":"memcache/memcache.go","file.line":187},"message":"transaction timeout: 10s","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.680+0100","log.logger":"memcache","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos/memcache.(*memcache).setFromConfig","file.name":"memcache/memcache.go","file.line":188},"message":"udp transaction timeout: 10s","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.683+0100","log.logger":"memcache","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos/memcache.(*memcache).setFromConfig","file.name":"memcache/memcache.go","file.line":189},"message":"maxValues = 0","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.684+0100","log.logger":"memcache","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos/memcache.(*memcache).setFromConfig","file.name":"memcache/memcache.go","file.line":190},"message":"maxBytesPerValue = 2147483647","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.692+0100","log.logger":"mongodb","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos/mongodb.(*mongodbPlugin).init","file.name":"mongodb/mongodb.go","file.line":87},"message":"Init a MongoDB protocol parser","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-10-03T09:12:31.695+0100","log.logger":"cfgwarn","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/protos/sip.New","file.name":"sip/plugin.go","file.line":68},"message":"BETA: packetbeat SIP protocol is used","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.696+0100","log.logger":"sniffer","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/sniffer.New","file.name":"sniffer/sniffer.go","file.line":114},"message":"interface: 0, BPF filter: ''","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.724+0100","log.logger":"sniffer","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/sniffer.New","file.name":"sniffer/sniffer.go","file.line":155},"message":"Sniffer type: pcap device: \\Device\\NPF_{47EB5153-834B-47D0-A8DC-1D51F647258F}","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.724+0100","log.logger":"sniffer","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/sniffer.(*Sniffer).Stop","file.name":"sniffer/sniffer.go","file.line":483},"message":"sending stop to all sniffers","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.728+0100","log.logger":"sniffer","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/sniffer.(*Sniffer).Stop","file.name":"sniffer/sniffer.go","file.line":485},"message":"sending closing to default_route","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.729+0100","log.logger":"flows","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/flows.(*worker).stop","file.name":"flows/worker.go","file.line":77},"message":"stop flows worker","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.729+0100","log.logger":"flows","log.origin":{"function":"github.com/elastic/beats/v7/packetbeat/flows.(*worker).stop","file.name":"flows/worker.go","file.line":80},"message":"stopped flows worker","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.730+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection","file.name":"eslegclient/connection.go","file.line":133},"message":"elasticsearch url: https://redacted-server-name:9200","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-10-03T09:12:31.730+0100","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig","file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.731+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":302},"message":"ES Ping(url=https://redacted-server-name:9200)","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-10-03T09:12:31.732+0100","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig","file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.753+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2","file.name":"transport/logging.go","file.line":43},"message":"Completed dialing successfully","service.name":"packetbeat","network":"tcp","address":"redacted-server-name:9200","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.755+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":321},"message":"Ping status code: 200","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.757+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":322},"message":"Attempting to connect to Elasticsearch version 8.15.1 (default)","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.771+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.774+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"PUT https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  map[description:Pipeline for processing amqp traffic on_failure:[map[append:map[field:error.message value:Processor \"{{ _ingest.on_failure_processor_type }}\" with tag \"{{ _ingest.on_failure_processor_tag }}\" in pipeline \"{{ _ingest.on_failure_pipeline }}\" failed with message \"{{ _ingest.on_failure_message }}\"]] map[set:map[field:event.kind value:pipeline_error]]] processors:[map[set:map[field:ecs.version value:8.11.0]] map[gsub:map[field:host.mac ignore_missing:true pattern:[-:.] replacement: tag:gsubmac]] map[gsub:map[field:host.mac ignore_missing:true pattern:(..)(?!$) replacement:$1- tag:gsubmac]] map[uppercase:map[field:host.mac ignore_missing:true]] map[append:map[allow_duplicates:false field:related.hosts if:ctx.observer?.hostname != null && ctx.observer?.hostname != '' value:{{{observer.hostname}}}]] map[foreach:map[field:observer.ip if:ctx.observer?.ip != null && ctx.observer.ip instanceof List processor:map[append:map[allow_duplicates:false field:related.ip value:{{{_ingest._value}}}]] tag:foreachip]] map[remove:map[field:host if:ctx.host != null && ctx.tags != null && ctx.tags.contains('forwarded')]] map[pipeline:map[if:ctx._conf?.geoip_enrich != null && ctx._conf.geoip_enrich name:{{ IngestPipeline \"geoip\" }} tag:pipelineprocessor]] map[remove:map[field:_conf ignore_missing:true]]]]","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.884+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":135},"message":"Elasticsearch pipeline loaded.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.884+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.890+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"PUT https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  map[description:GeoIP enrichment. on_failure:[map[append:map[field:error.message value:Processor \"{{ _ingest.on_failure_processor_type }}\" with tag \"{{ _ingest.on_failure_processor_tag }}\" in pipeline \"{{ _ingest.on_failure_pipeline }}\" failed with message \"{{ _ingest.on_failure_message }}\"]] map[set:map[field:event.kind value:pipeline_error]]] processors:[map[geoip:map[field:source.ip ignore_missing:true tag:source_geo target_field:source.geo]] map[geoip:map[database_file:GeoLite2-ASN.mmdb field:source.ip ignore_missing:true properties:[asn organization_name] tag:source_geo target_field:source.as]] map[rename:map[field:source.as.asn ignore_missing:true target_field:source.as.number]] map[rename:map[field:source.as.organization_name ignore_missing:true target_field:source.as.organization.name]] map[geoip:map[field:destination.ip ignore_missing:true tag:destination_geo target_field:destination.geo]] map[geoip:map[database_file:GeoLite2-ASN.mmdb field:destination.ip ignore_missing:true properties:[asn organization_name] tag:destination_geo target_field:destination.as]] map[rename:map[field:destination.as.asn ignore_missing:true target_field:destination.as.number]] map[rename:map[field:destination.as.organization_name ignore_missing:true target_field:destination.as.organization.name]] map[geoip:map[field:server.ip ignore_missing:true tag:server_geo target_field:server.geo]] map[geoip:map[database_file:GeoLite2-ASN.mmdb field:server.ip ignore_missing:true properties:[asn organization_name] tag:server_geo target_field:server.as]] map[rename:map[field:server.as.asn ignore_missing:true target_field:server.as.number]] map[rename:map[field:server.as.organization_name ignore_missing:true target_field:server.as.organization.name]] map[geoip:map[field:client.ip ignore_missing:true tag:client_geo target_field:client.geo]] map[geoip:map[database_file:GeoLite2-ASN.mmdb field:client.ip ignore_missing:true properties:[asn organization_name] tag:client_geo target_field:client.as]] map[rename:map[field:client.as.asn ignore_missing:true target_field:client.as.number]] map[rename:map[field:client.as.organization_name ignore_missing:true target_field:client.as.organization.name]]]]","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-03T09:12:31.984+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":135},"message":"Elasticsearch pipeline loaded.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:31.998+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.004+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.004+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.007+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.007+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.010+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.011+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.013+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.019+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.021+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.021+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.024+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.024+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.026+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.026+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.027+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.027+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.034+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.034+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.035+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.036+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.037+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.038+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.039+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.040+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.042+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.042+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.043+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.047+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.050+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.050+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.052+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.052+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.053+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.053+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.055+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.055+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.057+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.057+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.058+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.063+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.065+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.065+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.067+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.067+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.069+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.069+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.070+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.071+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.072+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.073+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.074+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.080+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.082+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.083+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.085+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.085+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.087+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.087+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-default  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.088+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.088+0100","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Request","file.name":"eslegclient/connection.go","file.line":377},"message":"GET https://redacted-server-name:9200/_ingest/pipeline/packetbeat-8.15.2-geoip  <nil>","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-10-03T09:12:32.090+0100","log.logger":"pipeline","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline","file.name":"fileset/pipelines.go","file.line":122},"message":"Pipeline already exists in Elasticsearch.","service.name":"packetbeat","ecs.version":"1.6.0"}
Loaded Ingest pipelines
PS C:\Program Files\packetbeat>

stephenb · October 3, 2024, 9:40pm

OK Weird... I just actually ran this on Windows, and I am seeing the same thing, only 2 pipelines....

Let me look and try to get back

stephenb · October 3, 2024, 10:09pm

@jimB100

So here is what I know...

Something is not right... and it it not you .. I see the same thing

If you can find a mac or linux box and download packetbeat and run the full setup it will load everything correct.

.. why the windows binary does not I am not sure ...

Then you should be able to run the window packetbeat ...

I did this and it worked...

I put an internal conversation in but not sure when I will hear back

This is data from my windows box after I ran setup elsewhere

stephenb · October 4, 2024, 3:40am

@jimB100 Yup bug...

github.com/elastic/beats

packetbeat/module: fix upload of bundled ingest pipelines on Windows

elastic:main ← efd6:packetbeat_modules_slash

opened 02:43AM - 04 Oct 24 UTC

efd6

+5 -2

Detection of data stream identity was using os.PathSeparator which will not matc…h the path separator used by embed.FS which is always "/"[1]. [1]https://pkg.go.dev/embed#hdr-Directives  ## Proposed commit message  ## Checklist  - [ ] My code follows the style guidelines of this project - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] I have made corresponding change to the default configuration files - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have added an entry in `CHANGELOG.next.asciidoc` or `CHANGELOG-developer.next.asciidoc`. ## Disruptive User Impact  ## Author's Checklist  - [ ] ## How to test this PR locally  ## Related issues  - ## Use cases  ## Screenshots  ## Logs

Time being you will need to use my workaround

jimB100 · October 4, 2024, 7:24am

brilliant, thanks @stephenb - appreciate you looking into it

i'll find a linux box to run the setup on

jimB100 · October 4, 2024, 1:14pm

packetbeat installed on linux and all working great!

Topic		Replies	Views
Pipeline with id does not exist Elasticsearch	3	3524	December 24, 2020
Pipeline not created during setup of packetbeat 8.15.1 on windows Beats packetbeat	4	37	October 15, 2024
"Pipeline does not exist" issue with sonicwall filebeat module Logstash ingest-pipeline	2	521	May 31, 2021
Issues with winlogbeat set up Beats winlogbeat	2	585	May 31, 2022
Failed to execute pipeline PIPELINE DOES NOT EXIST Elasticsearch	2	2704	August 7, 2017

Pipeline with id [packetbeat-8.15.2-routing] does not exist, dropping event!

Related topics