LOGSTASH - wrong pipeline processor Pipeline

Elastic Stack Logstash
1

Having update a three node Elasticsearch cluster from 8.12.1 to 8.17.3 Logstash is generating error logs containing the following:

, :error=>{"type"=>"illegal_state_exception", "reason"=>"Pipeline processor configured for non-existent pipeline [logs-endpoint.events.process-8.12.0]"}}

The Ingest pipeline is now named logs-endpoint.events.process-8.17.0

Can anyone suggest how i can resolve this?

2

You need to share your logstash configuration for the logstash pipeline that generated this error.

4

Thank you very much for the quick reply.

I am trying to remedy an installation which was done by someone else .

The following three config files are in place on the Logstash server. Other pipeline configuration is apparently managed over the Kibana Web Interface:

  1. logstash/config/conf.d/main.conf
input {
        beats {
                port => 5044
        }
}
output {
        elasticsearch {
                hosts => ["http://10.10.10.1:9200", "http://10.10.10.2:9200", "http://10.10.10.3:9200"]
                index => "main"
        }
}
  1. logstash/config/conf.d/testpipeline.conf
input {
        elastic_agent {
                port => 5044
        }
}
output {
        elasticsearch {
                hosts => ["http://10.10.10.1:9200", "http://10.10.10.2:9200", "http://10.10.10.3:9200"]
                index => "test"
                #api_key => "qwertzuioplkjhgfdsa123456789"
                user => "myuser"
                password => "myuserpassword"
        }
}
  1. logstash/config/pipelines.yml
# List of pipelines to be loaded by Logstash
#
# This document must be a list of dictionaries/hashes, where the keys/values are pipeline settings.
# Default values for omitted settings are read from the `logstash.yml` file.
# When declaring multiple pipelines, each MUST have its own `pipeline.id`.
#
# Example of two pipelines:
#
# - pipeline.id: test
#   pipeline.workers: 1
#   pipeline.batch.size: 1
#   config.string: "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }"
# - pipeline.id: another_test
#   queue.type: persisted
#   path.config: "/tmp/logstash/*.config"
- pipeline.id: test
  path.config: "/elastic/application/logstash/config/conf.d/testpipeline.conf"
  pipeline.workers: 6
  pipeline.batch.size: 250
#
# Available options:
#
#   # name of the pipeline
#   pipeline.id: mylogs

    *** default logstash pipeline.yml file contents, all commented out ***

#   dead_letter_queue.storage_policy: drop_newer

#
#   If using dead_letter_queue.enable: true, the directory path where the data files will be stored.
#   Default is path.data/dead_letter_queue
#
#   path.dead_letter_queue:
5

Can you share the full error log that you shared before? You shared only part of it.

It is not clear what may be generating this error.

6

Hi Leandro

Here is a sample log complete entry.

[2025-03-26T16:15:23
367][INFO ][logstash.outputs.elasticsearch][general-all][2b49dc6d7537e1bac8af69a452fd41023f893d40d14df4cbafe56787d35b872a] Retrying failed action {:status=>500
:action=>["create"
{:_id=>nil
:_index=>"index__server.logs_ip"
:routing=>nil}
{"dataset_name"=>"dhcp.logs"
event"=>{"created"=>"2025-03-25T20:23:10.5221007Z
original"=>"Endpoint process event
action"=>"end
kind"=>"event
dataset"=>"endpoint.events.process
id"=>"Nxq6Z9ckxrbO7h/z++++CKih
sequence"=>836269, "module"=>"endpoint
category"=>["process"], "type"=>["end"], "outcome"=>"unknown"}, "elastic"=>{"agent"=>{"id"=>"1945603bf-21c3-4a1e-805b-19f595452a0c"}}, "process"=>{"pid"=>14972, "executable"=>"C:\\Program Files (x86)\\baramundi\\BMA\\WindowsDefenderAgent\\WindowsDefenderAgent.exe
entity_id"=>"MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTE0OTcyLTE3NDI5MzQxOTAuMzc4ODE3NDAw
exit_code"=>0, "command_line"=>"C:\\Program Files (x86)\\baramundi\\BMA\\WindowsDefenderAgent\\WindowsDefenderAgent.exe
hash"=>{"md5"=>"f55a4fbd596dff3ed72567a5bb764b11
sha1"=>"3c97d6775caabeb8d5de52e4b66ae9ec6701973e
sha256"=>"737f4a5e77430665f99b398f495e892feaf48dfe9709298ec7381c5d503e0bf8"}, "code_signature"=>{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}, "args"=>["C:\\Program\\Files
\\(x86)\\baramundi\\BMA\\WindowsDefenderAgent\\WindowsDefenderAgent.exe"], "args_count"=>3, "name"=>"WindowsDefenderAgent.exe
pe"=>{"imphash"=>"f34d5f2d4577ed6d9ceec516c1f5a744
original_file_name"=>"WindowsDefenderAgent.exe"}, "Ext"=>{"mitigation_policies"=>["CET dynamic APIs can only be called out of proc"], "ancestry"=>["MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTI5NDQtMTc0MjU3MTY1NC4xNjU2MTM0MDA=
MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTc0OC0xNzQyNTcxNjQ5LjI1MjU2MDIwMA==
MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTYzNi0xNzQyNTcxNjQ4LjkwODc2OTIwMA=="], "code_signature"=>[{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}], "session_info"=>{"authentication_package"=>"Negotiate relative_logon_time"=>362540.3753987}}, "parent"=>{"command_line"=>"\"C:\\Program Files (x86)\\baramundi\\BMA\\bma.exe\", "pid"=>2944, "code_signature"=>{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}, "executable"=>"C:\\Program Files (x86)\\baramundi\\BMA\\bma.exe
args"=>["C:\\Program Files (x86)\\baramundi\\BMA\\bma.exe"], "args_count"=>1, "name"=>"bma.exe
Ext"=>{"code_signature"=>[{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}]}, "entity_id"=>"MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTI5NDQtMTc0MjU3MTY1NC4xNjU2MTM0MDA="}}, "data_stream"=>{"namespace"=>"server.logs
type"=>"logs
dataset"=>"endpoint.events.process"}, "host"=>{"mac"=>["00-40-76-39-ad-12"], "id"=>"e0bf74db-98ea-46d4-a9eb-6099928626c3
architecture"=>"x86_64
src"=>"DHCP (local - Windows)
ip"=>["10.10.10.5
fe70::bce:72df7:59c2:279a
127.0.0.1
::1"], "name"=>"dhcpserver2
hostname"=>"dhcpserver2
os"=>{"kernel"=>"21H2 (10.0.20348.3328)
version"=>"21H2 (10.0.20348.3328)
name"=>"Windows
platform"=>"windows
Ext"=>{"variant"=>"Windows Server 2022 Standard"}, "type"=>"windows
family"=>"windows
full"=>"Windows Server 2022 Standard 21H2 (10.0.20348.3328)"}}, "mandant"=>"gv
@version"=>"1
tags"=>["beats_input_codec_plain_applied
_grokparsefailure
netzwerk"], "ecs"=>{"version"=>"8.10.0"}, "agent"=>{"id"=>"193603bf-21c3-4a1e-805b-19f595452a0c
type"=>"endpoint
version"=>"8.12.1"}, "@timestamp"=>2025-03-25T20:23:10.522100700Z, "message"=>"Endpoint process event
user"=>{"name"=>"SYSTEM
domain"=>"NT AUTHORITY
id"=>"S-1-5-18"}}], :error=>{"type"=>"illegal_state_exception
reason"=>"Pipeline processor configured for non-existent pipeline [logs-endpoint.events.process-8.12.0]"}}
7

Hallo Leandro

Here is a sample log complete entry.

[2025-03-26T16:15:23
367][INFO ][logstash.outputs.elasticsearch][general-all][2b49dc6d7537e1bac8af69a452fd41023f893d40d14df4cbafe56787d35b872a] Retrying failed action {:status=>500
:action=>["create"
{:_id=>nil
:_index=>"index__server.logs_ip"
:routing=>nil}
{"dataset_name"=>"dhcp.logs"
event"=>{"created"=>"2025-03-25T20:23:10.5221007Z
original"=>"Endpoint process event
action"=>"end
kind"=>"event
dataset"=>"endpoint.events.process
id"=>"Nxq6Z9ckxrbO7h/z++++CKih
sequence"=>836269, "module"=>"endpoint
category"=>["process"], "type"=>["end"], "outcome"=>"unknown"}, "elastic"=>{"agent"=>{"id"=>"1945603bf-21c3-4a1e-805b-19f595452a0c"}}, "process"=>{"pid"=>14972, "executable"=>"C:\\Program Files (x86)\\baramundi\\BMA\\WindowsDefenderAgent\\WindowsDefenderAgent.exe
entity_id"=>"MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTE0OTcyLTE3NDI5MzQxOTAuMzc4ODE3NDAw
exit_code"=>0, "command_line"=>"C:\\Program Files (x86)\\baramundi\\BMA\\WindowsDefenderAgent\\WindowsDefenderAgent.exe
hash"=>{"md5"=>"f55a4fbd596dff3ed72567a5bb764b11
sha1"=>"3c97d6775caabeb8d5de52e4b66ae9ec6701973e
sha256"=>"737f4a5e77430665f99b398f495e892feaf48dfe9709298ec7381c5d503e0bf8"}, "code_signature"=>{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}, "args"=>["C:\\Program\\Files
\\(x86)\\baramundi\\BMA\\WindowsDefenderAgent\\WindowsDefenderAgent.exe"], "args_count"=>3, "name"=>"WindowsDefenderAgent.exe
pe"=>{"imphash"=>"f34d5f2d4577ed6d9ceec516c1f5a744
original_file_name"=>"WindowsDefenderAgent.exe"}, "Ext"=>{"mitigation_policies"=>["CET dynamic APIs can only be called out of proc"], "ancestry"=>["MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTI5NDQtMTc0MjU3MTY1NC4xNjU2MTM0MDA=
MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTc0OC0xNzQyNTcxNjQ5LjI1MjU2MDIwMA==
MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTYzNi0xNzQyNTcxNjQ4LjkwODc2OTIwMA=="], "code_signature"=>[{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}], "session_info"=>{"authentication_package"=>"Negotiate relative_logon_time"=>362540.3753987}}, "parent"=>{"command_line"=>"\"C:\\Program Files (x86)\\baramundi\\BMA\\bma.exe\", "pid"=>2944, "code_signature"=>{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}, "executable"=>"C:\\Program Files (x86)\\baramundi\\BMA\\bma.exe
args"=>["C:\\Program Files (x86)\\baramundi\\BMA\\bma.exe"], "args_count"=>1, "name"=>"bma.exe
Ext"=>{"code_signature"=>[{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}]}, "entity_id"=>"MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTI5NDQtMTc0MjU3MTY1NC4xNjU2MTM0MDA="}}, "data_stream"=>{"namespace"=>"server.logs
type"=>"logs
dataset"=>"endpoint.events.process"}, "host"=>{"mac"=>["00-40-76-39-ad-12"], "id"=>"e0bf74db-98ea-46d4-a9eb-6099928626c3
architecture"=>"x86_64
src"=>"DHCP (local - Windows)
ip"=>["10.10.10.5
fe70::bce:72df7:59c2:279a
127.0.0.1
::1"], "name"=>"server2
hostname"=>"server2
os"=>{"kernel"=>"21H2 (10.0.20348.3328)
version"=>"21H2 (10.0.20348.3328)
name"=>"Windows
platform"=>"windows
Ext"=>{"variant"=>"Windows Server 2022 Standard"}, "type"=>"windows
family"=>"windows
full"=>"Windows Server 2022 Standard 21H2 (10.0.20348.3328)"}}, "mandant"=>"gv
@version"=>"1
tags"=>["beats_input_codec_plain_applied
_grokparsefailure
netzwerk"], "ecs"=>{"version"=>"8.10.0"}, "agent"=>{"id"=>"193603bf-21c3-4a1e-805b-19f595452a0c
type"=>"endpoint
version"=>"8.12.1"}, "@timestamp"=>2025-03-25T20:23:10.522100700Z, "message"=>"Endpoint process event
user"=>{"name"=>"SYSTEM
domain"=>"NT AUTHORITY
id"=>"S-1-5-18"}}], :error=>{"type"=>"illegal_state_exception
reason"=>"Pipeline processor configured for non-existent pipeline [logs-endpoint.events.process-8.12.0]"}} 
e
8

Hallo Leandro

Here is a complete example log entry.

[2025-03-26T16:15:23
367][INFO ][logstash.outputs.elasticsearch][general-all][2b49dc6d7537e1bac8af69a452fd41023f893d40d14df4cbafe56787d35b872a] Retrying failed action {:status=>500
:action=>["create"
{:_id=>nil
:_index=>"index__server.logs_ip"
:routing=>nil}
{"dataset_name"=>"dhcp.logs"
event"=>{"created"=>"2025-03-25T20:23:10.5221007Z
original"=>"Endpoint process event
action"=>"end
kind"=>"event
dataset"=>"endpoint.events.process
id"=>"Nxq6Z9ckxrbO7h/z++++CKih
sequence"=>836269, "module"=>"endpoint
category"=>["process"], "type"=>["end"], "outcome"=>"unknown"}, "elastic"=>{"agent"=>{"id"=>"1945603bf-21c3-4a1e-805b-19f595452a0c"}}, "process"=>{"pid"=>14972, "executable"=>"C:\\Program Files (x86)\\baramundi\\BMA\\WindowsDefenderAgent\\WindowsDefenderAgent.exe
entity_id"=>"MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTE0OTcyLTE3NDI5MzQxOTAuMzc4ODE3NDAw
exit_code"=>0, "command_line"=>"C:\\Program Files (x86)\\baramundi\\BMA\\WindowsDefenderAgent\\WindowsDefenderAgent.exe
hash"=>{"md5"=>"f55a4fbd596dff3ed72567a5bb764b11
sha1"=>"3c97d6775caabeb8d5de52e4b66ae9ec6701973e
sha256"=>"737f4a5e77430665f99b398f495e892feaf48dfe9709298ec7381c5d503e0bf8"}, "code_signature"=>{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}, "args"=>["C:\\Program\\Files
\\(x86)\\baramundi\\BMA\\WindowsDefenderAgent\\WindowsDefenderAgent.exe"], "args_count"=>3, "name"=>"WindowsDefenderAgent.exe
pe"=>{"imphash"=>"f34d5f2d4577ed6d9ceec516c1f5a744
original_file_name"=>"WindowsDefenderAgent.exe"}, "Ext"=>{"mitigation_policies"=>["CET dynamic APIs can only be called out of proc"], "ancestry"=>["MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTI5NDQtMTc0MjU3MTY1NC4xNjU2MTM0MDA=
MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTc0OC0xNzQyNTcxNjQ5LjI1MjU2MDIwMA==
MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTYzNi0xNzQyNTcxNjQ4LjkwODc2OTIwMA=="], "code_signature"=>[{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}], "session_info"=>{"authentication_package"=>"Negotiate relative_logon_time"=>362540.3753987}}, "parent"=>{"command_line"=>"\"C:\\Program Files (x86)\\baramundi\\BMA\\bma.exe\", "pid"=>2944, "code_signature"=>{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}, "executable"=>"C:\\Program Files (x86)\\baramundi\\BMA\\bma.exe
args"=>["C:\\Program Files (x86)\\baramundi\\BMA\\bma.exe"], "args_count"=>1, "name"=>"bma.exe
Ext"=>{"code_signature"=>[{"exists"=>true, "trusted"=>true, "subject_name"=>"baramundi software
status"=>"trusted"}]}, "entity_id"=>"MTkzNjAzYmYtMjFjMy00YTFlLTgwNWItMTlmNTk1NDUyYTBjLTI5NDQtMTc0MjU3MTY1NC4xNjU2MTM0MDA="}}, "data_stream"=>{"namespace"=>"server.logs
type"=>"logs
dataset"=>"endpoint.events.process"}, "host"=>{"mac"=>["00-40-76-39-ad-12"], "id"=>"e0bf74db-98ea-46d4-a9eb-6099928626c3
architecture"=>"x86_64
src"=>"DHCP (local - Windows)
ip"=>["10.10.10.5
fe70::bce:72df7:59c2:279a
127.0.0.1
::1"], "name"=>"server2
hostname"=>"server2
os"=>{"kernel"=>"21H2 (10.0.20348.3328)
version"=>"21H2 (10.0.20348.3328)
name"=>"Windows
platform"=>"windows
Ext"=>{"variant"=>"Windows Server 2022 Standard"}, "type"=>"windows
family"=>"windows
full"=>"Windows Server 2022 Standard 21H2 (10.0.20348.3328)"}}, "mandant"=>"gv
@version"=>"1
tags"=>["beats_input_codec_plain_applied
_grokparsefailure
netzwerk"], "ecs"=>{"version"=>"8.10.0"}, "agent"=>{"id"=>"193603bf-21c3-4a1e-805b-19f595452a0c
type"=>"endpoint
version"=>"8.12.1"}, "@timestamp"=>2025-03-25T20:23:10.522100700Z, "message"=>"Endpoint process event
user"=>{"name"=>"SYSTEM
domain"=>"NT AUTHORITY
id"=>"S-1-5-18"}}], :error=>{"type"=>"illegal_state_exception
reason"=>"Pipeline processor configured for non-existent pipeline [logs-endpoint.events.process-8.12.0]"}}
9

Hi @Echo_01 please Be patient when posting your fairly new user and sometimes it takes a couple minutes for the post to show up

10

Hi Stephen - thanks for the info -noted

11

Hi Leandro.
The full log is now in the post - the post was temporarily blocked by Akismet.
Any further help gratefully received

12

Bump. Does anyone have an idea how i can resolve this problem?

13

Hi @Echo_01

Here is what I see from the information above... and so there is bit of a mystery and you are going to need to dig in..

:action=>["create"
{:_id=>nil
:_index=>"index__server.logs_ip". <<< THIS A
....
....
:error=>{"type"=>"illegal_state_exception
reason"=>"Pipeline processor configured for 
non-existent pipeline [logs-endpoint.events.process-8.12.0]"}}  <<< THIS B

THIS A: Means the documents are set with the index name index__server.logs_ip which is a quite unusual name AND according to you logstash configuration indicates the THIS C indicates that index should be test Which leads to question if you are running the actual logstash pipeline you think you are, since there you did not provide the startup logs we are not sure of this

output {
        elasticsearch {
                hosts => ["http://10.10.10.1:9200", "http://10.10.10.2:9200", "http://10.10.10.3:9200"]
                index => "test" <<< THIS C

Now THIS B above says that the .... Ingest Pipeline .... logs-endpoint.events.process-8.12.0 which is in Elasticsearch NOT logstash can not be found and that is why the document is failing to index into elasticsearch

You can check that by going to Kibana -> Stack Management -> Ingest Pipelines and check to see...

So there are several issues.... so you would need to back way up to the begining and tell us what you are trying to acomplish and what components you are using to do it.

Guess in it looks like perhaps you are using Elastic Agent / Defend and trying to send that data through Elasticsearch which may or may not have the integrations loaded but that is only a guess.

Please back up to the beginning and try to be very clear on:

  • What you are trying to accomplish?
  • What Elastic Components you are Using?
  • What Integrations you are using?
  • What Versions of All
  • What documentation you are following?
  • The Ingest Architecture Example : Elastic Agent -> Logstash -> Elasticsearch