Could not execute action: PipelineAction::Create<main>

Elastic Stack Logstash
1

Hello,

Logstash isn't able to create the index, that's what my conclusion is after checking the logs.

pipeline.yml

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/1.conf"

1.conf:

input {
        beats {
                port => 5044
        }
}

output {
  elasticsearch {
        hosts => ["https://172.16.1.226:9200"]
        index => "testindex1000"
        cacert => '/etc/logstash/http_ca.crt'
        user => "elastic"
        password => "********"
        }
stdout { codec => rubydebug }
}

[2022-02-28T14:52:12,476][INFO ][logstash.runner ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2022-02-28T14:52:12,486][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.0.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2022-02-28T14:52:12,488][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-02-28T14:52:14,039][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-02-28T14:52:14,985][INFO ][org.reflections.Reflections] Reflections took 53 ms to scan 1 urls, producing 120 keys and 417 values
[2022-02-28T14:52:15,894][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2022-02-28T14:52:15,940][INFO ][logstash.outputs.Elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["https://debian-elk-log:9200"]}
[2022-02-28T14:52:16,080][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Errno::EACCES: Permission denied - /etc/logstash/http_ca.crt>, :backtrace=>["org/jruby/RubyIO.java:1237:in sysopen'", "org/jruby/RubyFile.java:365:in initialize'", "org/jruby/RubyIO.java:1156:in open'", "org/jruby/RubyKernel.java:317:in open'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/open-uri.rb:37:in open'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.8.0-java/lib/manticore/client.rb:645:in setup_trust_store'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.8.0-java/lib/manticore/client.rb:633:in ssl_socket_factory_from_options'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.8.0-java/lib/manticore/client.rb:397:in pool_builder'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.8.0-java/lib/manticore/client.rb:405:in pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.8.0-java/lib/manticore/client.rb:208:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-Elasticsearch-11.4.1-java/lib/logstash/outputs/Elasticsearch/http_client/manticore_adapter.rb:26:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:325:in build_adapter'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-Elasticsearch-11.4.1-java/lib/logstash/outputs/Elasticsearch/http_client.rb:341:in build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-Elasticsearch-11.4.1-java/lib/logstash/outputs/Elasticsearch/http_client_builder.rb:106:in create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-Elasticsearch-11.4.1-java/lib/logstash/plugin_mixins/Elasticsearch/common.rb:34:in build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch.rb:279:in register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in block in register_plugins'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/1.conf"], :thread=>"#<Thread:0x47301572 run>"}
[2022-02-28T14:52:16,084][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-02-28T14:52:16,099][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2022-02-28T14:52:16,178][INFO ][logstash.runner ] Logstash shut down.
[2022-02-28T14:52:16,188][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

Any help much appreciated, Thanks!

2

It is getting an exception here, when it tries to open the CA file. It does not have permission to read the file. It doesn't get as far as trying to connect to elasticsearch.

3

Ahhh yes, I was so fixated on the 'Could not execute action' line, that I totally missed that.
I'll adjust the permissions and update in here if it resovled it :+1:
Thanks!

4

Ok so that one error is resolved, now I'm at this point:

root@debian-elk-log:/home/elkadmin# sudo tail -f /var/log/logstash/logstash-plain.log [2022-03-01T07:59:08,993][WARN ][logstash.runner ] SIGTERM received. Shutting down.
[2022-03-01T07:59:15,480][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-03-01T07:59:16,139][INFO ][logstash.runner ] Logstash shut down.
[2022-03-01T08:03:42,362][INFO ][logstash.runner ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2022-03-01T08:03:42,369][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.0.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2022-03-01T08:03:42,371][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-03-01T08:03:43,961][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-03-01T08:03:44,398][INFO ][org.reflections.Reflections] Reflections took 55 ms to scan 1 urls, producing 120 keys and 417 values
[2022-03-01T08:03:45,134][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2022-03-01T08:03:45,172][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["https://172.16.1.226:9200"]}
[2022-03-01T08:03:45,449][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://elastic:xxxxxx@172.16.1.226:9200/]}}
[2022-03-01T08:03:45,793][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@172.16.1.226:9200/"}
[2022-03-01T08:03:45,806][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.0.0) {:es_version=>8}
[2022-03-01T08:03:45,808][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2022-03-01T08:03:45,854][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
[2022-03-01T08:03:45,856][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with ecs_compatibility => v8, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-03-01T08:03:45,930][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-03-01T08:03:45,931][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/1.conf"], :thread=>"#<Thread:0x17b44762 run>"}
[2022-03-01T08:03:46,500][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.57}
[2022-03-01T08:03:46,526][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2022-03-01T08:03:46,544][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-03-01T08:03:46,633][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2022-03-01T08:03:46,664][INFO ][org.logstash.beats.Server][main][1743e77c3c9db85cbe456c83609b1e79c45441fcbfd3607baf023c5de24fcdcc] Starting server on port: 5044

I am using Elasticsearch 8, if that makes a difference in troubleshooting.

EDIT
logstash does not seem to start the server, there is nothing listening on port 5044 when running
netstat -tulpn | grep LISTEN

5

There are no errors there and logstash is saying it started listening.

6

Got it!

Almost there. Last issue I'm having, which I didn't have in 7.x due to security features not auto enabled in 7...when trying have Filebeat (on a windows machine) send data to Elasticsearch or Logstash, I receive a 'x509: certificate signed by unknown authority. I can replicate this exact error when running:
.\filebeat.exe setup --index-management via powershell.

Now I tried to add the http_ca.crt to the filebeat.yml file in multiple different ways as per some different documentations I have found, but to no avail.
Simple question; where in the filebeat.yml file do I need to define what? Elasticsearch generated the http_ca.crt, http.p12 and transport.p12 files upon installation.

I'm thinking of just ditching 8.x and going back yo 7.x, which is/was much simpler to configure I find.

thanks!

7

You should take that issue to the filebeat forum, it is not related to logstash.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.