LogStash 403 error to Elastic

Elastic Stack Logstash
1

I am trying to ingest data from wazuh to elastic using this .config file inside logstash

input {
  opensearch {
   hosts =>  ["[IP:PORT]"]
   user  =>  "[USER]"
   password  =>  "[PW]"
   index =>  "wazuh-alerts-*"
   ssl => true
   ca_file => "/usr/share/logstash/certs/root-ca-wazuh.pem"
   query => '{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "rule.level": {
              "value": "9"
            }
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": "now-1h"
            }
          }
        }
      ]
    }
  }
}'
}
}

output {
    elasticsearch {
         hosts => ["[IP:PORT]"]
         index  => "wazuh-alerts-%{+YYYY.MM.dd}"
         user => 'logstash'
         password => '[PW]'
         ssl_enabled => true
         ssl_certificate => "/usr/share/logstash/certs/ca/ca.crt"
         ssl_key => "/usr/share/logstash/certs/ca/ca.pkcs8.key"
         ssl_verification_mode => none
         template => "/usr/share/logstash/conf.d/templates/wazuh.json"
         template_name => "wazuh"
         template_overwrite => true
         data_stream => false
    }
}

But once i start logstash on docker the following error appears:

logstash01_1  | The client is unable to verify distribution due to security privileges on the server side. Some functionality may not be compatible if the server is running an unsupported product.
logstash01_1  | [2024-07-10T15:27:00,900][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<OpenSearch::Transport::Transport::Errors::Forbidden: [403] >, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/opensearch-ruby-3.3.0/lib/opensearch/transport/transport/base.rb:227:in `__raise_transport_error'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/opensearch-ruby-3.3.0/lib/opensearch/transport/transport/base.rb:349:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/opensearch-ruby-3.3.0/lib/opensearch/transport/transport/http/manticore.rb:93:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/opensearch-ruby-3.3.0/lib/opensearch/transport/client.rb:191:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/opensearch-ruby-3.3.0/lib/opensearch.rb:48:in `method_missing'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/opensearch-ruby-3.3.0/lib/opensearch/api/actions/ping.rb:46:in `ping'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-opensearch-1.0.0/lib/logstash/inputs/opensearch.rb:389:in `test_connection!'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-opensearch-1.0.0/lib/logstash/inputs/opensearch.rb:224:in `register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237:in `block in register_plugins'", "org/jruby/RubyArray.java:1989:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:395:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:320:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/conf.d/wazuh-elasticsearch.conf"], :thread=>"#<Thread:0x3a3154fb /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}

I don't know if it is a logstash user permission issue, even if I don't think so due to having tried with superuser also. I am using ELK stack version 8.12.2

What could it be?

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See What is OpenSearch and the OpenSearch Dashboard? | Elastic for more details.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

Removed #opensearch

4

Please share the complete logs, you will have other logs before this line here:

The client is unable to verify distribution

Are you using Elasticsearch or Opensearch in the destination? This error normally happens when you are using the elasticsearch output, but the destination is not elasticsearch.

5

That is the important remaining part i guess:

logstash01_1  | [2024-07-10T18:45:14,446][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
logstash01_1  | [2024-07-10T18:45:14,457][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.3.200.7:9200"]}
logstash01_1  | [2024-07-10T18:45:14,462][WARN ][logstash.outputs.elasticsearch][main] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure set `ssl_verification_mode => full`
logstash01_1  | [2024-07-10T18:45:14,627][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash:xxxxxx@10.3.200.7:9200/]}}
logstash01_1  | [2024-07-10T18:45:14,882][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash:xxxxxx@10.3.200.7:9200/"}
logstash01_1  | [2024-07-10T18:45:14,883][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.12.2) {:es_version=>8}
logstash01_1  | [2024-07-10T18:45:14,883][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
logstash01_1  | [2024-07-10T18:45:14,898][INFO ][logstash.outputs.elasticsearch][main] Using mapping template from {:path=>"/usr/share/logstash/conf.d/templates/wazuh.json"}
logstash01_1  | [2024-07-10T18:45:14,901][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["/usr/share/logstash/conf.d/wazuh-elasticsearch.conf"], :thread=>"#<Thread:0x3446fcd3 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
logstash01_1  | [2024-07-10T18:45:14,916][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"wazuh"}
logstash01_1  | [2024-07-10T18:45:15,389][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.49}
logstash01_1  | The client is unable to verify distribution due to security privileges on the server side. Some functionality may not be compatible if the server is running an unsupported product.

Destination is Elasticsearch (output).

6

Check does the user logstash has write permissions on that index and the wazuh template creation.

7

logstash user has the logstash_system permission plus a custom role with Cluster privileges: manage_index_templates and monitor and as Index privileges write, create and create_index on wazuh-*

8

Also, you have this issue, a 403 error from opensearch, check the credentials and your opensearch logs.

Authentication in Opensearch and Elasticsearch are completely different things.

This is on Elasticsearch side, but your logs shows error on the Opensearch side, which is not supported here, so you need to validate the credentials and check the logs.

10

I've decided to centrally manage pipeline directly on kibana. Now I have fixed the permission errors on opensearch (wazuh) side but I am still unable to view results on the kibana Discovery section. This is the output I get logstash side:

logstash01_1  | [2024-07-20T10:15:54,800][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:wazuh}
logstash01_1  | [2024-07-20T10:15:54,826][INFO ][logstash.javapipeline    ] Pipeline `wazuh` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
logstash01_1  | [2024-07-20T10:15:54,827][WARN ][org.logstash.execution.AbstractPipelineExt] Metric registration error: `input_throughput` could not be registered in namespace `[:stats, :pipelines, :wazuh, :flow]`
logstash01_1  | [2024-07-20T10:15:54,827][WARN ][org.logstash.execution.AbstractPipelineExt] Metric registration error: `filter_throughput` could not be registered in namespace `[:stats, :pipelines, :wazuh, :flow]`
logstash01_1  | [2024-07-20T10:15:54,827][WARN ][org.logstash.execution.AbstractPipelineExt] Metric registration error: `output_throughput` could not be registered in namespace `[:stats, :pipelines, :wazuh, :flow]`
logstash01_1  | [2024-07-20T10:15:54,827][WARN ][org.logstash.execution.AbstractPipelineExt] Metric registration error: `queue_backpressure` could not be registered in namespace `[:stats, :pipelines, :wazuh, :flow]`
logstash01_1  | [2024-07-20T10:15:54,827][WARN ][org.logstash.execution.AbstractPipelineExt] Metric registration error: `worker_concurrency` could not be registered in namespace `[:stats, :pipelines, :wazuh, :flow]`
logstash01_1  | [2024-07-20T10:15:54,827][WARN ][org.logstash.execution.AbstractPipelineExt] Metric registration error: `worker_utilization` could not be registered in namespace `[:stats, :pipelines, :wazuh, :flow]`
logstash01_1  | [2024-07-20T10:15:54,827][WARN ][org.logstash.execution.AbstractPipelineExt] Metric registration error: `throughput` could not be registered in namespace `[:stats, :pipelines, :wazuh, :plugins, :inputs, :"2eacf48fc21422d16a48446428c0bb90b53cf2bc73fa7f433baa2b28a6d29edc", :flow]`
logstash01_1  | [2024-07-20T10:15:54,827][WARN ][org.logstash.execution.AbstractPipelineExt] Metric registration error: `worker_millis_per_event` could not be registered in namespace `[:stats, :pipelines, :wazuh, :plugins, :outputs, :a32b883f077023054a52c21cfd106049dbc019a0a46f63fa00fac0f9136169e9, :flow]`
logstash01_1  | [2024-07-20T10:15:54,827][WARN ][org.logstash.execution.AbstractPipelineExt] Metric registration error: `worker_utilization` could not be registered in namespace `[:stats, :pipelines, :wazuh, :plugins, :outputs, :a32b883f077023054a52c21cfd106049dbc019a0a46f63fa00fac0f9136169e9, :flow]`
logstash01_1  | [2024-07-20T10:15:54,828][INFO ][logstash.outputs.elasticsearch][wazuh] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.3.200.7:9200"]}
logstash01_1  | [2024-07-20T10:15:54,829][WARN ][logstash.outputs.elasticsearch][wazuh] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure set `ssl_verification_mode => full`
logstash01_1  | [2024-07-20T10:15:54,871][INFO ][logstash.outputs.elasticsearch][wazuh] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash:xxxxxx@10.3.200.7:9200/]}}
logstash01_1  | [2024-07-20T10:15:54,973][WARN ][logstash.outputs.elasticsearch][wazuh] Restored connection to ES instance {:url=>"https://logstash:xxxxxx@10.3.200.7:9200/"}
logstash01_1  | [2024-07-20T10:15:54,973][INFO ][logstash.outputs.elasticsearch][wazuh] Elasticsearch version determined (8.14.3) {:es_version=>8}
logstash01_1  | [2024-07-20T10:15:54,973][WARN ][logstash.outputs.elasticsearch][wazuh] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
logstash01_1  | [2024-07-20T10:15:54,979][INFO ][logstash.javapipeline    ][wazuh] Starting pipeline {:pipeline_id=>"wazuh", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["central pipeline management"], :thread=>"#<Thread:0x58fb32ff /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
logstash01_1  | [2024-07-20T10:15:54,983][INFO ][logstash.outputs.elasticsearch][wazuh] Using mapping template from {:path=>"/usr/share/logstash/conf.d/templates/wazuh.json"}
logstash01_1  | [2024-07-20T10:15:54,994][INFO ][logstash.javapipeline    ][wazuh] Pipeline Java execution initialization time {"seconds"=>0.02}
logstash01_1  | [2024-07-20T10:15:54,996][INFO ][logstash.outputs.elasticsearch][wazuh] Installing Elasticsearch template {:name=>"wazuh"}
logstash01_1  | [2024-07-20T10:15:55,030][INFO ][logstash.inputs.opensearch][wazuh] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
logstash01_1  | [2024-07-20T10:15:55,030][INFO ][logstash.javapipeline    ][wazuh] Pipeline started {"pipeline.id"=>"wazuh"}
logstash01_1  | [2024-07-20T10:15:55,042][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:wazuh], :non_running_pipelines=>[]}
logstash01_1  | [2024-07-20T10:15:55,980][INFO ][logstash.javapipeline    ][wazuh] Pipeline terminated {"pipeline.id"=>"wazuh"}
11

Not clear if you are using the Centralized PIpeline Management in logstash, which requires a paid license or if you are talking about using Ingest pipelines to parse your data instead of Logstash, can you provide more context?

There is no errors in your logs and your logstash pipeline was terminated after a couple of seconds.

[2024-07-20T10:15:55,980][INFO ][logstash.javapipeline ][wazuh] Pipeline terminated {"pipeline.id"=>"wazuh"}

Your opensearch input does not seem to have an schedule, I'm assuming this input is a fork of the elasticsearch input and works in the same way, if so, without a schedule it will run only once.

If you didn't get any result from it you need to check if your query really have any hit.

From what you shared there is nothing wrong on Logstash and Elasticsearch side anymore.

12

I'm using it through a paid license.

By the way, the problem was that the query (opensearch side) was too restricted (1 hour old logs with a certain severity level). Now it works, thanks for the provided help.

1 Like