How to forward index from filebeat to elasticsearch via logstash using http output

Hello,
Hello, I am starting my journey with elasticsearch and I have a couple of questions. I tried to find answers in documentation but some areas are not clear for me and I am confused.

  1. If I understood correct, in order to group logs from some servers and find them in one place, I shoudl used diffrent "indexes" for diffrenst group of servers. For example, I have 5 servers in one location so I can put them into one index, and other servers from another location into second index and so on, am I right?
  2. I am using Filebeat to pass logs into Logstash and http output to pass logs from logstach. I tried with below solution,
    How to forward index from filebeat to elasticsearch via logstash - Elastic Stack / Beats - Discuss the Elastic Stack

but based on documentation for http output, https doesn't have index parameter so now I don't know how to create indexes and pass them to my log analytics tool.
Http output plugin | Logstash Reference [8.6] | Elastic

In my filebeat yml file

output.logstash:
  # The Logstash hosts
  hosts: ["servername.domain:5044"]
  index: "custom_location_90"

In my logstash.conf file

input {
  beats {
    port => 5044
  }
 syslog {
    port => 514
  }
}

output{
         http
                {
                 url=>"https://myurl/log-service/api/v1.0/logs"
                 index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
                 http_method=>"post"
                 content_type=>"application/json"
                 format=>"json_batch"
                 retry_failed=>false
                 http_compression=>true
                 headers => {
                     "Content-Type" => "application/json"
                     "Authorization" => "apiKey XXXXX"}
                 }
                 }

Can you provide more context about this? If you are using elasticsearch you need to use the elasticsearch output, since you are using the http output it looks like that you are not using elasticsearch, so the concept of index makes no sense in this case.

We are using BMC Helix Log Analytics. This is 3rd part tool vendor, which implemented for their solution Elasticsearch and Kibana and according to their documentation http output should be used to send data from Logstash.
Collecting logs by using Logstash and Filebeat - Documentation for BMC Helix Log Analytics 22.4 - BMC Documentation

If you followed their documentation and it still doesn't work, you probably will need to check with BMC on a forum for this tool or wait to see if someone that also uses this tool have a similar error and knows how to solve.

But from the documentation you shared there is nothing about indice name, probably you need to configure it in the tool or you may not even have this option, you need to check with BMC as this is not related to Logstash.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.