Logstash to Elasticsearch

[headtea]

I'm trying to send data from logstash to ES and I get an error. I can't make much sense of it. This is my logstash output:

output{
elasticsearch {
hosts => ["https://localhost:9200"]
cacert => "/etc/elasticsearch/http.crt"
ssl => true
ssl_certificate_verification => false 
index => "cleandata"
user => "logstash_internal"
password => "password"
}
}

This is the error I get:

[2020-08-23T20:57:01,689][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_internal:xxxxxx@localhost:9200/]}}
[2020-08-23T20:57:02,036][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

Does anyone know what it means? It says that and immidiatley shuts down. Huge thanks ahead

[elasticforme (Sachin)]

what is other config file looks like ?

message says it has problem starting pipeline?
are you trying to execute this from command line or as daemon?

[headtea]

Thanks for the response!

Just for the record, curl works:

curl https://logstash_internal:password@localhost:9200 -k
{
  "name" : "server150",
  "cluster_name" : "log",
  "cluster_uuid" : "48yas_dasd_123fas",
  "version" : {
    "number" : "7.9.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "a479a2a7fce0329512d6a9361301708b92dff667",
    "build_date" : "2020-08-11T21:36:48.204330Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

This is production deployment. There shouldn't be errors since I copied the config from the demo environment which works. I only have one config file and this is what it says:

[root@server150 ~]# cat /etc/logstash/conf.d/test.conf 
input{
beats{
port => 5044
ssl => true
ssl_certificate_authorities => ["/etc/elk/certs/ca/ca.crt"]
ssl_certificate => "/etc/elk/instances/instances.crt"
ssl_key => "/etc/elk/instances/logs.pkcs8.key"
ssl_key_passphrase => "password"
ssl_verify_mode => "force_peer"
}
}
filter {
mutate {
        add_tag => [ "insidefilter" ]
      }
if [log][file][path] =~ "mongos.log"{
grok { match => { "message" => "\A%{TIMESTAMP_ISO8601} I ACCESS   %{NOTSPACE} SCRAM-SHA-1 authentication failed for %{USER:User} on %{USER:DB} from client %{SYSLOGHOST:From}:%{INT:Port} ; %{GREEDYDATA:Reason}"
}
}
mutate { add_tag => [ "mongosIfStatement" ] }
}
if [log][file][path] =~ "mysqld.log"{
grok { match => { "message" => "%{TIMESTAMP_ISO8601:Date} %{INT:Number} \[Note] Access denied for user '%{USER:User}'@'%{SYSLOGHOST:IP}' \(using password: %{WORD:Password}%{GREEDYDATA}"
}
}
mutate { add_tag => [ "mysqldIfStatement" ] }
} 
if [log][file][path] =~ /\.json$/ {
     json {
        source => "message"
     }
  }
if [log][file][path] =~ "commands.log" {
grok{
match => { "message" => "\[(%{TIMESTAMP_ISO8601:sys_timestamp})\]\s(?<Hostname>[0-9a-zA-Z_-]+)\s(?<Logged>[0-9a-zA-Z_-]+)\:USER=(?<User>[0-9a-zA-Z_-]+)\sPWD=(?<Directory>[0-9a-zA-Z_/-]+)\sPID=\[(?<PID>[0-9]+)\]\sCMD=\"(?<Command>.*)\"\sExit=\[(?<Exit>[0-9]+)\]\sCONNECTION=(?<Connetion>.*)"
}
match => { "message" => "\A%{SYSLOGTIMESTAMP:sys_timestamp} %{HOSTNAME:Hostname} %{USER:Logged}: USER=%{USER:User} PWD=%{UNIXPATH:Directory} PID=\[%{INT:PID}] CMD=%{QUOTEDSTRING:Command} Exit=\[%{INT:Exit}] CONNECTION=%{GREEDYDATA:Connection}"}
}
}
}
output{
elasticsearch {
hosts => ["https://localhost:9200"]
cacert => "/etc/elasticsearch/http.crt"
ssl => true 
#ssl_certificate_verification => false
index => "cleandata"
user => "logstash_internal"
password => "password"
}
}

This is how I start logstash:

logstash --path.settings=/etc/logstash/ -f /etc/logstash/conf.d/test.conf

Which results this:

# logstash --path.settings=/etc/logstash/ -f /etc/logstash/conf.d/test.conf
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2020-08-24T09:20:41,083][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 OpenJDK 64-Bit Server VM 25.262-b10 on 1.8.0_262-b10 +indy +jit [linux-x86_64]"}
[2020-08-24T09:20:41,957][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-08-24T09:20:46,088][INFO ][org.reflections.Reflections] Reflections took 58 ms to scan 1 urls, producing 22 keys and 45 values 
[2020-08-24T09:20:47,480][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_internal:xxxxxx@localhost:9200/]}}
[2020-08-24T09:20:47,924][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-08-24T09:20:48,236][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-08-24T09:20:53,125][INFO ][logstash.runner          ] Logstash shut down.
[2020-08-24T09:20:53,150][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

[elasticforme (Sachin)]

message says it started Logstash API endpoint

test it like this. have a config without any filter. also comment out input to elasticsearch

and run

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

input{
  beats{
   port => 5044
   ssl => true
   ssl_certificate_authorities => ["/etc/elk/certs/ca/ca.crt"]
   ssl_certificate => "/etc/elk/instances/instances.crt"
   ssl_key => "/etc/elk/instances/logs.pkcs8.key"
   ssl_key_passphrase => "password"
   ssl_verify_mode => "force_peer"
   }
}
filter { }
output { 
   stdout { codec => rubydebug }
}