I am using Elasticsearch 8.13.4, Logstash 8.13.4, and Filebeat 8.13.4. I followed the steps in the documentation here to get familiar with Logstash, but I am unable to sync data to Elasticsearch. I am encountering the following warning message:
/home/tom/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:534: warning: already initialized constant Manticore::Client::ByteArrayEntity
config:
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
geoip {
source => "[source][address]"
target => "geoip"
}
}
output {
elasticsearch {
hosts => ["https://localhost:9200"]
index => "tom_test"
user => "user"
password => "password"
ssl_enabled => true
ssl_certificate_authorities => ["../elasticsearch-8.13.4/config/certs/http_ca.crt"]
}
}
Can anyone provide assistance or guidance on how to resolve this issue?
Additionally, I noticed that some examples in the documentation do not execute successfully:
- The
first-pipeline.confconfiguration file provided in the documentation is as follows:
plaintext
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
output {
stdout { codec => rubydebug }
}
The data shown in the documentation for Logstash is:
json
{
"request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"offset" => 325,
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"prospector" => {
"type" => "log"
},
"input" => {
"type" => "log"
},
"source" => "/path/to/file/logstash-tutorial.log",
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
"@timestamp" => "2017-11-09T02:51:12.416Z",
"response" => "200",
"bytes" => "203023",
"clientip" => "83.149.9.216",
"@version" => "1",
"beat" => {
"name" => "My-MacBook-Pro.local",
"hostname" => "My-MacBook-Pro.local",
"version" => "6.0.0"
},
"host" => "My-MacBook-Pro.local",
"httpversion" => "1.1",
"timestamp" => "04/Jan/2015:05:13:42 +0000"
}
However, the actual output data structure is as follows:
json
{
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"input" => {
"type" => "filestream"
},
"@timestamp" => "2024-08-12T06:38:39.793Z",
"timestamp" => "04/Jan/2015:05:30:37 +0000",
"user_agent" => {
"original" => "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"
},
"event" => {
"original" => "86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] \"GET /reset.css HTTP/1.1\" 200 1015 \"http://www.semicomplete.com/projects/xdotool/\" \"Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0\""
},
"ecs" => {
"version" => "8.0.0"
},
"host" => {
"name" => "bumblebee"
},
"log" => {
"offset" => 24033,
"file" => {
"inode" => "581022",
"device_id" => "64768",
"path" => "/home/tom/logstash-tutorial-dataset"
}
},
"source" => {
"address" => "86.1.76.62"
},
"agent" => {
"id" => "0b49ea59-b9e2-4782-a3e2-680c90a73d84",
"version" => "8.13.4",
"ephemeral_id" => "c66d1aae-5c33-422a-998a-86d4fbea79b9",
"name" => "bumblebee",
"type" => "filebeat"
},
"@version" => "1",
"http" => {
"request" => {
"method" => "GET",
"referrer" => "http://www.semicomplete.com/projects/xdotool/"
},
"version" => "1.1",
"response" => {
"body" => {
"bytes" => 1015
},
"status_code" => 200
}
},
"url" => {
"original" => "/reset.css"
},
"message" => "86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] \"GET /reset.css HTTP/1.1\" 200 1015 \"http://www.semicomplete.com/projects/xdotool/\" \"Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0\""
}
The following plugin configuration does not work:
plaintext
geoip {
source => "clientip"
}
- When outputting data to Logstash, the configuration file in the documentation is as follows:
plaintext
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}
Since Elasticsearch 8 and later versions start in secure mode, the above configuration will connect to Elasticsearch using HTTP instead of HTTPS.
I would like to inquire if the examples provided in the documentation are guaranteed to work, or if there might be issues with the documentation.