Parameter max_poll_records of kafka plagin doesn't works on service startup and message sequence doesn't save.
Logstash config:
input {
kafka {
bootstrap_servers => "0.dual.kafka.qa-env.com:9092,1.dual.kafka.qa-env.com:9092,2.dual.kafka.qa-env.com:9092"
topics => ["mt--tmpTest"]
auto_offset_reset => latest
client_id => "logstash_c1"
group_id => "logstash_p1"
max_poll_records => "1"
consumer_threads => 1
}
}
filter {
json {
source => "message"
}
sleep {
time => "1"
}
}
output {
file {
path => "messages.log"
}
http {
http_method => "post"
url => "http://sharedservices.qa-env.com:9977/SendMessage?number=%{[number]}"
headers => { "Content-type" => "application/json; charset=utf-8" }
format => "json"
connect_timeout => 60
retry_failed => false
pool_max_per_route => 1
}
}pipeline.workers=1
Steps for reproduce:
- Stop logstash
- Send 6 messages to kafka topic:
{"number": 1}
{"number": 2}
{"number": 3}
{"number": 4}
{"number": 5}
{"number": 6}
- Start logstash
The expected result:
- messages.log has messages:
{"number": 1}
{"number": 2}
{"number": 3}
{"number": 4}
{"number": 5}
{"number": 6}
- Logstash send requests
http://sharedservices.qa-env.com:9977/SendMessage?number=1
http://sharedservices.qa-env.com:9977/SendMessage?number=2
http://sharedservices.qa-env.com:9977/SendMessage?number=3
http://sharedservices.qa-env.com:9977/SendMessage?number=4
http://sharedservices.qa-env.com:9977/SendMessage?number=5
http://sharedservices.qa-env.com:9977/SendMessage?number=6
- Logstash sends message with interval 1 sec between each message
The result:
- messages.log has messages:
{"message":"{\"number\": 1}","number":1,"@timestamp":"2019-11-22T10:47:28.193Z","@version":"1"} {"message":"{\"number\": 2}","number":2,"@timestamp":"2019-11-22T10:47:28.209Z","@version":"1"} {"message":"{\"number\": 3}","number":3,"@timestamp":"2019-11-22T10:47:28.209Z","@version":"1"} {"message":"{\"number\": 4}","number":4,"@timestamp":"2019-11-22T10:47:28.209Z","@version":"1"} {"message":"{\"number\": 5}","number":5,"@timestamp":"2019-11-22T10:47:28.209Z","@version":"1"} {"message":"{\"number\": 6}","number":6,"@timestamp":"2019-11-22T10:47:28.209Z","@version":"1"}
- OK
- Message sequence doesn't saved
172.24.18.208 - - [22/Nov/2019:10:47:35 +0000] "POST /SendMessage?number=6 HTTP/1.1" 200 0 "-" "Manticore 0.6.4"
172.24.18.208 - - [22/Nov/2019:10:47:35 +0000] "POST /SendMessage?number=4 HTTP/1.1" 200 0 "-" "Manticore 0.6.4"
172.24.18.208 - - [22/Nov/2019:10:47:35 +0000] "POST /SendMessage?number=2 HTTP/1.1" 200 0 "-" "Manticore 0.6.4"
172.24.18.208 - - [22/Nov/2019:10:47:35 +0000] "POST /SendMessage?number=5 HTTP/1.1" 200 0 "-" "Manticore 0.6.4"
172.24.18.208 - - [22/Nov/2019:10:47:35 +0000] "POST /SendMessage?number=3 HTTP/1.1" 200 0 "-" "Manticore 0.6.4"
172.24.18.208 - - [22/Nov/2019:10:47:35 +0000] "POST /SendMessage?number=1 HTTP/1.1" 200 0 "-" "Manticore 0.6.4"
- Fail
- Web server received all messages in the same time 22/Nov/2019:10:47:35
- Fail