Couchdb_changes plugin resetting .couchdb_seq

uschtwill · September 11, 2017, 11:25am

Recently, irregularly, during a 3 day index, the .couchdb_seq file is deleted/resetted, so Logstash starts indexing from the start. I am fairly certain the file is not deleted from the outside, but that Logstash itself resets it.

Has this happened to anyone before? Are there known circumstances under which this can happen?

I am happy for any pointers, or pointing other people who might know something about this to this thread. Hey @pierhugues, @theuntergeek, @jordansissel, @jsvd, any ideas?

Some background info:

Version/OS/Docker:
FROM docker.elastic.co/logstash/logstash:5.3.0
input from logstash.conf:

    host => "${COUCHDB_HOST}"
    username => "${COUCHDB_USER}"
    password => "${COUCHDB_PASSWORD}"
    port => 5984
    db => "db"
    sequence_path => "/couchdb_seq/stage2/.db"
    always_reconnect => true
    type => "type"
    keep_revision => true
    keep_id => true
    add_field => { "[@metadata][stream_domain]" => "type" }
    add_field => { "[@metadata][stage]" => "s2" }
    add_field => { "[@metadata][environment]" => "${ENVIRONMENT}" }

logstash.yml:

xpack.monitoring.enabled: false
queue.type: "memory"
# log.level:

uschtwill · September 11, 2017, 12:14pm

More information:

The reset appears to coincide with the following line in the logs: [ERROR][logstash.inputs.couchdbchanges] Connection problem encountered: Retrying connection in 10 seconds... {:error=>"End of file reached", :host=>"couchdb_prod", :port=>"5984", :db=>"db"}
Around the time of the resets between 7,5Gb and 13Gb of disk space are freed up on the host. This might or might not be connected.

uschtwill · September 11, 2017, 9:07pm

More information:

This is what's written to the CouchDB logs every time this happens, but even after googling I am not too sure what it means. My hope is that one of you couchdb_changes contributors might now what to make of that.

September 8th 2017, 23:11:46.211	backendinfrastructure_couchdb_prod_1	[error] 2017-09-08T22:11:46.209985Z nonode@nohost <0.27801.2663> 463cf952bf rexi_server exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,349}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,783}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,710}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,754}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,220}]},{couch_db,changes_since,5,[{file,"src/couch_db.erl"},{line,1244}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,83}]}]
September 8th 2017, 23:11:46.211	backendinfrastructure_couchdb_prod_1	[error] 2017-09-08T22:11:46.209997Z nonode@nohost <0.29400.2663> 463cf952bf rexi_server exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,349}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,783}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,710}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,754}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,220}]},{couch_db,changes_since,5,[{file,"src/couch_db.erl"},{line,1244}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,83}]}]
September 8th 2017, 23:11:46.213	backendinfrastructure_couchdb_prod_1	[error] 2017-09-08T22:11:46.210273Z nonode@nohost <0.30569.2663> 463cf952bf rexi_server exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,349}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,783}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,710}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,754}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,220}]},{couch_db,changes_since,5,[{file,"src/couch_db.erl"},{line,1244}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,83}]}]
September 8th 2017, 23:11:46.213	backendinfrastructure_couchdb_prod_1	[error] 2017-09-08T22:11:46.210163Z nonode@nohost <0.23750.2663> 463cf952bf rexi_server exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,349}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,783}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,710}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,754}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,220}]},{couch_db,changes_since,5,[{file,"src/couch_db.erl"},{line,1244}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,83}]}]
September 8th 2017, 23:11:46.213	backendinfrastructure_couchdb_prod_1	[error] 2017-09-08T22:11:46.210259Z nonode@nohost <0.30473.2663> 463cf952bf rexi_server exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,349}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,783}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,710}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,754}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,220}]},{couch_db,changes_since,5,[{file,"src/couch_db.erl"},{line,1244}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,83}]}]
September 8th 2017, 23:11:46.213	backendinfrastructure_couchdb_prod_1	[error] 2017-09-08T22:11:46.210031Z nonode@nohost <0.31176.2663> 463cf952bf rexi_server exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,349}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,783}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,710}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,754}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,220}]},{couch_db,changes_since,5,[{file,"src/couch_db.erl"},{line,1244}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,83}]}]
September 8th 2017, 23:11:46.214	backendinfrastructure_couchdb_prod_1	[error] 2017-09-08T22:11:46.210328Z nonode@nohost <0.22764.2663> 463cf952bf rexi_server exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,349}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,783}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,710}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,754}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,220}]},{couch_db,changes_since,5,[{file,"src/couch_db.erl"},{line,1244}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,83}]}]
September 8th 2017, 23:11:58.608	backendinfrastructure_couchdb_prod_1	[error] 2017-09-08T22:11:58.608587Z nonode@nohost <0.27790.2663> 463cf952bf req_err(527034634) timeout : nil
September 8th 2017, 23:11:58.609	backendinfrastructure_couchdb_prod_1	[error] 2017-09-08T22:11:58.609078Z nonode@nohost <0.27790.2663> 463cf952bf Response abnormally terminated: {timeout,nil,[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,349}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,783}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,710}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,754}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,220}]},{couch_db,changes_since,5,[{file,"src/couch_db.erl"},{line,1244}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,83}]}]}
September 8th 2017, 23:11:58.609	backendinfrastructure_couchdb_prod_1	    [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,349}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,783}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,710}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,754}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,220}]},{couch_db,changes_since,5,[{file,"src/couch_db.erl"},{line,1244}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,83}]}]

uschtwill · September 17, 2017, 10:13am

Setting the following in the CouchDB's local.ini fixed it:

[fabric]
request_timeout = infinity

Cheers
Will

system · October 15, 2017, 10:13am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.