So I tried again on a 6.1.1 setup to verify that it works and now it doesn't. This is using /logstash -f pathtoconfig. I did notice, however, while there is still the primary ASCII8-bit error, there's a few additional error messages that might help. Anybody with the brain power to interpret these willing to take a look?
Looks like the same error, indicating that the problem isn't related to the Nokogiri 1.8.2 step-up.
It isn't, I thought the previous research you did indicated it was? I know if I take all xpath configurations out of the pipeline it works, as soon as I enter an xpath it errors out.
OK, trying to start back from the very basics and multiline has me scratching my head. This pipeline:
input {
file {
id => "C:\DMARC\*.xml"
path => "C:/DMARC/*.xml"
discover_interval => 5
codec => multiline {
auto_flush_interval => 5
negate => true
pattern => "<record>"
what => "previous"
}
}
}
filter {
xml {
id => "Field Extraction"
force_array => false
store_xml => false
source => "message"
}
}
Results in an output that looks like this in the message field, shouldn't it be one long line??
Each line ends with a newline character and they're preserved when the lines are merged.
No Idea what is going on...Setup a new 2012R2 VM thinking it was the OS, single node Elastic stack using 6.2.2, copied pipeline, template, powershell script, and visualizations export and bam everything works flawlessly...
So I go back to verify the OS the original working config using 6.1.1 and that's on Server 2016. I have an image deployment server so I know it's deploying the exact same image every time for 2012 R2 and 2016.
I genuinely have zero clue as to the random success/failures here....Guess it's time to be hyper-methodical and document every single change I make...
New 2016 VM 6.2.2, doesn't work....new 2012R2 VM 6.2.2, doesn't work...
AHHH!!!! I think I found the problem child!!
Went back to my working setup and looked at my logstash.yml and it was barebones configuration:
node.name: ElasticStack
config.reload.automatic: true
config.reload.interval: 3s
log.level: trace
So for the setup that works, if I enable persisted queue, it breaks with the same Nokogiri error message. Turn off persisted queue and it starts working again. However, I also noticed that my working setup is using Java 8u152, I've been testing other setups with 8u162 and once with 9.0.4 (all kinds of shit broke with 9x).
So I currently have a fully updated Server 2012 R2 box, I'm gonna build a quick 6.2.2 with JDK 8u162 and see if I still get an error on it without persisted queue enabled. I will report back with my results.
WOO!!!! SUCCESS!!!!
Turn off persisted queue on 6.2.2 and everything functions great!
So now the question is....wtf??? Why is disk buffering causing the failure???
Oh...I guess to be thorough, here are the settings that are causing the failure:
queue.type: persisted
path.queue: "C:/Buffer"Is the forward-slash correct there? I know it's weird sometimes with Java but double check that it shouldn't rather be "C:\\Buffer" or "C:\Buffer".
edit: or even "C:\/Buffer"
editedit: Does the user have permissions to write to C:\ ?
I will double check those config settings to be sure. I tried c:\ and Logstash wouldn't even start to the point that it would write to logstash-plain.log. Also, permissions are good, c:/buffer shows files being written into the folder.
Doesn't appear to be my syntax but rather a bug in the configuration. I enabled disk buffering but didn't specify a path. Below is a portion of the debug log after it finished loading the pipeline:
[2018-03-02T21:09:15,663][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@validate_after_inactivity = 10000
[2018-03-02T21:09:15,663][DEBUG][org.logstash.ackedqueue.Queue] opening tail page: 0, in: C:/logstash/data/queue/dmarcxml, with checkpoint: pageNum=0, firstUnackedPageNum=0, firstUnackedSeqNum=2, minSeqNum=1, elementCount=2, isFullyAcked=no
[2018-03-02T21:09:15,757][DEBUG][org.logstash.ackedqueue.Queue] opening head page: 1, in: C:/logstash/data/queue/dmarcxml, with checkpoint: pageNum=1, firstUnackedPageNum=0, firstUnackedSeqNum=0, minSeqNum=0, elementCount=0, isFullyAcked=no
[2018-03-02T21:09:15,788][DEBUG][org.logstash.ackedqueue.io.AbstractByteBufferPageIO] PageIO recovery element index:0, readNextElement exception: Element invalid length
[2018-03-02T21:09:15,835][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"dmarcxml", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
I see the second to last line throws an exception. Not sure if that's expected behavior or a recoverable error or what.
I got the same bug, but my logstash 6.2.2 is installed on CentOS7.
And the same, if I sharp the queue.type: persisted in the pipelines.yml, it works.
1 Like
Can someone share the config they were having issues with and the config they had working? Then I can see if one of the dev team can take a closer look.
1 Like
Here's the pipeline I use that works without persisted queue. Enable persisted queue and watch it burn to the ground, lol.
Can you share the part of the config you added with persistent queues (even just here)?
It'll help debug when we can replicate as close as possible.
I tried setting queue.type: persisted and path.queue: "C:/Buffer" in both logstash.yml and in pipelines.yml. There's no secret sauce here to break it, lol.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.