Missing EOL character - Can't read line

[sancroth (Nick)]

I am trying to capture magento's Report logs.
Magento report are generated into a single file every time so you can have multiple reports at multiple files.

I am harvesting them by using the path:
/path/to/magento/reports/*

I have tried capturing the log with and without multiline without any success.

The log looks like this:

{"0":"Specified invalid parent id (MageGuide_Mgbase::MageGuide)","1":"#0 \/home\/pharma\/public_html\/vendor\/magento\/framework\/Interception\/Interceptor.php(58): Magento\\Backend\\Model\\Menu\\Builder->getResult(Object(Magento\\Backend\\Model\\Menu))\n#1 \/home\/pharma\/public_html\/vendor\/magento\/framework\/Interception\/Interceptor.php(138): Magento\\Backend\\Model\\Menu\\Builder\\Interceptor->___callParent('getResult', Array)\n#2 \/home\/pharma\/public_html\/vendor\/magento\/framework\/Interception\/Interceptor.php(153): Magento\\Backend\\Model\\Menu\\Builder\\Interceptor->Magento\\Framework\\Interception\\{closure}(Object(Magento\\Backend\\Model\\Menu))\n#3 \/home\/pharma\/public_html\/generated\/code\/Magento\/Backend\/Model\/Menu\/Builder\/Interceptor.php(39): Magento\\Backend\\Model\\Menu\\Builder\\Interceptor->___callPlugins('getResult', Array, Array)\n#4 \/home\/pharma\/public_html\/vendor\/magento\/module-backend\/Model\/Menu\/Config.php(148): Magento\\Backend\\Model\\Menu\\Builder\\Interceptor->getResult(Object(Magento\\Backend\\Model\\Menu))\n#5 \/home\/pharma\/public_html\/vendor\/magento\/module-backend\/Model\/Menu\/Config.php(111): Magento\\BackendMagento\\Backend\\Controller\\Adminhtml\\Index\\Index\\Interceptor->_processUrlKeys()\n#11 \/home\/pharma\/public_html\/vendor\/magento\/framework\/Interception\/Interceptor.php(58): Magento\\Backend\\App\\AbstractAction->dispatch(Object(Magento\\Framework\\App\\Request\\Http))\n#12 \/home\/pharma\/public_html\/vendor\/magento\/framework\/Interception\/Interceptor.php(138): Magento\\Backend\\Controller\\Adminhtml\\Index\\Index\\Interceptor->___callParent('dispatch', Array)\n#14 \/home\/pharma\/public_html\/vendor\/magento\/framework\/Interception\/Interceptor.php(135): Magento\\Backend\\App\\Action\\Plugin\\Authentication->aroundDispatch(Object(Magento\\Backend\\Controller\\Adminhtml\\Index\\Index\\Interceptor), Object(Closure), Object(Magento\\Framework\\App\\Request\\Http))\n#15 \/home\/pharma\/public_html\/vendor\/magento\/framework\/Interception\/Interceptor.php(153): Magento\\Backend\\Controller\\Adminhtml\\Index\\Index\\Interceptor->Magento\\Framework\\Interception\\{closure}(Object(Magento\\Framework\\App\\Request\\Http))\n#16 \/home\/pharma\/public_html\/generated\/code\/Magento\/Backend\/Controller\/Adminhtml\/Index\/Index\/Interceptor.php(39): Magento\\Backend\\Controller\\Adminhtml\\Index\\Index\\Interceptor->___callPlugins('dispatch', Array, NULL)\n#17 \/home\/pharma\/public_html\/vendor\/magento\/framework\/App\/FrontController.php(55): Magento\\Backend\\Controller\\Adminhtml\\Index\\Index\\Interceptor->dispatch(Object(Magento\\Framework\\App\\Request\\Http))\n#18 \/home\/pharma\/public_html\/vendor\/magento\/framework\/Interception\/Interceptor.php(58): Magento\\Framework\\App\\FrontController->dispatch(Object(Magento\\Framework\\App\\Request\\Http))\n#19 \/home\/pharma\/public_html\/vendor\/magento\/framework\/Interception\/Interceptor.php(138): Magento\\Framework\\App\\FrontController\\Interceptor->___callParent('dispatch', Array)\n#20 \/home\/pharma\/public_html\/vendor\/magento\/framework\/Interception\/Interceptor.php(153): Magento\\Framework\\App\\FrontController\\Interceptor->Magento\\Framework\\Interception\\{closure}(Object(Magento\\Framework\\App\\Request\\Http))\n#21 \/home\/pharma\/public_html\/generated\/code\/Magento\/Framework\/App\/FrontController\/Interceptor.php(26): Magento\\Framework\\App\\FrontController\\Interceptor->___callPlugins('dispatch', Array, Array)\n#22 \/home\/pharma\/public_html\/vendor\/magento\/framework\/App\/Http.php(135): Magento\\Framework\\App\\FrontController\\Interceptor->dispatch(Object(Magento\\Framework\\App\\Request\\Http))\n#23 \/home\/pharma\/public_html\/vendor\/magento\/framework\/App\/Bootstrap.php(256): Magento\\Framework\\App\\Http->launch()\n#24 \/home\/pharma\/public_html\/pub\/index.php(37): Magento\\Framework\\App\\Bootstrap->run(Object(Magento\\Framework\\App\\Http))\n#25 {main}","url":"\/backpharm\/admin\/index\/index\/key\/9044c99afcf4336da610dab700bc8027979e52169bdae38fbe0d3cd0adb259c1\/","script_name":"\/index.php"}

I have tried multilines like:

pattern: '"\}$'
negate: true
match: before

pattern:'^{'
negate:true
match:after

And a few more but to no fruit.
Doesn't capture the message or any line at all even with no multiline context.

Message i get from debug is:

2018-04-11T11:41:04.836+0200    DEBUG   [prospector]    log/prospector.go:499   Harvester for file is still running: /home/pharma/public_html/var/report/123123123123123
2018-04-11T11:41:04.836+0200    DEBUG   [prospector]    log/prospector.go:361   Check file for harvesting: /home/pharma/public_html/var/report/1378880469402
2018-04-11T11:41:04.836+0200    DEBUG   [prospector]    log/prospector.go:447   Update existing file for harvesting: /home/pharma/public_html/var/report/1378880469402, offset: 0
2018-04-11T11:41:04.836+0200    DEBUG   [prospector]    log/prospector.go:499   Harvester for file is still running: /home/pharma/public_html/var/report/1378880469402
2018-04-11T11:41:04.836+0200    DEBUG   [prospector]    log/prospector.go:361   Check file for harvesting: /home/pharma/public_html/var/report/222222222222
2018-04-11T11:41:04.836+0200    DEBUG   [prospector]    log/prospector.go:447   Update existing file for harvesting: /home/pharma/public_html/var/report/222222222222, 

As well as a Back off message because it harvester reached the EOF.

As i understand there is no matching happening with the patterns so the harvester is still looking for the matching pattern or found the matching pattern and looks for another that never comes(depending on the the before-after match value) or something is getting messed up with the characters.

Anyone had any luck with this kind of messages?

Also. I guess harvesters will keep monitoring these files as long as they exist. Is there a way to stop this from happening everytime they read the message(when that happens ). Found the close_eof directives and some others but not sure if i should use them and how to use them properly yet.

[sancroth (Nick)]

As i mentioned i tried getting the message without multiline and did not work.
This was because i expected based on cat that the message is single line regardless of some posts that i read here and there.
In the end i tried the obvious:

cat /home/pharma/public_html/var/report/1111111111111 | wc -l

Which returned the value 0.

If i am not mistaken wc -l counts the end of line characters.
So if EOL is missing for some reason filebeat cannot really understand that this is the message and nothing more is coming, if i m guessing right.

So how do we work around this if that is the case?

Update: manually adding \r\n(or just an enter) to the end of the report file makes filebeat read and send the message. So how can we work around this if this is missing?
Also what about stop polling the files after a message has been sent?

[pierhugues (Pier-Hugues Pellerin)]

@sancroth like you saw filebeat monitor file and watch for EOL (\n), without that it assume that the line is incomplete, Filebeat will automatically close an inactive file after 5mins. You can also configure Filebeat to ignore file older than a certain time.

[sancroth (Nick)]

I ended up creating a bash script that looks for all ~/$projects/path/to/reports/ and appends an "" ,which creates a new line, to all files on that dir that were modified in the last minute(mmin -1).
Then added the script on cron running every minute. If you can think of a better alternative let me know.
Will add the ignore file and i think it should be fine.