Http table convert in docs

Elastic Stack Logstash
1

Hello,
I need tips to convert a health Check Http table in many doc's using a pipeline. I only want transform table and in many doc's and columns in different field.

I thought in getting page in txt and replace in a csv and after importing using csv input plugin. But if there are another way to do that only using logstash plugins, I'd appreciate to know how. I already read about http_poller, but I didn't found a filter to get only a table data at message field and convert in many outputs docs.

The table can be none lines or a lot lines. And include a Field with "SWE:" information on each line ("/ocs_validarelegibilidade/").

Page example:

image
image1345×540 33.3 KB

Text page:

<html><head><title>SWE: /ocs_validarelegibilidade/</title></head>
<body><h1>SWE: /ocs_validarelegibilidade/</h1>
<p>Statistics for SWE application "/ocs_validarelegibilidade/" (virtual directory: <tt>/ocs_validarelegibilidade/</tt>):</p>
<blockquote>627 requests, 264 errors, total time: 63s, min: &lt;1ms, max: 1.77s, avg: 100ms (initialization: &lt;1ms)</blockquote>
<p>Statistics for entire SWE ISAPI extension:</p>
<blockquote>2080 requests, 338 errors, total time: 83s, min: &lt;1ms, max: 2.6s, avg: 40ms (initialization: 10ms)</blockquote>
<blockquote>Last running average of 10 ==&gt;  Create: &lt;1ms, ReadClient: &lt;1ms, GetUser: 109ms, GetService: &lt;1ms, BuildArgs: &lt;1ms, ObjMgr: 27ms, BuildPage: &lt;1ms, SendClient: &lt;1ms</blockquote>
<blockquote>Averages ==&gt;  Create: &lt;1ms, ReadClient: &lt;1ms, GetUser: 31ms, GetService: &lt;1ms, BuildArgs: &lt;1ms, ObjMgr: 6ms, BuildPage: &lt;1ms, SendClient: &lt;1ms</blockquote>
<h2>Active SOM Users</h2>
<table border=1 cellspacing=0 cellpadding=0>
<tr valign=center><th align=left rowspan=2>&nbsp;User ID&nbsp;</th><th align=left rowspan=2>&nbsp;Session ID&nbsp;</th><th align=left rowspan=2>&nbsp;Client Host&nbsp;</th><th align=center rowspan=2>&nbsp;State&nbsp;</th><th align=left rowspan=2>&nbsp;User&nbsp;</th><th align=center rowspan=2>&nbsp;Requests&nbsp;</th><th align=center rowspan=2>&nbsp;Errors&nbsp;</th><th align=center colspan=5>&nbsp;Timings&nbsp;</th></tr>
<tr valign=center><th align=center>&nbsp;Total&nbsp;</th><th align=center>&nbsp;Min&nbsp;</th><th align=center>&nbsp;Max&nbsp;</th><th align=center>&nbsp;Avg.&nbsp;</th><th align=center>&nbsp;Init.&nbsp;</th></tr>
<tr><td align=left>&nbsp;A17U135&nbsp;</td><td align=left>&nbsp;ehtwqtn9kiyc0000000nkrkva2mqrtvli&nbsp;</td><td align=left>&nbsp;10.59.230.30&nbsp;</td><td align=center>&nbsp;idle 45s&nbsp;</td><td align=left>&nbsp;USER1&nbsp;</td><td align=center>&nbsp;2&nbsp;</td><td align=center>&nbsp;0&nbsp;</td><td align=center>&nbsp;496ms&nbsp;</td><td align=center>&nbsp;128ms&nbsp;</td><td align=center>&nbsp;368ms&nbsp;</td><td align=center>&nbsp;248ms&nbsp;</td><td align=center>&nbsp;366ms&nbsp;</td></tr>
<tr><td align=left>&nbsp;A17U134&nbsp;</td><td align=left>&nbsp;ehtwqtk8kftet0000000000c8ngybmbw&nbsp;</td><td align=left>&nbsp;10.59.231.30&nbsp;</td><td align=center>&nbsp;idle 2:15&nbsp;</td><td align=left>&nbsp;USER2&nbsp;</td><td align=center>&nbsp;2&nbsp;</td><td align=center>&nbsp;0&nbsp;</td><td align=center>&nbsp;524ms&nbsp;</td><td align=center>&nbsp;145ms&nbsp;</td><td align=center>&nbsp;379ms&nbsp;</td><td align=center>&nbsp;262ms&nbsp;</td><td align=center>&nbsp;376ms&nbsp;</td></tr>
</table>
<p>(2 SOM users active.)</p>
<p><hr size=1 noshade>
This report produced by the <i>Siebel Web Engine</i> (SWE) for <I>Microsoft Internet Information Server</I>.<br>Copyright &copy; Siebel Systems, Inc. 1999.  All rights reserved.</p>
</body></html>
2

You will need pipeline.workers 1 and to disable java_execution to maintain the order of the rows (the header row has to be processed first).

    if [message] =~ /^<html><head><title>/ {
        mutate { gsub => [ "message", "</[a-z]+>", "", "message", ".*SWE: ", "" ] }
        ruby { code => '@swe = event.get("message")' }
    }
    if [message] =~ /^<tr><td / {
        mutate {
            gsub => [
                "message", "<(/)?tr>", "",
                "message", "<[^>]+>&nbsp;", "",
                "message", "&nbsp;</td>", ",",
                "message", ",$", ""
             ]
        }
        csv {}
        ruby { code => 'event.set("swe", @swe)' }
    } else {
        drop {}
    }

I really don't like using class variables in ruby filters, but I can't think of an alternative right now.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.