Logstash email hyperlink length limit


#1

The email plug is working fine. I would like to add a hyperlink to Kibana to allow the user to display a Visualization.
All of that works, except the hyperlink is truncated in the Outlook email.
I have investigated different ways to shorten the URL. I cannot use solutions such as tinyurl for security reasons.
Do you have any suggestions?

Here is a Discovery example, which is as short as they get:

            if "_grokparsefailure" in [tags] {
                    grok {
                            match => { "message" => "%{GREEDYDATA:err_num}:%{SPACE}%{GREEDYDATA:narrative}" }
                            remove_tag => ["_grokparsefailure"]
                            add_field => [ "tags", "grok0714" ]
                    }

                    if( "grok0714" in [tags] ){

                            if( "Error" in [err_num] ){

                                    if( "No space left on device" in [narrative] ){

                                            mutate{
                                                    ### Replacing msg_level with err_num because it is blank. 2/7/2017 - Brian
                                                    replace => { "msg_level" => "%{err_num}" }
                                                    add_field => [ "[@metadata][my_body]", "

http://XXX.XXX.XXX.com:5602/app/kibana#/discover?_g=(refreshInterval:(display:'1%20minute',pause:!f,section:2,value:60000),time:(from:now-1h,mode:quick,to:now))&_a=(columns:!(component,host,err_num,narrative,tags),index:'dplr-%{ENVIRONMENT}-logstash-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'host:%20"%{host}"%20AND%20err_num:%20"%{err_num}"')),sort:!('@timestamp',desc))
"]
}
}
}
}
}


(Magnus Bäck) #2

So... it's Logstash that's truncating the string?


#3

I am not sure.
If I send it in an anchor tag, the text of href is truncated.


#4

Here is the body of the email:

HP-UX Error: 28 from hltv0777.hydc.sbc.com:/opt/app/globalid/data/server/Release-9-0/logs/135.213.40.121_45402_exception.log
Visualization http://blpi165.bhdc.att.com:5602/app/kibana#/visualize/create?type=histogram&indexPattern=dplr-%{Test}-logstash-&_g=%28refreshInterval:%28display:%271%20minute%27,pause:!f,section:2,value:60000%29,time:%28from:now-1h,mode:quick,to:now%29%29&_a=%28filters:!%28%29,linked:!f,query:%28query_string:%28analyze_wildcard:!t,query:%27host:%20%22hltv0777.hydc.sbc.com%22%20AND%20err_num:%20%22HP-UX Error: 28%22%27%29%29,uiState:%28%29,vis:%28aggs:!%28%28id:%271%27,params:%28%29,schema:metric,type:count%29,%28id:%272%27,params:%28customInterval:%272h%27,extended_bounds:%28%29,field:%27@timestamp%27,interval:auto,min_doc_count:1%29,schema:segment,type:date_histogram%29,%28id:%273%27,params:%28field:host.raw,order:desc,orderBy:%271%27,size:5%29,schema:group,type:terms%29%29,listeners:%28%29,params:%28addLegend:!t,addTimeMarker:!f,addTooltip:!t,defaultYExtents:!f,mode:stacked,scale:linear,setYExtents:!f,shareYAxis:!t,times:!%28%29,yAxis:%28%29%29,title:%27New%20Visualization%27,type:histo! gram%29%29
Discovery http://blpi165.bhdc.att.com:5602/app/kibana#/discover?_g=(refreshInterval:(display:'1%20minute',pause:!f,section:2,value:60000),time:(from:now-1h,mode:quick,to:now))&_a=(columns:!(component,host,err_num,narrative,tags),index:'dplr-%{Test}-logstash-
%27,interval:auto,query:%28query_string:%28analyze_wildcard:!t,query:%27host:%20%22hltv0777.hydc.sbc.com%22%20AND%20err_num:%20%22HP-UX Error: 28%22%27%29%29,sort:!%28%27@timestamp%27,desc%29%29
Narrative:
No space left on device
JavaClass:
%{javaclass}
Tags:
grok0714


(Magnus Bäck) #5

Okay... but has that URL really been concatenated? It ends with "%27,desc%29%29" in your Logstash config and in your sample email body.


#6

Apparently not. This may be an issue with Outlook.
Has anyone been able to email hyperlinks?


(Magnus Bäck) #7

Surrounding the URL by angle brackets might help.


#8

Thanks for your patience. Embedding the URL in a table surrounded by

tags is the secret:

dplrgid8@:/home/dplrgid8/logstash/dev/templates/$ cat example.my_body.txt

                    if( "grok0714" in [tags] ){

                            if( "Error" in [err_num] ){

                                    if( "No space left on device" in [narrative] ){

                                            mutate{
                                                    ### Replacing msg_level with err_num because it is blank. 2/7/2017 - Brian
                                                    replace => { "msg_level" => "%{err_num}" }
                                                    add_field => [ "[@metadata][my_body]", "
            <html>
            <body>
                <table style=width:\"100%\">
                    <tr>
                            <td><pre>%{err_num} from %{host}:%{path}</pre></td>
                    <tr>
                            <td><b>Copy & Paste Visualization Link to FireFox Browser:</b></td>
                    <tr>
                            <td>http://xx.xx.xx.com:5602/app/kibana#/visualize/create?embed=true&type=histogram&indexPattern=dplr-qa-logstash-*&_g=(refreshInterval:(display:'1%20minute',pause:!f,section:2,value:60000),time:(from:now-1h,mode:quick,to:now))&_a=(filters:!(),linked:!f,query:(query_string:(analyze_wildcard:!t,query:'host:%20%22hltv0777.hydc.xx.com%22%20AND%20err_num:%20%22HP-UX%20Error:%22')),uiState:(),vis:(aggs:!((id:'1',params:(),schema:metric,type:count),(id:'2',params:(customInterval:'2h',extended_bounds:(),field:'@timestamp',interval:auto,min_doc_count:1),schema:segment,type:date_histogram),(id:'3',params:(field:host.raw,order:desc,orderBy:'1',size:5),schema:group,type:terms)),listeners:(),params:(addLegend:!t,addTimeMarker:!f,addTooltip:!t,defaultYExtents:!f,mode:stacked,scale:linear,setYExtents:!f,shareYAxis:!t,times:!(),yAxis:()),title:'New%20Visualization',type:histogram))</td>
                    <tr>
                            <td><b>Copy & Paste Discovery Link to FireFox Browser:</b></td>
                    <tr>
                            <td>http://xx.xx.xxcom:5602/app/kibana#/discover?_g=(refreshInterval:(display:'1%20minute',pause:!f,section:2,value:60000),time:(from:now-1h,mode:quick,to:now))&_a=(columns:!(err_num,type,narrative),index:'dplr-qa-logstash-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'err_num:%20%22HP-UX%20Error:%2028%22%20AND%20type:%20%22habitat.exception%22%20AND%20%20host:%20%22hltv0777.hydc.xx.com%22')),sort:!('@timestamp',desc))</td>
                    <tr>
                            <td><b>Narrative:</b></tb>
                    <tr>
                            <td><pre>%{narrative}</pre></td>
                    <tr>
                            <td><b>Tags:</b></tb>
                    <tr>
                            <td><pre>%{tags}</pre></td>
                </table>
            </pre>
            </body>
            </html> " ]
                                            }

                                            throttle {
                                                    before_count =>-1 
                                                    after_count => 1
                                                    period => "3600"
                                                    key => "%{narrative}"
                                                    add_field => [ "tags", "throttled" ]
                                            }
                                    }
                            }
                    }
            }

Results in:

HP-UX Error: 28 from hltv0777.hydc.sbc.com:/opt/app/globalid/data/server/Release-9-0/logs/135.213.40.121_45402_exception.log
Copy & Paste Visualization Link to FireFox Browser:
http://xx.xx.xxcom:5602/app/kibana#/visualize/create?embed=true&type=histogram&indexPattern=dplr-qa-logstash-*&_g=(refreshInterval:(display:'1%20minute',pause:!f,section:2,value:60000),time:(from:now-1h,mode:quick,to:now))&_a=(filters:!(),linked:!f,query:(query_string:(analyze_wildcard:!t,query:'host:%20%22hltv0777.hydc.xx.com%22%20AND%20err_num:%20%22HP-UX%20Error:%22')),uiState:(),vis:(aggs:!((id:'1',params:(),schema:metric,type:count),(id:'2',params:(customInterval:'2h',extended_bounds:(),field:'@timestamp',interval:auto,min_doc_count:1),schema:segment,type:date_histogram),(id:'3',params:(field:host.raw,order:desc,orderBy:'1',size:5),schema:group,type:terms)),listeners:(),params:(addLegend:!t,addTimeMarker:!f,addTooltip:!t,defaultYExtents:!f,mode:stacked,scale:linear,setYExtents:!f,shareYAxis:!t,times:!(),yAxis:()),title:'New%20Visualization',type:histogram))

Copy & Paste Discovery Link to FireFox Browser:
http://xx.xx.xx.com:5602/app/kibana#/discover?_g=(refreshInterval:(display:'1%20minute',pause:!f,section:2,value:60000),time:(from:now-1h,mode:quick,to:now))&_a=(columns:!(err_num,type,narrative),index:'dplr-qa-logstash-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'err_num:%20%22HP-UX%20Error:%2028%22%20AND%20type:%20%22habitat.exception%22%20AND%20%20host:%20%22hltv0777.hydc.xx.com%22')),sort:!('@timestamp',desc))

Narrative:
No space left on device

Tags:
grok0714


#9

Also, some of the credit goes to using the "Share Link" feature of Kibana. It seems to do some magic, but the table is still required.


#10

This still does not work dynamically, i.e., replacing field names with text.


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.