Hi Team,
We were using filebeat 2.x for quite long time. Recently we decided to upgrade to 5.5 version of ELK stack along with kafka.
Filebeat -> Kafka-cluster -> logstash -> ES -> Kibana
We are able to successfully configure the cluster and logs are flowing as expected.
Original log on the disk -
2017-07-24 00:00:17,865 INFO [GlobalWebRedirectServlet] (ajp-/0.0.0.0:8009-199) Server ID: estore2a, Ip 173.239.240.38 -- ca.xxx.com/store/services/stibo/stiboLiteOrderingTable.jsp?orderGridID=orderTable_CAAAA13466-0B&catalogNumbers=CAAAA13466-0B,CAAAA13466-30,CAAAA13466-0I&initLoad=false&itemCount=3&_=1500868818879 - User Org Id : u920480552canada_cash_org - Is Anonymous : true - Client SSL Protocol: TLSv1.2 - Client SSL Cipher : ECDHE-ECDSA-AES128-GCM-SHA256
Output by filebeat -
{"@timestamp":"2017-08-26T12:09:11.355Z","beat":{"hostname":"lxelkprd01","name":"lxelkprd01","version":"5.5.0"},"input_type":"log","message":"2017-07-24 00:00:17,865 INFO [GlobalWebRedirectServlet] (ajp-/0.0.0.0:8009-199) Server ID: estore2a, Ip 173.239.240.38 -- ca.vwr.com/store/services/stibo/stiboLiteOrderingTable.jsp?orderGridID=orderTable_CAAAA13466-0B\u0026catalogNumbers=CAAAA13466-0B,CAAAA13466-30,CAAAA13466-0I\u0026initLoad=false**\u0026itemCount=3\u0026**_=1500868818879 - User Org Id : u920480552canada_cash_org - Is Anonymous : true - Client SSL Protocol: TLSv1.2 - Client SSL Cipher : ECDHE-ECDSA-AES128-GCM-SHA256","offset":485,"source":"/apps/elk/testsetup/test6.log","type":"log"}
I am seeing & is getting transformed to \u0026. I tried couple of encodings however I am not able to find any solution that works. I believe it has to do with HTML escaping and I saw some suggestions about using JSON. However this is log from server and not in JSON format.