Hello,
I'm working on sending my Snort logs to ELK. So far, if i send it directly to Elasticsearch. It works, but obviously with poor formatting.
However, when sending it to logstash; I recieve some weird issues.
Here is my logstash conf file:
input {
tcp{
host => "192.168.0.36"
port => 5044
codec => plain {
charset => "US-ASCII"
}
}
}filter {
csv {
columns => ["datetime","sig_gen","sig_id","sig_rev","msg","proto","src_ip","src_port","dst_ip","dst_port","packet","category","priority","additional"]
}
}output {
elasticsearch {
hosts => [ "192.168.0.36:9200" ]
}
}
Once it is received in Kibana, it is displayed as:
"message": "T��yT����\u00025���_\f�(��T�+Pe�n��J��8\u001f��\u0015ow�a1�v��8��^9V����2VJ(\u0004P�X����D��!~J�\u0018\u0014�����4��Q��C���r���e�\u001f�U�\u00186\u000f����N�����/FU�`8�y�4?:��",
I have ensured that the charset matches up (alert.csv in snort is:
> alert.csv: text/plain; charset=us-ascii
Finally, when logstash is running; i get the error:
[2018-03-14T19:34:05,131][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"�a�K%����(G�0*Q��B���I1\u0018��(��T+P�|\u0018���Q�..��$I>K���s��\u001C\f7@���g�\t\u0004��\f�\u0001�\u0002\u0001!�b��\t��e���C�\u00102\u000F�a��G�b�p;���\u000E���b�������Vj�:s#V�dk�JW�r��\u00116�3��t�]�Z\u00174�Hu\u000E��$i����<�\e 1�\u000FT'$\u007F�R�-��p\u0010���'Yw)%v\v�C��\t�\u0003�\r3\u00180�������#�r�������\f���_s;�����`W��]W�����8/�\"F�\u0010���\u0018�=]3��}�����p\u0019�{J_%I���\u0014\u0012��U\\\r��\u001Ak��+1\u0005 �\u0003����^s\u0012\u007F{\e��Y�O\u000F\u0014�j�.e�q`��p�U\u0000r�$�\u0016\u0019f>\u0014\a<3mx4��|\u0004x��c��r\u0006����3F\u0004[i�\u0003Z\r��\u0016�Lh�#�{���$i����\f�4�", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}
[2018-03-14T19:34:36,183][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"2W\u0000\u0000\u0000\u00052C\u0000\u0000\u0002Ix^���N�L\u0014\u0005p�������������]��D�\u0012\u0012����L�\t�\u0014��3�Z�<I�w�\b\v\u0010%�V\u0011���������gY6����\e8Nu\u0013b��50(!���G�>���fY��t\u001F\u0001��\t�/|��70\v>\u0001��^�Y�\t\u0010���\u0000\f�n\u000E\b����k���*W�E���~\u001E�a���h�]�b��i���O�<n\u0000�[.cH���ChB��*\u0000��#IG�d�YR����\u000EQ\"\t�\u0016/�trv>�BLu�S�����\u000F~���u\u0018�w}\u001A^�}������%����\u001Ce�ri\\.r�\u0010e�ri\\.r)\u0010�c!XY�'\\U\\I���b;fcyL�&(>\e���h\be�� kKt��\biP�F�\u0002\u0010f�'�\eh}\u0013��^D@Xu1��\t�P/\" lB\u001F��\u0005\u0006��\\�v�n���g�", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}
[2018-03-14T19:34:36,185][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"��3f��\"{H3�\u0018m:�����Q<=�r�$\u001A\u0019�}�P 9go����\b5j�[���������Uz�����JdK+�!Y~�/+X9vc\u0016S�����?T�\u001D����\fiY���%���G\u0000\u0000\u0000��tb@�2W\u0000\u0000\u0000\u00012C\u0000\u0000\u0001�x^\u0000�\u0001\\�2J\u0000\u0000\u0000\u0001\u0000\u0000\u0001�{\"@timestamp\":\"2018-03-14T19:32:50.797Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"doc\",\"version\":\"6.2.2\"},\"source\":\"/var/log/snort/alert.csv\",\"offset\":4690,\"message\":\"03/14-19:32:43.464851 ,1,1384,8,\\\"MISC UPnP malformed advertisement\\\",UDP,192.168.0.10,54819,239.255.255.250,1900,70:4D:7B:67:B4:2C,01:00:5E:7F:FF:FA,0x150,,,,,,255,0,3724,322,67588,,,,\",\"beat\":{\"version\":\"6.2.2\",\"name\":\"ids\",\"hostname\":\"ids\"}}\u0001\u0000\u0000��\u0016\u001Ap�2W\u0000\u0000\u0000\u000F2C\u0000\u0000\u0004�x^��]k\eG\u0017\a������+��\f'�sf�����%Y�@� 1����<�\u0005���n����hiZ", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}
[2018-03-14T19:35:07,195][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"�BA�vx�I\u0012\"\u0005&b��9�!\u000F5zbS�\u001F��<4�FH��BDDt\u0006e�PYm\u0003\u0006���t�Q#!�J\u001D\v!�\u0013b�L��\u007Fs�]�\u0006���-L��%Sx�I\u001FqUq%_�d�&��0�/D����i/LFj�\"���v�$7�L�J���J6C����\\P�*�\u0013��c�J[C\v���\e��h\u0019^P2�J����d\u001A�x<b\u0013�)\u001E\u001Fq9~�����[g\u001E���=Z��kc6\u0012��?[L��{���r~��\"\u001F��'V���i���\a�^��/FK�D�����\u0012}p\u001F\u001F��j�;\u0000\u0000���n@I2W\u0000\u0000\u0000\u00052C\u0000\u0000\u0002Px^��Mk�N\u0010\u0006p���=��'��iw5�TV\f)\u0004rh.%����\u0005�6h��%������.!!i��\u0018�(]���yF|��l�e��wp������[ \u0010��\u0013&O���K��x���~\u0006���'7w�\u0001����\u0004\u0004�f���%@H�n=\u0010��\f\u0010���M��@�\"\u0017�A��K?�@0Z�~�", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}
I've tried to supply as much information as possible, if there is any more information required; please say so.