SweetSecurity in ELK Stack


#1

I am trying to integrate SweetSecurity sensor install on a Raspberry Pi and the Wazuh ELK 6.2.4 stack. Unfortunately it appears that there was a major change going from ELK 5.x.x to ELK 6.x.x, which is detailed here: https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

More detailed information is here: https://logz.io/blog/removal-elasticsearch-mapping-types/

Does anyone know how I can update the install files so that they would work with ELK 6.2.4?

It appears that the original programmer of this tool has used multiple mapping types. Is there a tool I can use to convert these mapping types to something that is compliant with ELK 6.x without breaking everything? I have looked everywhere but I did not find anything useful...

Please let me know if you need any other kinds of information to assist!


#2

Here is one example of the types of errors I receive when trying to compile:

Importing Kibana Index Patterns
Importing PATTERN /home/username/SweetSecurity/kibana/patterns/sweet_security_alerts.json
{u'fields': [{u'count': 0, u'name': u'_id', u'searchable': False, u'analyzed': False, u'aggregatable': False, u'indexed': False, u'doc_values': False, u'type': u'string', u'scripted': False}, {u'count': 0, u'name': u'_index', u'searchable': False, u'analyzed': False, u'aggregatable': False, u'indexed': False, u'doc_values': False, u'type': u'string', u'scripted': False}, {u'count': 0, u'name': u'_score', u'searchable': False, u'analyzed': False, u'aggregatable': False, u'indexed': False, u'doc_values': False, u'type': u'number', u'scripted': False}, {u'count': 0, u'name': u'_source', u'searchable': False, u'analyzed': False, u'aggregatable': False, u'indexed': False, u'doc_values': False, u'type': u'_source', u'scripted': False}, {u'count': 0, u'name': u'_type', u'searchable': True, u'analyzed': False, u'aggregatable': True, u'indexed': False, u'doc_values': False, u'type': u'string', u'scripted': False}, {u'count': 0, u'name': u'addressed', u'searchable': True, u'analyzed': False, u'aggregatable': True, u'indexed': True, u'doc_values': True, u'type': u'number', u'scripted': False}, {u'count': 0, u'name': u'addressedDate', u'searchable': True, u'analyzed': True, u'aggregatable': False, u'indexed': True, u'doc_values': False, u'type': u'string', u'scripted': False}, {u'count': 0, u'name': u'addressedDate.keyword', u'searchable': True, u'analyzed': False, u'aggregatable': True, u'indexed': True, u'doc_values': True, u'type': u'string', u'scripted': False}, {u'count': 0, u'name': u'addressedOn', u'searchable': False, u'analyzed': False, u'aggregatable': False, u'indexed': True, u'doc_values': True, u'type': u'date', u'scripted': False}, {u'count': 0, u'name': u'firstSeen', u'searchable': True, u'analyzed': False, u'aggregatable': True, u'indexed': True, u'doc_values': True, u'type': u'date', u'scripted': False}, {u'count': 0, u'name': u'mac', u'searchable': True, u'analyzed': True, u'aggregatable': False, u'indexed': True, u'doc_values': False, u'type': u'string', u'scripted': False}, {u'count': 0, u'name': u'mac.raw', u'searchable': True, u'analyzed': False, u'aggregatable': True, u'indexed': True, u'doc_values': True, u'type': u'string', u'scripted': False}, {u'count': 0, u'name': u'message', u'searchable': True, u'analyzed': True, u'aggregatable': False, u'indexed': True, u'doc_values': False, u'type': u'string', u'scripted': False}, {u'count': 0, u'name': u'message.raw', u'searchable': True, u'analyzed': False, u'aggregatable': True, u'indexed': True, u'doc_values': True, u'type': u'string', u'scripted': False}, {u'count': 0, u'name': u'source', u'searchable': True, u'analyzed': True, u'aggregatable': False, u'indexed': True, u'doc_values': False, u'type': u'string', u'scripted': False}, {u'count': 0, u'name': u'source.raw', u'searchable': True, u'analyzed': False, u'aggregatable': True, u'indexed': True, u'doc_values': True, u'type': u'string', u'scripted': False}], u'timeFieldName': u'firstSeen', u'title': u'sweet_security_alerts'}
Traceback (most recent call last):
File "setup2.py", line 287, in
importIndexMapping("/home/username/SweetSecurity/kibana/patterns/sweet_security_alerts.json")
File "setup2.py", line 284, in importIndexMapping
esService.index(index='.kibana', doc_type='index-pattern', id=dashboardJson['title'], body=dashboardJson)
File "/home/username/.local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
return func(*args, params=params, **kwargs)
File "/home/username/.local/lib/python2.7/site-packages/elasticsearch/client/init.py", line 319, in index
_make_path(index, doc_type, id), params=params, body=body)
File "/home/username/.local/lib/python2.7/site-packages/elasticsearch/transport.py", line 314, in perform_request
status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
File "/home/username/.local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 180, in perform_request
self._raise_error(response.status, raw_data)
File "/home/username/.local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, u'illegal_argument_exception', u'Rejecting mapping update to [.kibana] as the final mapping would have more than 1 type: [index-pattern, doc]')


#3

Some further errors I received when trying to import mapping types in Python:

ssIndex='curl -XPUT 'localhost:9200/sweet_security_example?pretty' -H 'Content-Type: application/json' -d''
' {"mappings" : {'
' "ports" : {"properties" : {'
' "mac" : {"type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "port" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}},'
' "protocol" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}},'
' "name" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "product" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "version" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "lastSeen": { "type" : "date" }}}, '
' "devices" : { "properties" : { '
' "hostname" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "nickname" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "ip4" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "mac" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "vendor" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "ignore" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "active" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "defaultFwAction" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "isolate" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "firstSeen" : { "type" : "date" }, '
' "lastSeen" : { "type" : "date" }}}, '
' "firewallProfiles" : { "properties" : { '
' "mac" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "destination" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "action" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}}},'
' "sensors" : { "properties" : { '
' "mac" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "sensorName" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "broHealth" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "logstashHealth" : { "type" : "text", "fields": {"keyword": {"type": "keyword"}}}, '
' "diskUsage" : { "type" : "integer"}, '
' "memAvailable" : { "type" : "integer"}, '
' "memPercent" : { "type" : "integer"}, '
' "memConsumed" : { "type" : "integer"}, '
' "firstSeen" : { "type" : "date" }, '
' "lastSeen" : { "type" : "date" }}} '
'}}''

When importing I get the error:

{u'status': 400, u'error': {u'root_cause': [{u'reason': u'Rejecting mapping update to [sweet_security_example] as the final mapping would have more than 1 type: [devices, ports, firewallProfiles]', u'type': u'illegal_argument_exception'}], u'type': u'illegal_argument_exception', u'reason': u'Rejecting mapping update to [sweet_security_example] as the final mapping would have more than 1 type: [devices, ports, firewallProfiles]'}}


(Bill McConaghy) #4

I would suggest opening an issue with the SweetSecurity project, as they will need to update to accommodate the changes between 5 and 6 in Elasticsearch.


#5

I have let the software developer know, but the project only has one main contributor so it will probably be a while until something is available.

Unfortunately what I had to do was blow away the entire ELK stack, then reload version 5.5.1 which is compatible with SweetSecurity. Then I upgraded from version 5.5.1 to 6.3 which was successful, so now it appears to be working normally. It is horrible that I had to use this workaround as it took more than ten hours of my life.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.