[Solved] ELK - Logstash not processing data

Elastic Stack Logstash
1

for starters, please excuse me if I am not posting the correct details or in the correct format. If there is anything I need to post, please let me know.

History:
I was running security onion and upgraded to their Alpha using ELK. Everything was running fine, after I had to delete the queue. Then new data came through and it was great.
I then upgraded to their recently released beta, and well... Here I am.

Issues:
For starters, the docker containers so-kibana, logstash and elasticsearch wouldn't run because they didn't have permission to access their own logs (weird). But that's now changed, and they run fine.

Now in Kibana, there is no data (unless I backdate the filter to pre-update). However, there is recent data on the "Squert" page.

I've run:
curl -XGET http://localhost:9200/_cat/indices
@security-sbc-ihsw:/var/log/nsm⟫ curl -XGET http://localhost:9200/_cat/indices
green open logstash-ids-2017.10.30 lT0QX-__QVmGXpyUdjqdHw 1 0 32609 0 10.7mb 10.7mb
green open logstash-syslog-2017.10.30 fXUlBC_hSOm9K7rw0P_1dA 1 0 81955 0 32.3mb 32.3mb
green open logstash-ids-2017.11.04 NySacsHZTuKVcDYo6tMXiA 1 0 557 0 466.6kb 466.6kb
green open logstash-ids-2017.10.25 KId7M7kPQ-6663-FsMqTTg 1 0 24 0 62.2kb 62.2kb
green open logstash-ids-2017.11.02 EH-fKMbPQHaxeVD8ej03Vw 1 0 28021 0 9.1mb 9.1mb
green open logstash-ids-2017.11.03 Bl4VVIFTQmCgFEvJlqxlqw 1 0 10416 0 3.5mb 3.5mb
green open logstash-syslog-2017.11.02 xNrVCywKQrCQc2dTIQhACw 1 0 74159 0 26.7mb 26.7mb
green open logstash-bro-2017.10.29 JanqpgL9QbiU3vskgMQX9Q 1 0 525036 0 448.2mb 448.2mb
[plus heaps more]

so-status - gave me all [ OK ]'s.

so-elastic-status

image
image.png1271×305 24.6 KB

The following file paths are populated:
/nsm/sensor_data/security-sbc-ihsw-eth1/dailylogs/2017-11-13 -
Filled with snort logs

/nsm/sensor_data/security-sbc-ihsw-eth1/ -
Argus
dailylogs
portscans
sancp
snort-1
snort-1.stats

/nsm/bro/logs -
Filled with logs

/nsm/logstash/queue/main -
checkpoint.head
.lock
page.0

If anyone could help me troubleshoot this, it would be appreciated.

Update:
The issue was that for some reason, logstash wasn't processing the data. Turns out the security onion config script was adding duplicate lines to the syslog-ng config and crashing the service.

All good now. Thanks for the help.

2

FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats and APM feel left out! :wink:

I don't know Security Onion in detail, but it looks like data is in Elasticsearch, you can see the logstash- type indices, even though they are from last week.

With that in mind, what are you expecting to see here? If it's more data then we can look at ingestion, if it's that you want to see the data that's already there then Kibana should show it, just check the time range.

3

hah my bad "Elastic Stack"!

So in Kibana there is no data at all. Even though there are logs/data in the directories and in Squert.

The only thing I can think of is either:

1 - Kibana isn't reading from the right directory
or
2 - Whatever is responsible for sending the data to Kibana isn't sending it to the right place.

I'm assuming the correct place is /nsm/logstash/queue/main

However I only have the files as listed in my original post...
Is there any chance the .lock file is stopping files from going in there?

4

Nothing sends data to Kibana, it is a front end for Elasticsearch, it reads whatever is in there. Just wanted to make that clear given you're new to the stack :slight_smile:

Check out https://www.elastic.co/guide/en/kibana/5.6/tutorial-define-index.html and use the steps there to check if there are index patterns defined. If not, add them based on the output from _cat/indices above. If there are, then head to Discover and then check that the time range (top right) is sufficiently long enough to show the last 30 days.

5

by sends data, I meant more as, a process moves the data to a folder/directory that kibana uses. Perhaps some post filter.

Discovery is the exact same as the dashboard page of Kibana. There is no data since 6th of November (upgrade from Alpha to Beta).

6

Kibana calls the Elasticsearch APIs, it doesn't store any of the data it reads anywhere local. So that means that the ingestion part isn't working.

How does this app send data to the Elastic Stack exactly?

7

So I had a look at the 'management tab' and played with the indexes.

Now on the Discover page I have found:
NB: My * isn't appearing below and comes up below. So I'll use % for wildcard

%:logstash-% index/filter/whatever it is, the default option. Yields no results.

% - gives me results

Elastalert_status% - gives me results.....

So now I wonder why logstash* isn't working

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.