# \[Solved\] ELK - Logstash not processing data **URL:** \ **Category:** Logstash\ **Created:** [November 12, 2017, 11:00pm UTC](https://discuss.elastic.co/t/solved-elk-logstash-not-processing-data/107318 "2017-11-12T23:00:48Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Adam\_Black](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@Adam\_Black](https://discuss.elastic.co/u/Adam_Black)\ **Post date:** [November 12, 2017, 11:00pm UTC](https://discuss.elastic.co/t/solved-elk-logstash-not-processing-data/107318/1 "2017-11-12T23:00:48Z")
for starters, please excuse me if I am not posting the correct details or in the correct format. If there is anything I need to post, please let me know. **History:** I was running security onion and upgraded to their Alpha using ELK. Everything was running fine, after I had to delete the queue. Then new data came through and it was great. I then upgraded to their recently released beta, and well... Here I am. **Issues:** For starters, the docker containers so-kibana, logstash and elasticsearch wouldn't run because they didn't have permission to access their own logs (weird). But that's now changed, and they run fine. Now in Kibana, there is no data (unless I backdate the filter to pre-update). However, there is recent data on the "Squert" page. **I've run:** **curl -XGET [http://localhost:9200/\_cat/indices](http://localhost:9200/_cat/indices)** @security-sbc-ihsw:/var/log/nsm⟫ curl -XGET [http://localhost:9200/\_cat/indices](http://localhost:9200/_cat/indices) green open logstash-ids-2017.10.30 lT0QX-\_\_QVmGXpyUdjqdHw 1 0 32609 0 10.7mb 10.7mb green open logstash-syslog-2017.10.30 fXUlBC\_hSOm9K7rw0P\_1dA 1 0 81955 0 32.3mb 32.3mb green open logstash-ids-2017.11.04 NySacsHZTuKVcDYo6tMXiA 1 0 557 0 466.6kb 466.6kb green open logstash-ids-2017.10.25 KId7M7kPQ-6663-FsMqTTg 1 0 24 0 62.2kb 62.2kb green open logstash-ids-2017.11.02 EH-fKMbPQHaxeVD8ej03Vw 1 0 28021 0 9.1mb 9.1mb green open logstash-ids-2017.11.03 Bl4VVIFTQmCgFEvJlqxlqw 1 0 10416 0 3.5mb 3.5mb green open logstash-syslog-2017.11.02 xNrVCywKQrCQc2dTIQhACw 1 0 74159 0 26.7mb 26.7mb green open logstash-bro-2017.10.29 JanqpgL9QbiU3vskgMQX9Q 1 0 525036 0 448.2mb 448.2mb [plus heaps more] **so-status** - gave me all [OK]'s. **so-elastic-status** ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/f/cff0bd57b44480efee25441b663e92ccc3f5d64d.png) The following file paths are populated: **/nsm/sensor\_data/security-sbc-ihsw-eth1/dailylogs/2017-11-13 -** Filled with snort logs **/nsm/sensor\_data/security-sbc-ihsw-eth1/ -** Argus dailylogs portscans sancp snort-1 snort-1.stats **/nsm/bro/logs -** Filled with logs **/nsm/logstash/queue/main -** checkpoint.head .lock page.0 If anyone could help me troubleshoot this, it would be appreciated. **Update:** The issue was that for some reason, logstash wasn't processing the data. Turns out the security onion config script was adding duplicate lines to the syslog-ng config and crashing the service. All good now. Thanks for the help. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 12, 2017, 11:32pm UTC](https://discuss.elastic.co/t/solved-elk-logstash-not-processing-data/107318/2 "2017-11-12T23:32:14Z")
FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats and APM feel left out! 😉 I don't know Security Onion in detail, but it looks like data is in Elasticsearch, you can see the `logstash-` type indices, even though they are from last week. With that in mind, what are you expecting to see here? If it's more data then we can look at ingestion, if it's that you want to see the data that's already there then Kibana should show it, just check the time range. ---
**Author:** ![Adam\_Black](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@Adam\_Black](https://discuss.elastic.co/u/Adam_Black)\ **Post date:** [November 13, 2017, 12:02am UTC](https://discuss.elastic.co/t/solved-elk-logstash-not-processing-data/107318/3 "2017-11-13T00:02:53Z")
hah my bad "Elastic Stack"! So in Kibana there is no data at all. Even though there are logs/data in the directories and in Squert. The only thing I can think of is either: 1 - Kibana isn't reading from the right directory or 2 - Whatever is responsible for sending the data to Kibana isn't sending it to the right place. I'm assuming the correct place is /nsm/logstash/queue/main However I only have the files as listed in my original post... Is there any chance the .lock file is stopping files from going in there? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 13, 2017, 12:15am UTC](https://discuss.elastic.co/t/solved-elk-logstash-not-processing-data/107318/4 "2017-11-13T00:15:14Z")
Nothing sends data to Kibana, it is a front end for Elasticsearch, it reads whatever is in there. Just wanted to make that clear given you're new to the stack 🙂 Check out [https://www.elastic.co/guide/en/kibana/5.6/tutorial-define-index.html](https://www.elastic.co/guide/en/kibana/5.6/tutorial-define-index.html) and use the steps there to check if there are index patterns defined. If not, add them based on the output from `_cat/indices` above. If there are, then head to Discover and then check that the time range (top right) is sufficiently long enough to show the last 30 days. ---
**Author:** ![Adam\_Black](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@Adam\_Black](https://discuss.elastic.co/u/Adam_Black)\ **Post date:** [November 13, 2017, 12:17am UTC](https://discuss.elastic.co/t/solved-elk-logstash-not-processing-data/107318/5 "2017-11-13T00:17:46Z")
by sends data, I meant more as, a process moves the data to a folder/directory that kibana uses. Perhaps some post filter. Discovery is the exact same as the dashboard page of Kibana. There is no data since 6th of November (upgrade from Alpha to Beta). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 13, 2017, 12:32am UTC](https://discuss.elastic.co/t/solved-elk-logstash-not-processing-data/107318/6 "2017-11-13T00:32:14Z")
Kibana calls the Elasticsearch APIs, it doesn't store any of the data it reads anywhere local. So that means that the ingestion part isn't working. How does this app send data to the Elastic Stack exactly? ---
**Author:** ![Adam\_Black](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@Adam\_Black](https://discuss.elastic.co/u/Adam_Black)\ **Post date:** [November 13, 2017, 12:32am UTC](https://discuss.elastic.co/t/solved-elk-logstash-not-processing-data/107318/7 "2017-11-13T00:32:29Z")
So I had a look at the 'management tab' and played with the indexes. Now on the **Discover** page I have found: _NB:_ My \* isn't appearing below and comes up below. So I'll use % for wildcard %:logstash-% index/filter/whatever it is, _the default_ option. Yields no results. % - gives me results Elastalert\_status% - gives me results..... So now I wonder why logstash\* isn't working ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 11, 2017, 12:32am UTC](https://discuss.elastic.co/t/solved-elk-logstash-not-processing-data/107318/8 "2017-12-11T00:32:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.