# Atlassian access logs Index not getting created or data not sent / visible in Opensearch **URL:** \ **Category:** Logstash\ **Created:** [August 26, 2023, 11:17am UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742 "2023-08-26T11:17:40Z")\ **Posts on this page:** 11\ **Page:** 2
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [August 27, 2023, 5:37pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742/22 "2023-08-27T17:37:53Z")
There are multiple errors. First, if you have the compressed files in the same directory, you need to add a [exclude line](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-exclude) in your input. For example, add this line inside the `file` input. ```auto exclude => "*.bz2" ``` Second, your elasticsearch disk is full: > :error=\>{"type"=\>"cluster\_block\_exception", "reason"=\>"index [jira-access-log-2023.08.27] blocked by: [TOO\_MANY\_REQUESTS/12/ **disk usage exceeded flood-stage watermark** , index has read-only-allow-delete block];"}} You need to free up some space in your elasticsearch node, it cannot write anything now. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 27, 2023, 6:10pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742/23 "2023-08-27T18:10:57Z")
> [@danmed](#): > > `path => "/path/to/the/access_log.*"` Perhaps you can avoid reading the zip files by using a pattern like `/path/to/the/access_log.[-0-9]{10}` ---
**Author:** ![danmed](https://avatars.discourse-cdn.com/v4/letter/d/aeb1de/32.png) [@danmed](https://discuss.elastic.co/u/danmed)\ **Post date:** [August 27, 2023, 6:39pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742/24 "2023-08-27T18:39:23Z")
Thanks leandrojmp, I added the exclude like you said, but I **still** see it processing .bz2 files after I restarted LS. ```auto input { file { path => "/path/to/access_log.*" start_position => "beginning" exclude => "*.bz2" } } ``` Additionally, I am using this dissect filter below which seems to work as welll.. ```auto filter { dissect { mapping => { "message" => "%{ip} %{id} %{user} [%{[@metadata][timestamp]} %{timezone}] %{message}" } } date { match => ["[@metadata][timestamp]", "dd-MMM-yyyy:HH:mm:ss" ] target => "@timestamp" } } ``` But even with it, the problem of it processing the .bz2 file remains. Is there anything else I need to do to **not** process compressed files? Thanks ---
**Author:** ![danmed](https://avatars.discourse-cdn.com/v4/letter/d/aeb1de/32.png) [@danmed](https://discuss.elastic.co/u/danmed)\ **Post date:** [August 27, 2023, 6:43pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742/25 "2023-08-27T18:43:58Z")
Thanks Badger, I will try it out. ---
**Author:** ![danmed](https://avatars.discourse-cdn.com/v4/letter/d/aeb1de/32.png) [@danmed](https://discuss.elastic.co/u/danmed)\ **Post date:** [August 27, 2023, 6:53pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742/26 "2023-08-27T18:53:43Z")
Badger, do you mean /path/to/access\_log.????-??-?? This way I can get: ```auto ls -l /path/to/access_log.????-??-?? -rw-r----- 1 182208 Aug 27 20:48 /path/to/log/access_log.2023-08-27 ``` or would that not work? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 27, 2023, 6:59pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742/27 "2023-08-27T18:59:44Z")
That is another regexp that will mostly have the same effect. ---
**Author:** ![danmed](https://avatars.discourse-cdn.com/v4/letter/d/aeb1de/32.png) [@danmed](https://discuss.elastic.co/u/danmed)\ **Post date:** [August 27, 2023, 7:12pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742/28 "2023-08-27T19:12:00Z")
yes that is working, the file is being processed with the '?'s . Now to get the filter right, because I get an error in my **date parsing**. See tags below. I could try switching to grok again, which I would prefer... But here's the error when I use dissect [`2023-08-27T20:56:35,321][INFO][logstash.outputs.opensearch][jira-acc-pipeline][06c5861d0b1e2b5a4752a90d08ee0a042812fa2e1863bae9615aa42c05232fda] Retrying failed action {:status=>429, :action=>["index", {:_id=>nil, :_index=>"access_log-new2-2023.08.27", :routing=>nil}, {"@timestamp"=>2023-08-27T18:56:02.010709461Z, "message"=>"\"GET /rest/api/lates....................................etc............ HTTP/1.0\" 404 54 11 \"-\" \"Atlassian HttpClient 0.23.0 / Atlassian JIRA Rest Java Client-4.0.3-sc (0) / Default\" \"xyz1234\"", "@version"=>"1", "log"=>{"file"=>{"path"=>"/path/to/access_log.2023-08-27"}}, "host"=>{"name"=>"myhost.com"}, "event"=>{"original"=>"nnn.nnn.nnn.nn 123x234x567x123 some.user.name [27/Aug/2023:11:00:21 +0200] \"GET /rest/api/latest...........................etc.................... HTTP/1.0\" 404 54 11 \"-\" \"Atlassian HttpClient 0.23.0 / Atlassian JIRA Rest Java Client-4.0.3-sc (0) / Default\" \"xyz1234\""}, "tags"=>["_dateparsefailure"], "ip"=>"nnn.nnn.nnn.nn", "user"=>"some.user.name", "id"=>"123x234x567x123, "timezone"=>"+0200"}], :error=>{"type"=>"cluster_block_exception", "reason"=>"index [access_log-new2-2023.08.27] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}}` plus I don't know why I have the disk usage issue, I have enough space. I could try to reset it's disk usage params for this particular index, I don't know. ---
**Author:** ![danmed](https://avatars.discourse-cdn.com/v4/letter/d/aeb1de/32.png) [@danmed](https://discuss.elastic.co/u/danmed)\ **Post date:** [August 27, 2023, 7:45pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742/29 "2023-08-27T19:45:11Z")
I think I managed to parse the date / time. Because there is no date parsing failure. But, I still get the disk issue though. The disk on which logstash stores it's logs and where it's installed has plenty of space left. What exactly does is mean then? Is it talking of disk space on the target opensearch server's disk? Here's another log line below. The timestamp seems ok, but the cluster\_block\_exception is the error: `[2023-08-27T21:28:54,683][INFO][logstash.outputs.opensearch][jira-acc-pipeline][b1bc0ac19696818e24519653a5a52c38186c3ca52c116d9b501783e7d6aa28db] Retrying failed action {:status=>429, :action=>["index", {:_id=>nil, :_index=>"jira-access-new3-2023.08.27", :routing=>nil}, {"@timestamp"=>2023-08-27T19:28:50.000Z, "log"=>{"file"=>{"path"=>"/path/to/access_log.2023-08-27"}}, "@version"=>"1", "message"=>"\"GET /rest/api/2/searc..............etc.....Results=2000 HTTP/1.0\" 200 53 17 \"-\" \"Java/1.8.0_45\" \"xyz1234\"", "event"=>{"original"=>"nnn.nnn.nnn.nn 1234x123445x1 some.user.name [27/Aug/2023:21:28:50 +0200] \"GET /rest/api/2/search..............................etc.................Results=2000 HTTP/1.0\" 200 53 17 \"-\" \"Java/1.8.0_45\" \"xyz1234\""}, "ip"=>"nnn.nnn.nnn.nn", "host"=>{"name"=>"myhost.com"}, "id"=>"1234x123445x1", "user"=>"some.user.name"}], :error=>{"type"=>"cluster_block_exception", "reason"=>"index [jira-access-new3-2023.08.27] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}}` Thanks. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 27, 2023, 7:57pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742/30 "2023-08-27T19:57:01Z")
> [@danmed](#): > > Is it talking of disk space on the target opensearch server's disk? Yes. I don't know opensearch, but I believe when I was using elasticsearch that once ES had stopped ingesting data due to a disk space issue it was not enough to free up disk space, you needed to tell ES that you had done so. This thread has got you to the point where logstash is reading the right logs from a file input. You should probably start a new thread if you have filter issues. ---
**Author:** ![danmed](https://avatars.discourse-cdn.com/v4/letter/d/aeb1de/32.png) [@danmed](https://discuss.elastic.co/u/danmed)\ **Post date:** [August 27, 2023, 8:04pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742/31 "2023-08-27T20:04:18Z")
Thanks Badger. I think the file input issue has been resolved. Even the filter issue I think is resolved because : the only error I see is for the disk space - and if that is for the target server then I can close this issue for this thread here. I fully appreciate all the help from everyone. Thanks again ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 24, 2023, 8:05pm UTC](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742/32 "2023-09-24T20:05:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/atlassian-access-logs-index-not-getting-created-or-data-not-sent-visible-in-opensearch/341742.md?page=1)