# Index not creating

**URL:** <https://discuss.elastic.co/t/index-not-creating/282345>\
**Category:** Logstash\
**Created:** [August 24, 2021, 12:29pm UTC](https://discuss.elastic.co/t/index-not-creating/282345 "2021-08-24T12:29:46Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\
**Post date:** [August 24, 2021, 12:29pm UTC](https://discuss.elastic.co/t/index-not-creating/282345/1 "2021-08-24T12:29:47Z")

</div>

After configuring filter.conf and logstash input index not showing in Kibana. Successfully tested logstash conf files. No errors.

Below are the conf files

Filter.conf

```auto
filter {
  if [type] == "signinattempts" {
      json {
        source => "message"
      }
    }
 }

```

Logstash input & output

```auto
input {
  file {
    type => "signinattempts"
    path => "/path/*.json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

output { 
if [type] == "signinattempts" {
    elasticsearch {
    ssl => true
    ssl_certificate_verification => false
    user => 
    password => 
    action => "index"
    hosts => ["https://localhost:9200"]
    index => "test-events-%{+yyyy.MM.dd}"
   }
  }
}

```

Could somebody can verify the configuration

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [August 24, 2021, 6:17pm UTC](https://discuss.elastic.co/t/index-not-creating/282345/2 "2021-08-24T18:17:11Z")

</div>

Does it work if you remove all of the conditional if statements?

---

<div class="post-metadata">

**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\
**Post date:** [August 24, 2021, 8:00pm UTC](https://discuss.elastic.co/t/index-not-creating/282345/3 "2021-08-24T20:00:28Z")

</div>

No. It is not working with if statements

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [August 24, 2021, 8:07pm UTC](https://discuss.elastic.co/t/index-not-creating/282345/4 "2021-08-24T20:07:29Z")

</div>

Does it work if you just output to the screen?

```auto
output { stdout {} }

```

What does your logstash [log](https://www.elastic.co/guide/en/logstash/current/logging.html) say? Any errors or warnings?

---

<div class="post-metadata">

**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\
**Post date:** [August 24, 2021, 8:09pm UTC](https://discuss.elastic.co/t/index-not-creating/282345/5 "2021-08-24T20:09:12Z")

</div>

No errors or warnings. I can see other if condition logs. Example using same logstash file to fetch syslogs which is working with beat configuration inputs

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [August 24, 2021, 9:50pm UTC](https://discuss.elastic.co/t/index-not-creating/282345/6 "2021-08-24T21:50:30Z")

</div>

You said it happened after you added the filter. Are you saying if you remove the filter the data will flow properly?

Can you post a single message from the log file that you expect to work for this pipeline?

---

<div class="post-metadata">

**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\
**Post date:** [August 24, 2021, 10:04pm UTC](https://discuss.elastic.co/t/index-not-creating/282345/7 "2021-08-24T22:04:07Z")

</div>

What I'm saying is the logstash has other input configurations which are working fine, even adding this one also. But I'm not able to fetch this file data to elk. My previous configurations fetching logs without any issues after adding this. I ran a config test and the result is a success, but indexing is not working. Is it any way to check log stash is taking data from this file and sending it to elasticsearch ?

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [August 24, 2021, 10:06pm UTC](https://discuss.elastic.co/t/index-not-creating/282345/8 "2021-08-24T22:06:44Z")

</div>

Yes. Change your output to this. You are looking on the screen to see if messages are getting through Logstash or not.

You will need to start Logstash not as a service in order to see the results output.

```auto
output { 
if [type] == "signinattempts" {
    stdout { }
  }
}

```

---

<div class="post-metadata">

**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\
**Post date:** [August 24, 2021, 10:09pm UTC](https://discuss.elastic.co/t/index-not-creating/282345/9 "2021-08-24T22:09:18Z")

</div>

Okay will change.  
FYI: I'm running logstash on docker

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [August 24, 2021, 10:09pm UTC](https://discuss.elastic.co/t/index-not-creating/282345/10 "2021-08-24T22:09:57Z")

</div>

Then you probably need to output to a file. I am not that familiar with Docker.

---

<div class="post-metadata">

**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\
**Post date:** [August 25, 2021, 10:11am UTC](https://discuss.elastic.co/t/index-not-creating/282345/11 "2021-08-25T10:11:14Z")

</div>

This is also not worked. ☹

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [August 25, 2021, 10:16am UTC](https://discuss.elastic.co/t/index-not-creating/282345/12 "2021-08-25T10:16:09Z")

</div>

I think in order to help further I would need to see your logstash logs when you try to process this file.

---

<div class="post-metadata">

**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\
**Post date:** [August 25, 2021, 10:24am UTC](https://discuss.elastic.co/t/index-not-creating/282345/13 "2021-08-25T10:24:31Z")

</div>

Here is the log when i starting logstash container

```auto
 logstash.javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/usr/share/logstash/config/conf.d/1password-filter.conf", "/usr/share/logstash/config/conf.d/filebeat-filter.conf", "/usr/share/logstash/config/conf.d/logstash.conf", "/usr/share/logstash/config/conf.d/pfsense-filter.conf", "/usr/share/logstash/config/conf.d/webhook-filter.conf"], :thread=>"#<Thread:0x575d69ef run>"}
04:51:44.933 [[main]-pipeline-manager] INFO logstash.javapipeline - Pipeline Java execution initialization time {"seconds"=>1.68}
04:51:44.951 [[main]-pipeline-manager] INFO logstash.inputs.beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
04:51:45.387 [[main]-pipeline-manager] INFO logstash.javapipeline - Pipeline started {"pipeline.id"=>"main"}
04:51:45.400 [[main]<http] INFO logstash.inputs.http - Starting http input listener {:address=>"0.0.0.0:3233", :ssl=>"false"}
04:51:45.505 [[main]<beats] INFO org.logstash.beats.Server - Starting server on port: 5044
04:51:45.530 [[main]<udp] INFO logstash.inputs.udp - Starting UDP listener {:address=>"0.0.0.0:5044"}
04:51:45.551 [[main]<file] INFO filewatch.observingtail - START, creating Discoverer, Watch with file and sincedb collections
04:51:45.554 [Agent thread] INFO logstash.agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
04:51:45.585 [[main]<udp] INFO logstash.inputs.udp - UDP listener started {:address=>"0.0.0.0:5044", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
04:51:45.775 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}

```

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [August 25, 2021, 10:41am UTC](https://discuss.elastic.co/t/index-not-creating/282345/14 "2021-08-25T10:41:15Z")

</div>

Everything looks normal. It appears Logstash is watching for files at `/path/*.json` but might not be seeing them. Can you verify the path is correct?

---

<div class="post-metadata">

**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\
**Post date:** [August 25, 2021, 10:46am UTC](https://discuss.elastic.co/t/index-not-creating/282345/15 "2021-08-25T10:46:06Z")

</div>

Yes, the path is correct. re-verified

---

<div class="post-metadata">

**Author:** ![rohitarorait82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohitarorait82/32/82981_2.png) [@rohitarorait82](https://discuss.elastic.co/u/rohitarorait82)\
**Post date:** [August 25, 2021, 4:08pm UTC](https://discuss.elastic.co/t/index-not-creating/282345/16 "2021-08-25T16:08:54Z")

</div>

Hi,

Please change index name to small letter, elasticsearch will not be able to understand 'MM' , replace this with mmm and then check in Index management

---

<div class="post-metadata">

**Author:** ![jubin03](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@jubin03](https://discuss.elastic.co/u/jubin03)\
**Post date:** [August 26, 2021, 4:24am UTC](https://discuss.elastic.co/t/index-not-creating/282345/17 "2021-08-26T04:24:42Z")

</div>

Modified the same but didn't work.  
Note: I'm using the same 'MM' format for other syslogs and it is working

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [August 26, 2021, 10:55am UTC](https://discuss.elastic.co/t/index-not-creating/282345/18 "2021-08-26T10:55:13Z")

</div>

1. Verify file path and file contents.
2. Verify you can do a straight input to stdout or file output with no filters.

What I am reading is you are failing step #2. If that's the case then this is probably a docker question which I can't answer. Maybe something to do with mounting or volumes. But you said you are doing something similar already so I don't think you would miss that part.

Below is what I would use to test.

```auto
input {
  file {
    path => "/path/*.json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
output { 
 stdout {}
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 23, 2021, 10:55am UTC](https://discuss.elastic.co/t/index-not-creating/282345/19 "2021-09-23T10:55:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.