# Logstash config not creating an index in elasticsearch

**URL:** <https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736>\
**Category:** Logstash\
**Created:** [August 19, 2019, 12:05pm UTC](https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736 "2019-08-19T12:05:38Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![e526e9a7b99f61623fe7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/e526e9a7b99f61623fe7/32/48485_2.png) [@e526e9a7b99f61623fe7](https://discuss.elastic.co/u/e526e9a7b99f61623fe7)\
**Post date:** [August 19, 2019, 12:05pm UTC](https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736/1 "2019-08-19T12:05:38Z")

</div>

Hey elastic team, ive been trying to write a filter for json files so that it gets indexed into elasticsearch but its not creating an index in elastic search. There is no error in the logstash config file.  
This is a sample of json file :  
{"asctime": "2019-08-13 06:17:45,132", "filename": "taskinstance.py", "lineno": 616, "levelname": "INFO", "message": "Dependencies all met for \<TaskInstance: tutorial.print\_date 2019-08-13T06:17:05.595788+00:00 [queued]\>", "dag\_id": "tutorial", "task\_id": "print\_date", "execution\_date": "2019\_08\_13T06\_06\_05\_595788", "try\_number": "1"}  
{"asctime": "2019-08-13 06:17:45,132", "filename": "taskinstance.py", "lineno": 834, "levelname": "INFO", "message": "\n--------------------------------------------------------------------------------", "dag\_id": "tutorial", "task\_id": "print\_date", "execution\_date": "2019\_08\_13T06\_06\_05\_595788", "try\_number": "1"}  
{"asctime": "2019-08-13 06:17:45,132", "filename": "taskinstance.py", "lineno": 835, "levelname": "INFO", "message": "Starting attempt 1 of 2", "dag\_id": "tutorial", "task\_id": "print\_date", "execution\_date": "2019\_08\_13T06\_06\_05\_595788", "try\_number": "1"}  
{"asctime": "2019-08-13 06:17:45,132", "filename": "taskinstance.py", "lineno": 836, "levelname": "INFO", "message": "\n--------------------------------------------------------------------------------", "dag\_id": "tutorial", "task\_id": "print\_date", "execution\_date": "2019\_08\_13T06\_06\_05\_595788", "try\_number": "1"}

There are multiple json files that has the same format.  
This is my config file:  
input  
{  
file  
{  
codec =\> multiline  
{  
pattern =\> '^{'  
negate =\> true  
what =\> previous  
}  
path =\> [“/Users/raks/Documents/test.json"]  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
exclude =\> "\*.gz"  
}  
}

```
filter 
{
mutate

{
    replace => ["message", "%{message}}"]
    gsub => ['message','\n','']
}
if [message] =~ /^{.*}$/ 
{
    json { source => message }
}

}
output {
elasticsearch {
  hosts => [“elasticsearch:9200"]
  index => "rakstest"
}
}

```

Any help would be really appreciated. Thanks!!

---

<div class="post-metadata">

**Author:** ![e526e9a7b99f61623fe7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/e526e9a7b99f61623fe7/32/48485_2.png) [@e526e9a7b99f61623fe7](https://discuss.elastic.co/u/e526e9a7b99f61623fe7)\
**Post date:** [August 19, 2019, 12:19pm UTC](https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736/2 "2019-08-19T12:19:54Z")

</div>

The output i desire is that the fields present in the JSON should be indexed in elastic search so that i can visualise that in kibana.  
So I basically want asctime,filename,lineno,levelname,message,dag\_id,task\_id,execution\_date,try\_number as fields in kibana!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 19, 2019, 1:24pm UTC](https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736/3 "2019-08-19T13:24:49Z")

</div>

The input file you show has a complete JSON object on each line, so you can remove the multiline codec and the mutate. All you need is a json filter.

```
"execution_date" => "2019_08_13T06_06_05_595788",
       "task_id" => "print_date",
      "filename" => "taskinstance.py",
        "lineno" => 616,
    "try_number" => "1",
        "dag_id" => "tutorial",
     "levelname" => "INFO",
       "asctime" => "2019-08-13 06:17:45,132"
```

---

<div class="post-metadata">

**Author:** ![e526e9a7b99f61623fe7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/e526e9a7b99f61623fe7/32/48485_2.png) [@e526e9a7b99f61623fe7](https://discuss.elastic.co/u/e526e9a7b99f61623fe7)\
**Post date:** [August 19, 2019, 1:42pm UTC](https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736/4 "2019-08-19T13:42:01Z")

</div>

Hey @Badger! Thank you for the response! I have few doubts regarding this, the JSON file contains multiple documents and ive provided just a sample, will all the lines in the json file get considered? and also i need the message values also from the json

---

<div class="post-metadata">

**Author:** ![e526e9a7b99f61623fe7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/e526e9a7b99f61623fe7/32/48485_2.png) [@e526e9a7b99f61623fe7](https://discuss.elastic.co/u/e526e9a7b99f61623fe7)\
**Post date:** [August 20, 2019, 1:50am UTC](https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736/5 "2019-08-20T01:50:43Z")

</div>

Hey @Badger is this what is required to be changed in the config file?

Config file:

```
input{
file{
 type => "json"
 path => "/Users/rakshith.bhat/Documents/*.json"
 start_position => "beginning"
 sincedb_path => "/dev/null"
    }
}
filter{
    json{
        source => "message"
    }
}
output {
elasticsearch{
hosts => "elasticsearch:9200"
index => "jsontest"
}
stdout { codec => rubydebug }
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 20, 2019, 11:47am UTC](https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736/6 "2019-08-20T11:47:17Z")

</div>

That looks right to me.

---

<div class="post-metadata">

**Author:** ![e526e9a7b99f61623fe7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/e526e9a7b99f61623fe7/32/48485_2.png) [@e526e9a7b99f61623fe7](https://discuss.elastic.co/u/e526e9a7b99f61623fe7)\
**Post date:** [August 21, 2019, 2:01am UTC](https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736/7 "2019-08-21T02:01:13Z")

</div>

Thanks @Badger! The solution works!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 18, 2019, 2:01am UTC](https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736/8 "2019-09-18T02:01:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.