# Logstash configuration not creating index in elasticseach

**URL:** <https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540>\
**Category:** Logstash\
**Created:** [August 2, 2019, 1:24pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540 "2019-08-02T13:24:09Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\
**Post date:** [August 2, 2019, 1:24pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540/1 "2019-08-02T13:24:09Z")

</div>

i am trying to create index based on quarter and each document is unique in elasticsearch via logstash. the data is coming from multiple xml files. there seems to be no error in configuration file and pipeline is started but i am unable to see index created in elasticsearch.

ELK version is 7.2  
Java is 1.8  
OS : windows 10

Please find below my logstash configuration:-

input {  
file  
{  
path =\> "D:\xml\*.xml"  
start\_position =\> "beginning"  
sincedb\_path =\> "NUL"  
exclude =\> "\*.gz"  
type =\> "xml"  
codec =\> multiline {  
pattern =\> "\<?xml"  
negate =\> "true"  
what =\> "previous"  
}  
}  
}

filter  
{

```
fingerprint {
    method => "SHA256"
    source => ["abc", "def", "ghi"]
    concatenate_sources => true
    target => "documentId"
}

```

xml {  
source =\> "message"  
store\_xml =\> false  
xpath =\> ["/Metadata/alphabet/BILL", "bill"]  
}

split {  
field =\> "bill"  
}

xml  
{  
source =\> "message"  
store\_xml =\> false  
xpath =\> [  
"/bill/abc/text()", "abc",  
"/bill/def/text()", "def",  
"/bill/ghi/text()", "ghi",  
"/bill/jkl/text()", "jkl" ]  
}

ruby {  
code =\> "  
event.set('quarter','q' + ((Time.now.month/3.0).ceil).to\_s + '-' + Time.now.year.to\_s )"  
}

}

output  
{  
elasticsearch  
{  
codec =\> json  
hosts =\> "localhost:9200"  
index =\> "customerdoc-%{quarter}"  
document\_id =\> "%{documentId}"  
}  
stdout  
{  
codec =\> rubydebug  
}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 2, 2019, 1:35pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540/2 "2019-08-02T13:35:26Z")

</div>

Do not use backslash in the path option of a file input, use forward slash.

---

<div class="post-metadata">

**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\
**Post date:** [August 2, 2019, 5:00pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540/3 "2019-08-02T17:00:13Z")

</div>

Thanks @Badger . that 's correct.

---

<div class="post-metadata">

**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\
**Post date:** [August 2, 2019, 5:02pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540/4 "2019-08-02T17:02:44Z")

</div>

Now the index is created but all the xml is not split into multiple documents but a single document is created and the entire xml is copied to the message field.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 2, 2019, 5:31pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540/5 "2019-08-02T17:31:09Z")

</div>

Unless you can show us what the XML looks like it will be hard to help you.

---

<div class="post-metadata">

**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\
**Post date:** [August 2, 2019, 6:11pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540/6 "2019-08-02T18:11:20Z")

</div>

```
<?xml version="1.0"?>
<Metadata xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation='XML.xsd'>
  <Total-Bills-Processed>165</Total-Bills-Processed>
  <BILL-DATA>
    <BILL>
  <abc>191560004687</abc>
  <def>08012018</def>
  <ghi>906202553</ghi>
  <jkl>191560004687.08012018.PROD.pdf</jkl>
  </BILL>
 <BILL>
  <abc>191560002058</abc>
  <def>06012019</def>
  <ghi>909168139</ghi>
  <jkl>191560002058.06012019.PROD.pdf</jkl>
</BILL>
 </BILL-DATA>
</Metadata>

```

required output in elasticsearch should be :-

doc1  
{  
abc:191560004687  
def:08012018  
ghi:906202553  
jkl:191560004687.08012018.PROD.pdf  
}

doc2  
{  
abc:191560002058  
def:06012019  
ghi:909168139  
jkl:191560002058.06012019.PROD.pdf  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 2, 2019, 6:33pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540/7 "2019-08-02T18:33:52Z")

</div>

The first problem is your multiline codec. It produces a single event that only contains

```
   "message" => "<?xml version=\"1.0\"?>",

```

There is no "\<?xml..." line at the end of the file to trigger the rest of the file to be flushed to the pipeline. If you add auto\_flush\_interval =\> 1 then it will get flushed after a 1 second timeout.

Personally I would ingest the entire file as a single event by using a pattern that does not match plus a timeout.

```
codec => multiline {
    pattern => "^Spalanzani"
    negate => true
    what => previous
    auto_flush_interval => 1
}

```

Once you have an event to work on this bit works OK

```
    xml {
        source => "message"
        store_xml => false
        xpath => ["/Metadata/BILL-DATA/BILL", "bill"]
    }
    split {
        field => "bill"
    }

```

Although I would add 'remove\_field =\> ["message"]' to the xml filter so that message is removed if it is successfully parsed.

At that point I would have chosen to do

```
    xml { source => "bill" target => "[@metadata][theXML]" force_array => false remove_field => ["bill"] }
    ruby { code => 'event.get("[@metadata][theXML]").each { |k, v| event.set(k, v) }' }

```

(which is really a matter of taste) which will get you

```
{
      "tags" => [
    [0] "multiline"
],
       "jkl" => "191560002058.06012019.PROD.pdf",
       "abc" => "191560002058",
"@timestamp" => 2019-08-02T18:31:17.751Z,
       "def" => "06012019",
       "ghi" => "909168139"
}
```

---

<div class="post-metadata">

**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\
**Post date:** [August 3, 2019, 7:22pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540/8 "2019-08-03T19:22:13Z")

</div>

Thanks @Badger , but in the logstash console , the output looks fine but when i view the index in kibana , only the last document(1 document) in the last file among the list of files is visible. could you help with that?

Also can you tell me why below part of the code does not work :-

```
xml
{
source => "bill"
store_xml => false
xpath => [
"/bill/abc/text()", "abc",
"/bill/def/text()", "def",
"/bill/ghi/text()", "ghi",
"/bill/jkl/text()", "jkl" ] 
 remove_field => ["bill"] 
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 3, 2019, 10:01pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540/9 "2019-08-03T22:01:11Z")

</div>

> [@Abdul\_Gaffar\_Shaikh](#):
>
> Also can you tell me why below part of the code does not work

It does not work because element names are case sensitive. bill cannot be used to refer to BILL, which is what your document has. Also, you probably want force\_array =\> false on that.

---

<div class="post-metadata">

**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\
**Post date:** [August 13, 2019, 12:41pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540/11 "2019-08-13T12:41:42Z")

</div>

thanks @Badger

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 10, 2019, 12:41pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540/12 "2019-09-10T12:41:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.