# Data can not be indexed

**URL:** <https://discuss.elastic.co/t/data-can-not-be-indexed/137470>\
**Category:** Logstash\
**Created:** [June 26, 2018, 4:01pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470 "2018-06-26T16:01:35Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yong\_Rhee](https://avatars.discourse-cdn.com/v4/letter/y/aeb1de/32.png) [@Yong\_Rhee](https://discuss.elastic.co/u/Yong_Rhee)\
**Post date:** [June 26, 2018, 4:01pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/1 "2018-06-26T16:01:35Z")

</div>

I am trying to play with toy data example from the book. I can not load data because of the following problem.

```
[2018-06-26T11:58:16,824][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_in
dex=>"accidents-2013", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x2db31fdc>], :response=>{"index"=>{"_index"=>"accidents-2013", "_type"=>"doc", "_
id"=>"yWLRPGQBGV7OARY3Aw30", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [accidents-2013] as the 
final mapping would have more than 1 type: [doc, accident]"}}}}
[2018-06-26T11:58:16,824][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_in
dex=>"accidents-2012", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x6b51130b>], :response=>{"index"=>{"_index"=>"accidents-2012", "_type"=>"doc", "_
id"=>"ymLRPGQBGV7OARY3Aw30", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [accidents-2012] as the 
final mapping would have more than 1 type: [doc, accident]"}}}}
[2018-06-26T11:58:16,824][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_in
dex=>"accidents-2012", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0xa63a0ee>], :response=>{"index"=>{"_index"=>"accidents-2012", "_type"=>"doc", "_i
d"=>"y2LRPGQBGV7OARY3Aw30", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [accidents-2012] as the f
inal mapping would have more than 1 type: [doc, accident]"}}}}

```

refer to the link relating the issue

> [@Login is currently disabled. Administrators should consult the Kibana logs for more details.\[still unsatisfactory\]](https://discuss.elastic.co/t/login-is-currently-disabled-administrators-should-consult-the-kibana-logs-for-more-details-still-unsatisfactory/136218/4):
>
> problem still remained both are 6.3.0 and I have also auto-generated credentials using elasticsearch\_password.bat and reflected on kibana.yml correctly {elastic:'PASSWORD'} so Kibana can connect to elasticsearch. problem is logstash is not working properly as I posted. My machine is Windows and I installed using .msi so I don't directly can see elasticsearch.yml on my machine. Elastic log [2018-06-26T11:39:20,863][INFO][o.e.n.Node] [DESKTOP-C93E9FL] initializing ... [2018-06-…

---

<div class="post-metadata">

**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\
**Post date:** [June 26, 2018, 7:15pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/2 "2018-06-26T19:15:14Z")

</div>

Indices created in 6.x are only allowed to have one document type per index. Your logstash configuration is trying to create two types: `doc` and `accident`. You need to change it so there is only one type for both data sources, or put each data source in their own index.

More details here: [https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html)

---

<div class="post-metadata">

**Author:** ![Yong\_Rhee](https://avatars.discourse-cdn.com/v4/letter/y/aeb1de/32.png) [@Yong\_Rhee](https://discuss.elastic.co/u/Yong_Rhee)\
**Post date:** [June 27, 2018, 12:30pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/3 "2018-06-27T12:30:27Z")

</div>

I do not think my data has two types.

Could you kindly check my filter and input output method?  
First one is my data and the second is json type of data  
Third picture is my input method and the last is filter.

 ![IMG_20180627_074120](https://us1.discourse-cdn.com/elastic/original/3X/a/7/a7809e14cbf911802d1cdec72d0f227ff837438f.jpg) ![IMG_20180627_074105](https://us1.discourse-cdn.com/elastic/original/3X/c/6/c6e0c85c7d7d0340167e47f295dad37f87a92cec.jpg)  
 ![IMG_20180627_074038](https://us1.discourse-cdn.com/elastic/original/3X/3/5/359563ed117864b729b6b15286e989ea6ba7cb22.jpg) ![IMG_20180627_074053](https://us1.discourse-cdn.com/elastic/original/3X/2/b/2b4ed5b45122d239ae2698d21eedb0a3588fbd27.jpg)

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 27, 2018, 1:17pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/4 "2018-06-27T13:17:17Z")

</div>

Did you print it then took a picture? Unbelievable!

Please don't post images of text as they are hardly readable and not searchable.

Instead paste the text and format it with `</>` icon. Check the preview window.

---

<div class="post-metadata">

**Author:** ![Yong\_Rhee](https://avatars.discourse-cdn.com/v4/letter/y/aeb1de/32.png) [@Yong\_Rhee](https://discuss.elastic.co/u/Yong_Rhee)\
**Post date:** [June 27, 2018, 2:32pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/5 "2018-06-27T14:32:42Z")

</div>

Hello, here is a code.  
Actually, that was the book Learning Kibana 5.0 from Bahaaldine Azami from Elastic Search.  
Maybe different version (5.0 for book and 6.3 for my local machine) prevent me from following the example. However, help me do the right job for 6.3 with this example. Thank you!!

> 20/04/2012 16:05,20/04/2012,16:05,75,111,"172, RUE DE LA  
> ROQUETTE",,1\_75111\_10314,,"172, RUE DE LA ROQUETTE, 75011 Paris",RUE  
> MERLIN,Motor Scooter,RESPONSIBLE,Car,RUN  
> AWAY,,,Cond,Injured,RESPONSIBLE,,,,,,,,,,"172, RUE DE LA ROQUETTE, 75011  
> Paris",48.8591106,2.3862735,spring,2,afternoon

```
 {
 "Address": "172, RUE DE LA ROQUETTE",
 "Zip code": null,
 "Dept": "75",
 "Person 2 Tag": null,
 "Segment": null,
 "Corner": "1_75111_10314",
 "Person 1 Category": "Cond",
 "involvedCount": "2",
 "Person 4 Cat": null,
 "season": "spring",
 "periodOfDay": "afternoon",
 "Person 3 Tag": null,
 "timestamp": "20/04/2012 16:05",
 "Com": "111",
 "Person 2 Category": null,
 "Person Tag": "RESPONSIBLE",
 "Vehicle 2 Description": "Car",
 "Hour": "16:05",
 "Vehicle 3 Description": null,
 "Person 3 Cat": null,
 "Address2": "RUE MERLIN",
 "Address1": "172, RUE DE LA ROQUETTE, 75011 Paris",
 "Person 4 Tag": null,
 "Date": "20/04/2012",
"Vehicle 2": "RUN AWAY",
 "Vehicle 3": null,
 "Vehicle 1": "RESPONSIBLE",
 "Vehicle 1 description": "Motor Scooter",
 "fullAddress": "172, RUE DE LA ROQUETTE, 75011 Paris",
 "Person 2 Status": null,
 "location": {
 "lon": "2.3862735",
 "lat": "48.8591106"
 },
 "Person 4 Status": null,
 "Person 1 Status": "Injured",
 "Person 3 Status": null
 }

```

#input method

```
input {
 file {
 path => "/path/to/accidents/files/directory/accident*"
 type => "accident"
 start_position => "beginning"
 }
 }

```

#output method

```
output {
 elasticsearch {
 action => "index"
 hosts => "localhost:9200"
 index => "accidents-%{+YYYY}"
 user => "elastic"
 password => "changeme"
 template => "/path_to_template/template.json"
 template_overwrite => true
 }
 }

```

finally, filter

```
filter {
 csv {
 separator => ","
 columns => ["timestamp","Date","Hour","Dept","Com","Address","Zip
code","Corner","Segment","Address1","Address2","Vehicle 1
description","Vehicle 1","Vehicle 2 Description","Vehicle 2","Vehicle 3
Description","Vehicle 3","Person 1 Category","Person 1 Status","Person
Tag","Person 2 Category","Person 2 Status","Person 2 Tag","Person 3
Cat","Person 3 Status","Person 3 Tag","Person 4 Cat","Person 4
Status","Person 4
Tag","fullAddress","latitude","longitude","season","involvedCount","periodO
fDay"]
 }
 if ([Corner] == "Corner") {
 drop { }
 }
 date {
 match => ["timestamp", "dd/MM/YYYY HH:mm"]
 target => "@timestamp"
 locale => "fr"
 timezone => "Europe/Paris"
 }
 mutate {
 convert => ["latitude", "float"]
 convert => ["longitude","float"]
 rename => ["longitude", "[location][lon]", "latitude",
"[location][lat]" ]
 }
 }

```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 27, 2018, 6:49pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/6 "2018-06-27T18:49:50Z")

</div>

Please format your code, logs or configuration files using `</>` icon as explained in [this guide](https://discuss.elastic.co/t/about-the-elasticsearch-category/21) and **not the citation button**. It will make your post more readable.

---

<div class="post-metadata">

**Author:** ![Yong\_Rhee](https://avatars.discourse-cdn.com/v4/letter/y/aeb1de/32.png) [@Yong\_Rhee](https://discuss.elastic.co/u/Yong_Rhee)\
**Post date:** [June 27, 2018, 7:19pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/7 "2018-06-27T19:19:59Z")

</div>

thank you.

Let me know anything you need more information to get topic solved.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 27, 2018, 7:31pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/8 "2018-06-27T19:31:46Z")

</div>

Great. Would be even more readable if you correctly indent your code. The preview window is also helpful.

---

<div class="post-metadata">

**Author:** ![Yong\_Rhee](https://avatars.discourse-cdn.com/v4/letter/y/aeb1de/32.png) [@Yong\_Rhee](https://discuss.elastic.co/u/Yong_Rhee)\
**Post date:** [June 27, 2018, 8:59pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/9 "2018-06-27T20:59:31Z")

</div>

tidy up. Could you see the problem here regarding type issue?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 28, 2018, 1:27am UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/10 "2018-06-28T01:27:48Z")

</div>

You need to change the doc type in elasticsearch output. See [https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document\_type](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_type)

I'm moving your post to #logstash

---

<div class="post-metadata">

**Author:** ![Yong\_Rhee](https://avatars.discourse-cdn.com/v4/letter/y/aeb1de/32.png) [@Yong\_Rhee](https://discuss.elastic.co/u/Yong_Rhee)\
**Post date:** [July 12, 2018, 10:14pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/11 "2018-07-12T22:14:47Z")

</div>

Hello,  
I deleted type for Kibana 6.0 and tried again.  
Now I have this result and still not able to import data from the csv file.  
Could you help me?

```
input {
 file {
 path => "/path/to/accidents/files/directory/accident*"

```

` **Deleted this row type => "accident"** `

```
   start_position => "beginning"
     }
     }

```

And this is now a new log

```
 C:\Program Files\logstash-6.3.0\bin>logstash -f csv_to_es.conf
    Sending Logstash's logs to C:/Program Files/logstash-6.3.0/logs which is now configured via log4j2.properties
    [2018-07-12T18:10:46,610][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
    [2018-07-12T18:10:47,109][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.3.0"}
    [2018-07-12T18:10:50,689][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
    [2018-07-12T18:10:51,067][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@localhost:9200/]}}
    [2018-07-12T18:10:51,085][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
    [2018-07-12T18:10:51,301][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@localhost:9200/"}
    [2018-07-12T18:10:51,363][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
    [2018-07-12T18:10:51,367][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
    [2018-07-12T18:10:51,379][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/Program Files/logstash-6.3.0/config/template.json"}
    [2018-07-12T18:10:51,393][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"accident*", "mappings"=>{"accident"=>{"properties"=>{"location"=>{"type"=>"geo_point"}, "involvedCount"=>{"type"=>"double"}}}}}}
    [2018-07-12T18:10:51,423][INFO][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
    [2018-07-12T18:10:51,496][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
    [2018-07-12T18:10:52,674][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x31ac7ad0 run>"}
    [2018-07-12T18:10:52,750][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
    [2018-07-12T18:10:53,112][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}

```

And, This is a Dev tool command in Kibana.  
Other settings are same as prevous thread. Thank you

```
GET accident*/_count
{
  "count": 0,
  "_shards": {
    "total": 0,
    "successful": 0,
    "skipped": 0,
    "failed": 0
  }
}
```

---

<div class="post-metadata">

**Author:** ![Alex\_Mendez](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Alex\_Mendez](https://discuss.elastic.co/u/Alex_Mendez)\
**Post date:** [August 1, 2018, 8:55pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/12 "2018-08-01T20:55:13Z")

</div>

@Yong_Rhee, the other "type" is being set in the template

template =\> "/path\_to\_template/template.json"

I was getting the same error in the logs

```
{
  "template" : "accident*",
  "mappings" : {
    "accident": {
      "properties": {
        "location": { "type": "geo_point" }, < ---------  
        "involvedCount": { "type": "double" } < ---------
      }
    }
  }
} 

```

comment them out and try to import the data again

delete the indices and try again,

output after

```
GET accident*/_count 

{
  "count": 13629, < ---- 
  "_shards": {
    "total": 10,
    "successful": 10,
    "skipped": 0,
    "failed": 0
  }
}

```

I am on the latest 6.x version, good luck with the tutorial

---

<div class="post-metadata">

**Author:** ![Young\_Hoon\_Jang](https://avatars.discourse-cdn.com/v4/letter/y/b77776/32.png) [@Young\_Hoon\_Jang](https://discuss.elastic.co/u/Young_Hoon_Jang)\
**Post date:** [August 21, 2018, 6:49am UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/13 "2018-08-21T06:49:56Z")

</div>

Please refer to '[https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html)'

```
{
  "index_patterns" : "accident*",
  "mappings" : {
    "doc": { <------- rename a 'accident' type to 'doc' 
      "properties" : {
        "location": { "type": "geo_point" },
        "involvedCount": { "type": "double" }
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![Yong\_Rhee](https://avatars.discourse-cdn.com/v4/letter/y/aeb1de/32.png) [@Yong\_Rhee](https://discuss.elastic.co/u/Yong_Rhee)\
**Post date:** [August 21, 2018, 2:04pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/14 "2018-08-21T14:04:53Z")

</div>

Thank you. Very helpful!!!

---

<div class="post-metadata">

**Author:** ![Alex\_Mendez](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Alex\_Mendez](https://discuss.elastic.co/u/Alex_Mendez)\
**Post date:** [August 24, 2018, 7:17pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/15 "2018-08-24T19:17:05Z")

</div>

> [@Young\_Hoon\_Jang](#):
>
> doc

than you too

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 21, 2018, 7:17pm UTC](https://discuss.elastic.co/t/data-can-not-be-indexed/137470/16 "2018-09-21T19:17:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.