Can't write data to elasticsearch (cannot be changed from type [date] to [text)

Elastic Stack Elasticsearch
1

Hi
can't write data to elasticsearch vi logstash(http_poller) here is the scenario:

influxdb > logstash(http_poller) > elasticsearch

error that I get:

"status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [results.series.values] cannot be changed from type [date] to [text]"}}}

logstash config:

input {
http_poller {
urls => {
test1 => {
method => get
url => "http://192.168.1.1:8086/query?pretty=true&q=SELECT+*+FROM+cpu%20WHERE%20time%20>%20now()%20-%201m&db=monitorindb"
headers => {
Authorization => "Token "
}
}
}
schedule => { cron => "* * * * * UTC"}
request_timeout => 60
codec => "json"
metadata_target => "http_poller_metadata"

}
}

output {
elasticsearch {
hosts => "localhost:9200"
index => "http_poller"
user => "elastic"
password => "PASS"

}
}

2

Check your mapping and or your data. results.series.values can not be parsed as a date.

3

@dadoonet where should i check it?

4

I believe I wrote this:

Check your mapping

So I'd do something like:

GET /http_poller/_mapping
5

@dadoonet return below result, what should i do next?

GET /http_poller/_mapping

{
"http_poller": {
"mappings": {}
}
}

http_poller
http_poller1034×644 35.8 KB

6

I'm surprised by the mapping... It's not aligned with the error message.
So I believe you did something in the middle, like removing the index....

7

@dadoonet yes i did that, because add this for test:

filter {
mutate {
remove_field => ["date"]
}
}

for sure remove index, and it will recreated automatically but issue still remain.
Any idea?

8

If that is the format you are trying to index you will have problems. All values in the values array need to have the same mapping. If you are using dynamic mapping the first value envountered will define the mapping type. If you have a mixed array like you do it is possible indexing the document on its own fails due to mapping conflict so that nothing gets inserted.

I believe you will need to restructure the document into something that is actually usable in Elasticsearch.

9

@Christian_Dahlqvist someone do this,

filter {
split {
    field  => "results"
}
split {
    field  => "[results][series]"
}
split {
    field  => "[results][series][values]"
	
	
}
if[name]== "cpu"
{
	ruby 
	{

	 #event.set(event.get("[results][series][columns][0]"),event.get("[results][series][values][0])")
	  code => '
				
				 event.set(event.get("[results][series][columns][0]"),event.get("[results][series][values][0]"))
	
  '

	}
}

any idea?

10

No, I have no idea. Have never pulled data from InfluxDB. It may help if you show a sample document you are trying to index.

11

after several try and error finally fix with this:

DELETE /http_poller

PUT /http_poller
{
  "mappings": {
    "properties": {
      "results": {
        "type": "nested",
        "properties": {
          "series": {
            "type": "nested",
            "properties": {
              "values": {
                "type": "text"
              }
            }
          }
        }
      }
    }
  }
}
12

You may want to map it as keyword instead of text and that will not analyse it. If you are indexing it exactly as shown in the screenshot above I think you will have issues querying it, so I would recommend restructuring the document if that is the case so that it contains key-value pairs with the data instead of separate arrays of keys and values.

13

@Christian_Dahlqvist here is the json file:

14

You may be able to index that document into Elasticsearch but I do not see how you will be able to use it for anything useful in that format. I would split this large document into multiple documents and transform the data arrays to neat key-value pairs. That would allow you to search and use the data in Kibana, which the current structure will not.

You may also want to assign an ID to reduce the risk of duplicates, but that depends on the data the http poller will be returning.

15

@Christian_Dahlqvist would you please give me a sample config to extract key/values from json that i post?

e.g: extract key/values from this part: time,cpu,host,usage_guest

  "series": [
      "columns": [
        "time",
        "cpu",
        "host",
        "usage_guest",
      "values": [
        "2023-04-15T12:13:00Z",
        "cpu1",
        "server1",
        "0",
16

Try something like this:

filter {
  split {
    field => "results"
  }

  split {
    field => "[results][series]"
  }

  split {
    field => "[results][series][values]"
  }

  mutate {
    rename => { "[results][series]" => "series" }
  }

  mutate {
    rename => { "[series][name]" => "name" }
  }

  ruby { 
    code => 'series = event.get("series"); series["columns"].each_with_index {|val, index| event.set(val, event.get("[series][values][" + index.to_s() + "]"))}'
  }

  prune {
    blacklist_names => [ "event", "host", "series" ]
  }
}
1 Like
17

Why you put these on blacklist? These keywords use by elastic?
"event", "host", "series"

18

It removes those fields from the event. You can keep them if you want to...

19

@Christian_Dahlqvist I mean these are reserved for elsticsearch? Or just put it for example?

20

The event field field contains the original event string, which I think is no longer needed. The series field contains the raw arrays that have been parsed out, so is in my opinion no longer needed either. The host filed might possibly be useful, so keep that if you need it.

Deleting these fields just cleans up the event.

1 Like