Import Data via Machine Learning - Data not sent

[aducSG]

Hello,
I have been using the experimental feature to import Data from a CSV file (in Machine Learning > Import Data).
The import is working perfectly fine, all the documents are imported in a new index.

However, when I tried to send new documents through my usual way, it's not working for the new index (even for other new ones, created the same way). But if I use the original index, it still works.

Is the issue linked to the way I created the index ? or am I missing some configurations ?
FYI, I am using ajax call to send data to Elasticsearch, calling the url directly: 'http://servername/els/Index/_doc/'

[droberts195 (David Roberts)]

when I tried to send new documents through my usual way

Are these new documents in CSV format? If they are then it makes a big difference which version of Elasticsearch you are using, as we completely changed the way the ML file import handles CSV in 7.7.

it's not working

What error message do you get?

Please give an example of the document you're trying to add to the index, and also the mappings of the index you're trying to add to.

[aducSG]

Hello droberts195,
Thank you for answering.

I am in version 7.4 for both Kibana and Elastisearch.
The new documents sent are not in CSV format; they are sent using the regular way of monitoring data, that I used before to implement the index (ajax call).

I don't have a lot of details in the error message:

{"ts":"15:48:10.819","ctx":"Action","action":"ctx.log","params":[" ctx.ajax.call  error: ",0],"verb":"LOGMESS","parentId":279}
	ts: '15:48:10.819'
	ctx: 'Action'
	action: 'ctx.log'
	params: 
		[0]: ' ctx.ajax.call  error: '
		[1]: 0
	verb: 'LOGMESS'
	parentId: 279

And here is an example of document:

     {
        "_index" : "import",
        "_type" : "_doc",
        "_id" : "yqjRgHMBHmE512Ofb9cj",
        "_score" : 1.0,
        "_source" : {
          "RPA" : "PSC_IMPAYES",
          "dateStart" : "2020-02-06 15:08:44.354",
          "dateStop" : "2020-02-06 15:10:36.762",
          "duration" : "112",
          "error" : "Step GLOBAL.scStep1_CollectDataScenario_Contact.stContactNavigateToConnaissanceClientInterlocuteur: Error, CONTACT.pConnaissanceInterlocuteur.execScript: Argument ou appel de procédure incorrect",
          "scenario" : "Etape 1 : scStep1_CollectDataScenario",
          "stopAuto" : "YES",
          "user" : "A242135",
          "version" : "1.0.1",
          "volume" : 1
        }

For the mapping:

{
  "mapping": {
    "_meta": {
      "created_by": "ml-file-data-visualizer"
    },
    "properties": {
      "RPA": {
        "type": "keyword"
      },
      "anomalie": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "checkpoint": {
        "type": "keyword"
      },
      "dateStart": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss.SS||yyyy-MM-dd HH:mm:ss.S"
      },
      "dateStop": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss.SS||yyyy-MM-dd HH:mm:ss.S"
      },
      "duration": {
        "type": "keyword"
      },
      "error": {
        "type": "text"
      },
      "scenario": {
        "type": "text"
      },
      "script_source": {
        "type": "text"
      },
      "stopAuto": {
        "type": "keyword"
      },
      "stopUser": {
        "type": "keyword"
      },
      "timeout": {
        "type": "keyword"
      },
      "user": {
        "type": "keyword"
      },
      "version": {
        "type": "keyword"
      },
      "volume": {
        "type": "long"
      }
    }
  }
}

[aducSG]

Could anyone help me please ?

[droberts195 (David Roberts)]

I cannot see anything wrong with the mappings.

I think the next step is to try indexing one of your new documents to the index that was created by the file uploader using curl instead of your ajax framework. For example:

curl -X POST "https://<es-host>:9200/import/_doc?pretty" -H 'Content-Type: application/json' -d '
{
          "RPA" : "PSC_IMPAYES",
          "dateStart" : "2020-02-06 15:08:44.354",
          "dateStop" : "2020-02-06 15:10:36.762",
          "duration" : "112",
          "error" : "Step GLOBAL.scStep1_CollectDataScenario_Contact.stContactNavigateToConnaissanceClientInterlocuteur: Error, CONTACT.pConnaissanceInterlocuteur.execScript: Argument ou appel de procédure incorrect",
          "scenario" : "Etape 1 : scStep1_CollectDataScenario",
          "stopAuto" : "YES",
          "user" : "A242135",
          "version" : "1.0.1",
          "volume" : 1
}
'

You'll need a -u username:password in the arguments too if you have security enabled.

This will show the underlying error from Elasticsearch.

[aducSG]

Thank you for your answer, I will try that then.