CSV to Elasticsearch via Python - Not able to see documents in Dashboard / Discover

Elastic Stack Kibana
1

Hi everyone,

Good day!
I am trying to do ingest csv to elasticsearch and hopefully to be able to visualize it in Kibana. Index is created via UI[upload CSV]. I've used a simple Python code for the succeeding upload of documents.

I can see my documents are successful ingested via CURL, but I can't see them in Discover and Dashboard.
Can anyone help point me to the right direction on what seems to be the issue here? I've read about it is something to do with the timestamp, but I am using DATE only and customized the format of it.
Thanks in advance.

Kibana v 7.17.5

    for file in os.listdir('/RANDOMPATH/NEW/'):
        if fnmatch.fnmatch(file, 'recolog_*.csv'):
            with open(file, 'r') as infile:
                reader = csv.DictReader(infile)
                response = helpers.bulk(es, reader, index='recolog')
                print ("\nActions Response:", response)

es.indices.refresh(index="recolog")

"recolog" : {
    "aliases" : { },
    "mappings" : {
      "_meta" : {
        "created_by" : "file-data-visualizer"
      },
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "Count" : {
          "type" : "long"
        },
        "DATE" : {
          "type" : "date",
          "format" : "M/dd/yy"
        },
        "A" : {
          "type" : "keyword"
        },
        "B" : {
          "type" : "long"
        },
        "TIME" : {
          "type" : "keyword"
        },
        "C" : {
          "type" : "keyword"
        }
      }
    }

SAMPLE DATA:

A,DATE,TIME,COUNT,B,C
RECO_NONSYNCH,9/14/22,17:21:36,5,225725,RECORULES_1
RECO_MISSING,9/14/22,17:21:36,401,225725,RECORULE_2
RECO_OBSOLETE,9/14/22,17:21:36,533,225725,RECORULE_3

Cheers!

2

Welcome to our community! :smiley:

Did you create a data view for recolog?

3

Hi @warkolm ,

Thanks for this comment. This is already resolved, thanks for the insights.
Seems that Data views is the same as index pattern so I was able to do that.
The mistake I have is that I wasn't able to customize the timestamp based on my field of data.
since I created the index pattern automatically.

Thanks again!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.