.NET setting field as Date

[Neal_Walters]

I'm trying to define the field createdDate below as a datetime, using .NET:

Here's my existing code; had trouble finding any examples for the .NET classes.
Maybe my date format needs to have the milliseconds on it?

                 CreateIndexRequest request = new CreateIndexRequest(index.ToString().ToLower());
                 TypeMapping typeMapping = new TypeMapping();
                 typeMapping.DateDetection = false;
                 request.Mappings = new Mappings();
                 request.Mappings.Add("object", typeMapping);

                 TypeMapping dateTypeMapping = new TypeMapping();
                 dateTypeMapping.DateDetection = true; 
                 string[] formats = new string[1]; 
                 formats[0] = "YYYY-MM-DD HH:MM:SS"; 
                 dateTypeMapping.DynamicDateFormats = formats; 
                 request.Mappings.Add("createdDate", dateTypeMapping );
                 shared.CreateIndex(request);

If I used the field typeMapping instead of the newDateTypeMapping that I added, it was being stored as "t" (text) and was not sortable in Kibana.

I tried to simpify, added a new "sortDate" with an easier date/time.

                 CreateIndexRequest request = new CreateIndexRequest(index.ToString().ToLower());
                 TypeMapping typeMapping = new TypeMapping();
                 typeMapping.DateDetection = false;
                 request.Mappings = new Mappings();
                 request.Mappings.Add("object", typeMapping);

                 TypeMapping dateTypeMapping = new TypeMapping();
                 dateTypeMapping.DateDetection = true; 
                 string[] formats = new string[1];
                 //2018-04-05T09:11:29.5267304-05:00 createdDate
                 //yyyy-MM-dd HH:mm:ss sortDate 
                 formats[0] = "yyyy-MM-dd HH:mm:ss"; 
                 dateTypeMapping.DynamicDateFormats = formats; 

                 request.Mappings.Add("createdDate", typeMapping );
                 request.Mappings.Add("sortDate", dateTypeMapping);
                 shared.CreateIndex(request);

It's showing up as a ? for data type.
This is how dates were set in C#:

                Document<string> document = new Document<string>
                {
                    Message = message,
                    CreatedDate = DateTime.Now,
                    SortDate = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"),
                    TransactionType = index
                };

[forloop (Russ Cam)]

With NEST, you typically represent a JSON document in Elasticsearch with a Plain Old CLR Object (POCO) in your .NET application.

Assuming you have a document that looks something like the following, based on the fields you have in Kibana

public class MyDocument
{
    public DateTime CreatedDate {get;set;}
    public string Message {get;set;}
    public string TransactionType {get;set;}
}

creating the index along with the mapping would be as follows

var client = new ElasticClient();

var createIndexResponse = client.CreateIndex("my-index", c => c
    .Mappings(m => m
        .Map<MyDocument>(mm => mm.AutoMap())
    )
);

which would generate the following request to Elasticsearch

PUT http://localhost:9200/my-index
{
  "mappings": {
    "mydocument": {
      "properties": {
        "createdDate": {
          "type": "date"
        },
        "message": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "transactionType": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

take a look at automapping for more details, and other mapping documentation to see how you can control mapping to suit your needs.

[Neal_Walters]

Thanks. Your post helped me finalize realize that I was trying to format a date that was already in Elasticsearch, and that I had the createdDate was already defined as a DateTime,

I'm trying to improve some code that I inherited. So you have one person (me) not knowing what they are doing, updating code of maybe another person that didn't know what they were doing. I'm just in experimentation mode, seeing if I can improve. No official time to work on this yet.

The Document class has a generic type. I think the reason this was done was so that Elasticsearch could handle JSON in the message.

public class Document<T>
{
    public T Message { get; set; }
    public DateTime CreatedDate { get; set; }
    public string TransactionType { get; set; }
}

I tried making it non-generic, and defining Message as string (top example using your automapping, other example is from the original program):

Basically same except the serialization was slightly different, which I can change.
Seem like once I saw Kibana show message.firstname, and message.lastname as a column name; but now I forgot which iteration that was.

Then i tried making it non-generic and defining Message as an object, but got this error:

{Type: mapper_parsing_exception Reason: "object mapping for [message] tried to parse field [message] as object, but found a concrete value"}

You can see the date as the "t" for text beside it. I'd like to keep the current code as close as possible. That's why I tried adding the dateTypeMapping.

So even with automap, should I expect to see "d" by the createdDate?

Thanks,
Neal

[Neal_Walters]

From Kibana - Dev Tools, I ran
GET local-neal-test-kibananew2c-2018.04.06
which give sthe following - showing that createdDate is actually a date.

So now back to my original quandry - why can't I sort by it?

How sort by date/time on Kibana Discover

Below is an example of what I am seeing: [Kibana_Sort_Question] I dont' see the "time" choice as shown in this thread: I'd like to sort by descending date/time ]the field createdDate (even by default if possible). It's one of the fields in every message: [Kibana_Sort_Fields] Thanks, Neal

> {
>   "local-neal-test-kibananew2c-2018.04.06": {
>     "aliases": {},
>     "mappings": {
>       "document2": {
>         "properties": {
>           "createdDate": {
>             "type": "date"
>           },
>           "message": {
>             "type": "text",
>             "fields": {
>               "keyword": {
>                 "type": "keyword",
>                 "ignore_above": 256
>               }
>             }
>           },
>           "transactionType": {
>             "type": "text",
>             "fields": {
>               "keyword": {
>                 "type": "keyword",
>                 "ignore_above": 256
>               }
>             }
>           }
>         }
>       }
>     },
>     "settings": {
>       "index": {
>         "creation_date": "1523026621059",
>         "number_of_shards": "5",
>         "number_of_replicas": "1",
>         "uuid": "3DzKTpm7TYKk_u35pj3eBA",
>         "version": {
>           "created": "6000099"
>         },
>         "provided_name": "local-neal-test-kibananew2c-2018.04.06"
>       }
>     }
>   }
> }

[Neal_Walters]

Well, "I'll be switched" (as they used to say).

If I create an index pattern on "local-neal-test-kibananew4a-", it works.
I was using an index pattern on "local-neal-" it doesn't, even though all the data is displayed. I presume that's because not all messages in local-neal-* have the exact same type.

When I use the management tab to create the index pattern: local-neal-test-kibananew4a-*
it pops up the "Time Filter field name" box":

I had never seen that before. I guess I need to read up on what index patterns really do.
I thought Kibana would be super user-friendly, but that are apparently tricks to be learned.

We have naming standards for our indexes. What I was hoping to do was watch the "trace" of my business process across different indexes and be able to get them in date/time sequence, since they all had the same createdDate field. Apparently this won't work??

Neal

[forloop (Russ Cam)]

message is a string, based on the mapping response provided. Looking at Document<T>, I guess the system was designed to handle different T, where T is some other type into which the JSON can be deserialized.

@Neal_Walters

You can see the date as the "t" for text beside it. I'd like to keep the current code as close as possible. That's why I tried adding the dateTypeMapping.

So even with automap, should I expect to see "d" by the createdDate?

You would yes, if the index pattern is created after creating the index. Note that you cannot change the existing mappings in an existing index (save for a few exceptions, such as making a text field a multi-field), so if documents have been indexed with the wrong mapping, you would need to reindex the documents again (using the reindex API, if you have the _source for each document in Elasticsearch).

I'm not sure what happens in Kibana if you have multiple indices that match an index pattern, where those indices have a field with the same name but mapped differently.

@Neal_Walters

When I use the management tab to create the index pattern: local-neal-test-kibananew4a-*

it pops up the "Time Filter field name" box":

Glad you got this part sorted

[Neal_Walters]

I'm not sure what happens in Kibana if you have multiple indices that match an index pattern, where those indices have a field with the same name but mapped differently.

createdDate is never mapped differently.
Neal