Timestamp search from file

[mihai.radulescu]

Hello,
I have no experience with Elasticsearch, however I wish to setup an environment to be able to search through some large csv files. The import with logstash works, however I have two issues:

1. timestamp scale is using the time each record was inserted to elasticsearch, while I have a field date that is holding the decoded TimeStamp for each record from the file
2. each bar is limited at 8000 records, while in some cases I have much more

My logstash configuration is below:

input {
  file {
    path => "/mnt/xxxxxxxxxxxxx/*.txt"
    start_position => "beginning"
  }
}

filter {
      csv {
        columns => [ "xx1","xx2","xx3","xx4","Responsexx","xx5","TimeStamp","Username","SourceIP" ]
     }
        date {
                        match => [ "TimeStamp", "yyyyMMddHHmmss" ]
                        target => "date"
                        locale => "en"
                }
    }

output {
  elasticsearch {
  hosts => ["localhost:9200"]
  index => "xxxx"
  }

 }

Here is an image on how it looks like:

Can you help me solve these issues?

Best Regards,
Mihai Radulescu

[dadoonet (David Pilato)]

It looks good. What's look like a typical line?

[mihai.radulescu]

Hi dadoonet,
I don't think it looks good enough. If you take a look at my example, the Time is "Dec 7, 2020 @ 14:37:26 129", which corresponds to the time the field was inserted and based on this one the search is done, while the timestamp/date from the record is 2020-12-07T11:41:02:000Z, which was obtained by converting the Timestamp field 20201207114102.

How can I have the Time as Timestamp or how do I configure the search to be done based on the field timestamp?

I'll check about the typical line and revert.

Best Regards,
Mihai Radulescu

[Badger]

If you want to copy the [date] field to [@timestamp] then you can do it using

 mutate { copy => { "date" => "@timestamp" } }

Alternatively, do not set the target option on the date filter and it will write the value to @timestamp to start with.

[mihai.radulescu]

Hi Badger,
Removing the target option worked like a charm. Thank you!