# Logstash backfilling two type of logs, problem!

**URL:** <https://discuss.elastic.co/t/logstash-backfilling-two-type-of-logs-problem/40646>\
**Category:** Logstash\
**Created:** [February 1, 2016, 3:52pm UTC](https://discuss.elastic.co/t/logstash-backfilling-two-type-of-logs-problem/40646 "2016-02-01T15:52:07Z")\
**Posts on this page:** 1\
**Showing post:** 33

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [February 18, 2016, 7:33pm UTC](https://discuss.elastic.co/t/logstash-backfilling-two-type-of-logs-problem/40646/33 "2016-02-18T19:33:13Z")

</div>

5 days now ive been playing around, still havent figured out how to get it to work ...

I have 20G of logs that need to be backfilled into ELK stack ... now upgraded to 4.4 kibana 2.2 elastricsearch etc.

heres another roundup of the config setup :

I use a 10- to 49-\*.conf input setup 50-output.conf

10-.conf is working as intended. so I have copied it to 11-.conf and played around with it .. no succes

heres 11-.conf

> ```
> input {
> file {
> path => ["/var/log/fortigate/fg.log"]
> start_position => "beginning"
> sincedb_path => "/tmp/sucemamarde1"
> type => "fgt-backfill"
> }
> }
> 
> ```

> ```
> filter{
> 
> ```

> ```
> #grok {
> # match => [
> # "message",
> # "%{GREEDYDATA:kv}"
> # ]
> # remove_field => ["message"]
> #}
> 
> ```

> ```
> kv {
> source => "message"
> field_split => " "
> value_split => "="
> }
> 
> ```

> ```
> #date {
> # match => ["itime", "UNIX_MS"]
> # locale => "en"
> #}
> 
> ```

> ```
> geoip{
> source =>"dstip"
> database =>"/opt/logstash/GeoLiteCity.dat"
> }
> 
> ```

> ```
> }
> 
> ```

Tried different setup, using grok to match only %{GREEDYDATA:kv} --- according to grok debugger I should match the log !

one exemple of logs into fg.log :

`"itime=1453486381 date=2016-01-22 time=13:13:01 devid=FG200D3913801116 logid=0001000014 type=traffic subtype=local level=notice vd=root srcip=192.168.0.7 srcport=137 srcintf="port1" dstip=192.168.0.255 dstport=137 dstintf="root" sessionid=781856124 status=deny policyid=0 dstcountry="Reserved" srccountry="Reserved" trandisp=noop service=137/udp proto=17 app=137/udp duration=0 sentbyte=0 rcvdbyte=0"`

As Magnus Baeck Said, absolutely everything should be parsed with the KV filter by it self. its only a key value chain!, still it dont...

Nothing relevent is output to logstash.log .err or stdout

heres my 50-output.conf

> ```
> output {
> #if [type] == "fgt-backfill" {
> if [path] == "/var/log/fortigate/fg.log" {
> elasticsearch {
> hosts => ["localhost:9200"]
> index => "fgt-backfill-%{+YYYY.MM.dd}"
> }
> stdout { codec => rubydebug }
> }
> 
> ```

> ```
> else {
> elasticsearch {
> hosts => ["localhost:9200"]
> }
> }
> #DEBUG TOUT
> #stdout { codec => rubydebug }
> }
> 
> ```

indices fgt-backfill-\* simply doesnt get created at all ... tried with if [path] == XXXX if [type] == "fgt-backfill"  
still, a no go. the else work correctly ( meaning my other input are working as intended.

Please. help, I dont know where else, I could ask for help nor. how am i suppose to figure it out by my self...

According to the doc, the index doesnt have to be created PRIOR to logstash indexation.. so im lost.

---

_[View the full topic](https://discuss.elastic.co/t/logstash-backfilling-two-type-of-logs-problem/40646)._