Logstash parsing input filter beat csv file

Elastic Stack Logstash
1

Hi everoene,

I getting an issue between filebeat and logstash that I can not uderstand what is gonig on, I need some help with that.
Let me tell you what is going on...
The filebeat setting is the following....

//
filebeat.inputs:

- type: log
  enabled: true
  tags:
    - Backoffice
  paths:
      - /home/backoffice.csv
 - type: log
  enabled: true
  tags:
    - Vendas
  paths:
      - /home/vendas.csv
 - type: log
  enabled: true
  tags:
    - Final
  paths:
      - /home/final.csv

and the logstash.conf is the following ...

input {
  beats {
    port => 5044
    #codec => palin { charset => "ISO-8859-1"}
    #workers => 3
   # queue_size => 72000
    #receive_buffer_bytes => 31457280
  }
}

filter {
    if "Backoffice" in [tags]{
     csv {
      separator => ","
      columns => ["fases","tipo","quantidade","valor","event1","event2"]
       }
      mutate {
         convert => {
         #"epoch_timestamplisted_action" => "integer"
         #"uniqueid" => "string"
         #"queue_name" => "integer"
         #"bridged_channel" => "string"
         #"event" => "string"
         "quantidade" => "integer"
         "valor" => "integer"
         "event1" => "integer"
         }
      }
    }
}
filter {
    if "Vendas" in [tags]{
     csv {
       separator => ","
       skip_empty_columns => true
       columns => ["nome","unidade","tipo","quantidade","valor","event3","event4"]
         }
       mutate {
          convert => {
          #"epoch_timestamplisted_action" => "integer"
          #"uniqueid" => "string"
          #"queue_name" => "integer"
          #"bridged_channel" => "string"
          #"event" => "string"
          "quantidade" => "integer"
          "valor" => "integer"
          "event3" => "integer"
         }
       }
    }
}
filter {
    if "Final" in [tags]{
     csv {
      separator => ","
      columns => ["fases","tipo","quantidade","valor","event5","data"]
      }
      mutate  {
         convert => {
         #"epoch_timestamplisted_action" => "integer"
         #"uniqueid" => "string"
         #"queue_name" => "integer"
         #"bridged_channel" => "string"
         #"event" => "string"
         "quantidade" => "integer"
         "valor" => "integer"
         "event5" => "integer"
       }
     }
  }
}
output {
  if "Backoffice" in [tags]{
     elasticsearch {
     hosts => "localhost:9200"
     manage_template => false
     index => "%{[@metadata][beat]}-backoffice"
     #document_type => "%{[@metadata][type]}"
     }
   }
}
output {
  if "Vendas" in [tags]{
     elasticsearch {
     hosts => "localhost:9200"
     manage_template => false
     index => "%{[@metadata][beat]}-vendas"
     #document_type => "%{[@metadata][type]}"
     }
   }
}
output {
  if "Final" in [tags]{
     elasticsearch {
     hosts => "localhost:9200"
     manage_template => false
     index => "%{[@metadata][beat]}-final"
     }
  }
}
 # for debug purpose of pipeline with command: ./logstash -f /etc/logstash/conf.d/logstash.conf
//

also the pipeline.yml is setting as the following ....

//
- pipeline.id: main
  pipeline.workers: 8
  pipeline.batch.size: 1000
  pipeline.batch.delay: 120
  path.config: "/etc/logstash/conf.d/*.conf"
  #codec: plain { charset => "ISO-8859-1" }
  # workers: 3
  #queue_size: 72000
  #receive_buffer_bytes: 31457280
  queue.type: persisted
//

Here is the issue the first content backoffice file there is no issue ...every content comes every 15 minutes fine ...but the second csv file "vendas.csv" tha comes every 1 hour has te following situation...

 ,*
11:40:04.423  Vanderleia_Souza,Unidade-1_1100,Portabilidade,1,37066,0,*
11:40:04.423 Rebeca_Xavier,Unidade-1_2100,Portabilidade,1,11706,0,*
11:40:04.423 Rayssa_Lima_Fernandes_de_Souza,Unidade-1_2100,Novos,1,4791,0,*
11:40:04.423 Rayssa_Lima_Fernandes_de_Souza,Unidade-1_2100,Portabilidade,1,9052,0,*
11:40:04.423 Victoria_Christina_Batista�,Unidade-1_2100,Novos,1,10071,0,*
11:40:04.424 Vitoria_Eberhardt_Machado_dos_Passos,Unidade-2_6100,Novos,1,2738,0,*
11:40:04.424 Vitoria_Eberhardt_Machado_dos_Passos,Unidade-2_6100,Novos,1,2965,0,*
11:40:04.424 Orestes_Novaes,Unidade-2_6100,Portabilidade,1,12724,0,*
11:40:04.424 Izadora_Elizabeth_Gama_dos_Santos,Unidade-2_5100,Portabilidade,1,13052,0,*
11:40:04.424 Lucas_Catania_Marques_De_Oliveira,Unidade-3_7100,Novos,1,9494,0,*
11:40:04.424 Lucas_Catania_Marques_De_Oliveira,Unidade-3_7100,Novos,1,9790,0,*
11:40:04.424 Lucas_Catania_Marques_De_Oliveira,Unidade-3_7100,Cartao,1,1822,0,*
11:40:04.425 Gabrielly_de_Lima_Barbosa,Unidade-3_7100,Novos,1,6662,0,*
11:40:04.425 Igor_De_Andrade_B._Mathias,Unidade-3_7100,Novos,1,11490,0,*
11:40:04.425 Igor_De_Andrade_B._Mathias,Unidade-3_7100,Novos,1,11478,0,*
11:40:04.425 Igor_De_Andrade_B._Mathias,Unidade-3_7100,Cartao,1,2139,0,*

before de first line must be the following ...

11:47:34.439  Quesia_Farias_da_Silva,Unidade-2_3100,Portabilidade,1,11946,0,*
11:47:34.439 Andre_Tavares_do_Nascimento,Unidade-2_3100,Portabilidade,1,11994,0,*
11:47:34.439 Tayna_Donofrio_Barbosa,Unidade-2_3100,Novos,1,735,0,*
11:47:34.439 Manoela_Lopes,Unidade-1_1100,Portabilidade,1,34839,0,*
11:47:34.440 Thaillyn_Tamires_da_Silva,Unidade-1_1100,Portabilidade,1,12594,0,*
11:47:34.440 Nayara_Brandao,Unidade-1_1100,Portabilidade,1,13787,0,*
11:47:34.440 Nayara_Brandao,Unidade-1_1100,Portabilidade,1,8274,0,*
11:47:34.440 Vanderleia_Souza,Unidade-1_1100,Portabilidade,1,14143,0,*

I don´t know why the data is beeing truncate...

IF i go to the same original file at the filebeat host and edit it and leave just the data that it not went to logstash host ..after save the file evey contemt goes to logstash with no issue...

So anyeone already have this situation ? I need help to solve that

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.