Filebeat not loading one index in Kibana Dataview

Hello I have the following Index and pipeline created from a csv

{
  "mappings": {
    "_meta": {
      "created_by": "file-data-visualizer"
    },
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "DESCRIPCION": {
        "type": "text"
      },
      "FAMILIA": {
        "type": "keyword"
      },
      "MARCA": {
        "type": "keyword"
      },
      "REFERENCIA": {
        "type": "keyword"
      },
      "STOCK": {
        "type": "long"
      },
      "STOCK MINIMO": {
        "type": "keyword"
      }
    }
  }
}

[
  {
    "csv": {
      "field": "message",
      "target_fields": [
        "REFERENCIA",
        "FAMILIA",
        "DESCRIPCION",
        "MARCA",
        "STOCK",
        "STOCK MINIMO"
      ],
      "separator": ";",
      "ignore_missing": false
    }
  },
  {
    "convert": {
      "field": "STOCK",
      "type": "long",
      "ignore_missing": true
    }
  },
  {
    "remove": {
      "field": "message"
    }
  }
]

I have the following filebeat.yml that I execute and the rest of the indexes and pipelines work perfectly fine, but the one mentioned before does not load any data


filebeat.inputs:
- type: log
  paths:
  - C:\AA_CopiaNAS_to_Local\ELASTIC\AVD\*.csv
  exclude_lines: ['^"?CorrelationId"?,"?UserName"?,"?SessionHostName"?,"?StartTime_String"?,"?EndTime_String"?']
  tags: ["avd"]
- type: log
  paths:
  - C:\AA_CopiaNAS_to_Local\ELASTIC\VDI\*.csv
  exclude_lines: ['^"?SessionKey"?,"?Session"?,"?Machine"?,"?StartDate"?,"?EndDate"?']
  tags: ["vdi"] 
- type: log
  paths:
  - C:\AA_CopiaNAS_to_Local\ELASTIC\Aprovisionamiento\Almacen\*.csv
  exclude_lines: ['^"?REFERENCIA"?;"?FAMILIA"?;"?DESCRIPCION"?;"?MARCA"?;"?STOCK"?;"?STOCK MINIMO"?']
  tags: ["aprovalmacen"]
- type: log
  paths:
  - C:\AA_CopiaNAS_to_Local\ELASTIC\Aprovisionamiento\Almacen_ICP\*.csv
  exclude_lines: ['^"?REFERENCIA"?;"?FAMILIA"?;"?DESCRIPCION"?;"?MARCA"?;"?STOCK"?;"?STOCK MINIMO"?']
  tags: ["aprovalmacenicp"]
- type: log
  paths:
  - C:\AA_CopiaNAS_to_Local\ELASTIC\Aprovisionamiento\Envios\*.csv
  exclude_lines: ['^"?Fecha envio"?;"?INCIDENCIA/ PETICION"?;"?Stock sacado de:"?;"?SOLICITUD"?;"?Entidad"?;"?CECO"?;"?UDS"?;"?MODELO/ DESCRIPCION"?;"?FAMILIA"?;"?USUARIO FINAL"?;"?Enviar a:"?;"?PROVINCIA"?;"?LOCALIDAD"?;"?CP"?;"?DIRECCION"?;"?PERSONA DE CONTACTO"?;"?TELEFONO"?;"?Compra"?']
  tags: ["aprovenvios"]
- type: log
  paths:
  - C:\AA_CopiaNAS_to_Local\ELASTIC\Aprovisionamiento\Cestas_ARIBA\*.csv
  exclude_lines: ['^"?\[PC\] ID de pedido"?;"?\[SOCC\]Contrato \(ID de contrato\)"?;"?\[SOCC\] ID de solicitud de compra"?;"?\[SOCC\] Título de la solicitud de compra"?;"?\[PC\]Proveedor \(Proveedor común\)"?;"?\[PC\]Código de la empresa \(Código de la empresa\)"?;"?\[PC\] Número de línea del pedido de compra"?;"?\[PC\] Descripción"?;"?\[PC\]Mercancía \(ID de la mercancía\)"?;"?\[PC\]Mercancía \(Mercancía\)"?;"?\[PC\]Pieza \(Nombre de la pieza\)"?;"?\[PC\] Es un artículo de PunchOut"?;"?\[PC\]Centro de costes \(ID de centro de costes\)"?;"?\[SOCC\]Centro de costes \(Centro de costes\)"?;"?\[PC\]Unidad de medida \(Unidad de medida\)"?;"?\[PC\] Estado del pedido de compra"?;"?\[PC\]Fecha de pedido \(Fecha\)"?;"?\[PC\]Solicitante \(Usuario\)"?;"?\[SOCC\]Preparador \(Usuario\)"?;"?\[SOCC\] Entregar a"?;"?\[SOCC\]Pieza \(Número de pieza del fabricante\)"?;"?sum\(if\(Gasto de PO<=0\.0,0\.0,Cantidad de PO<=0\.0,0\.0,Gasto de PO\)\)"?;"?sum\(if\(Cantidad de PO<=0\.0,0\.0,Cantidad de PO\)\)"?.*']
  tags: ["aprovcestasariba"]
- type: log
  paths:
  - C:\AA_CopiaNAS_to_Local\ELASTIC\Aprovisionamiento\Facturacion_ARIBA\*.csv
  exclude_lines: ['^"?Doc\.compr\."?;"?Solicitante"?;"?Número de documento de documen"?;"?Soc\."?;"?Fecha doc\."?;"?Proveedor"?;"?Nombre 1"?;"? Valor neto de pedido "?;"? Importe facturado "?;"?FaF \(Indicador de Factura final\) - X"?;"?B \(Indicador de borrado en el doc\) - L \(borrado\) y S \(bloqueada posición\)"?;"?Texto breve"?;"?Código Proyecto Clarity"?;"?Código Proyecto Clarity2"?;"?Cta\.mayor"?;"?Ce\.coste"?;"?UMP"?;"?Grupo art\."?;"?II"?;"?Ctd\.pedido"?;"?Activo fijo"?;"?SNº"?;"?Fe\.entrega"?;"?Fecha Fin Contrato"?;"?Cantidad entregada"?;"?Número de posición de document"?;"?%"?;"?Código de proyecto"?;"?Nº ident\.fis\.1"?;"?Cantidad"?;"?Nombre Sociedad"?']
  tags: ["aprovfactariba"]

cloud.id: "id"
cloud.auth: "pass"

output.elasticsearch:
  indices:
  - index: "avd.prod"
    when.contains:
      tags: "avd"
  - index: "vdi.prod"
    when.contains:
      tags: "vdi"
  - index: "aprov.almacen.prod"
    when.contains:
      tags: "aprovalmacen"
  - index: "aprov.almacenicp.prod"
    when.contains:
      tags: "aprovalmacenicp"
  - index: "aprov.envios.prod"
    when.contains:
      tags: "aprovenvios"
  - index: "aprov.cestasariba.prod"
    when.contains:
      tags: "aprovcestasariba"
  - index: "aprov.factariba.prod"
    when.contains:
      tags: "aprovfactariba"

  pipelines:
  - pipeline: "avd.prod-pipeline"
    when.contains:
      tags: "avd"
  - pipeline: "vdi.prod-pipeline"
    when.contains:
      tags: "vdi"
  - pipeline: "aprov.almacen.prod-pipeline"
    when.contains:
      tags: "aprovalmacen"
  - pipeline: "aprov.almacenicp.prod-pipeline"
    when.contains:
      tags: "aprovalmacenicp"
  - pipeline: "aprov.envios.prod-pipeline"
    when.contains:
      tags: "aprovenvios"
  - pipeline: "aprov.cestasariba.prod-pipeline"
    when.contains:
      tags: "aprovcestasariba"
  - pipeline: "aprov.factariba.prod-pipeline"
    when.contains:
      tags: "aprovfactariba"

setup:
  template.enabled: false
  ilm.enabled: false

Hi @FileFile Welcome to the community.

Apologies but I don't know which one was mentioned so can you just be direct which index is not loading?

And whichever they're one there is is their data. Is there new data?

Did you look in the logs for any errors?

The one with the tag "aprovalmacenicp"
I have created the index, pipeline and dataview with the upload a file option.
I have tried to load new data but it does not.
I don't see any error in the log

Have you tried debugging it by just taking out all the other ones and all the conditions and see that it actually works on its own?

That's what I would probably do

There might be a bug with the ingest pipeline or something else

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.