Help - No configuration found in the configured sources

[adamdharris (Adam Harris)]

Having trouble running logstash on local windows 10 instances. I have looked at the other posts but cannot figure it out.

Config File

input{
    file {
        path => "C:/Users/adamd/Data/Invoice_01062016_14032018.csv"
        start_position => "beginning"
        sincedb_path => "NUL"
    }
}
filter{
    csv{
        separator => ","
        columns["HeaderKeyId","DetailKeyId","InvId","InvDate",
            "InvDateNum","InvDateTime","InvLineId","InvXRefId","InvXRefLineId",
            "InvMonth","InvYear","InvMonthYear","InvFinPeriod","InvFinYear",
            "OrdId","OrdPrefix","OrdLineNo","OrdLineSeqNum","OrdDate","OrdDateTime",
            "ContractNum","ContractType","PurOrdId","CustOrdNum","CustRef2","CustRef3",
            "CustReqDate","CustReqDateTime","CustId","CustAXId","CustCode","CustName",
            "CustGrpCode","CustGrpName","CustStateId","CustStateDescr","CustLocId","CustLocDescr",
            "CustGrpClassId","CustGrpClassDescr","CallerId","CallerName","CallerPhone","Caller2Id",
            "Caller2Name","Caller2Phone","Caller2Email","Caller2Company","DispatchRef","ConsignRef",
            "CustDelName","CustDelInstruct","CustDelInstruct2","CustDelAddr","CustDelCity",
            "CustDelAddrState","CustDelPostCode","ReceiveDate","DispatchDate","IsDirectShip",
            "IsShipped","QtyOrd","QtyInv","ItemId","ItemDescr","SuplItemId","SuplId","SuplName",
            "ManufItemId","ManufId","ManufName","ItemClass","ItemClass1Id","ItemClass1Descr",
            "ItemClass2Id","ItemClass2Descr","ItemVendClass1Id","ItemVendClass1Descr",
            "ItemVendClass2Id","ItemVendClass2Descr","ItemVendClass3Id","ItemVendClass3Descr",
            "ItemVendClass4Id","ItemVendClass4Descr","FinCodeClassId","ItemStatus","WarehouseId",
            "WarehouseDescr","WarehouseFinCode","DealNum","POLineComment","InvExtLineCostEx",
            "InvExtLineSellEx","InvExtLineTax","InvExtLineSellInc","TaxJurisdiction","TaxPercent",
            "InvCre","RepId","RepCode","RepName","RepCostCodeId","RepCostCodeLofBId","RepCostCodeDivId",
            "RepCostCodeRegId","RepCostCodeStateId","Rep2Id","Rep2Code","Rep2Name","Rep2CostCodeId",
            "Rep2CostCodeLofBId","Rep2CostCodeDivId","Rep2CostCodeRegId","Rep2CostCodeStateId",
            "TransCostCodeId","TransCostCodeLofBId","TransCostCodeDivId","TransCostCodeRegId","TransCostCodeStateId","WorkPerformById","WorkPerformByName",
            "WorkPerformByGrpId","WorkPerformByGrpDescr","WorkPerformByNodeId","WorkPerformByNodeDescr","PSFlag","NumProcDays","OrderType","InvOrdType",
            "SaleOrderType","SaleContractType","SaleContractLine","InvIsExcluded","CustIsExcluded","ContractIsExcluded","ContractClass","InvIsConsolid",
            "DataSource","DateTimeCreated","IsAdjustment","ContractWorkPerformed","ItemSerialNumber","AssetTag","DODGUID","MACAddress","CustRef1"]
    }
    mutate {convert => ["QtyInv", "integer"]}
    mutate {convert => ["QtyOrd", "integer"]}
    mutate {convert => ["InvExtLineCostEx", "float"]}
    mutate {convert => ["InvExtLineSellEx", "float"]}
}
output{
    elasticsearch {
        hosts => "localhost:9200"
        index => "liberate"
        document_type => "invoices"
    }
    stdout{}
}

Command Line

C:\Users\adamd\Documents\Logstash>bin\logstash -f C:/Users/adamd/Data/logstash_liberateinvoices.conf
Sending Logstash logs to C:/Users/adamd/Documents/Logstash/logs which is now configured via log4j2.properties
[2018-10-15T17:04:19,903][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-10-15T17:04:20,436][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2018-10-15T17:04:20,531][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2018-10-15T17:04:20,903][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

[JKCode (JYK)]

+1

My issue is wrt the pipelines.yml

pipeline.yml config:

 pipeline.id: test
  pipeline.workers: 1
  pipeline.batch.size: 500
  queue.type: persisted
  queue.max_bytes: 1gb # Default = 1024mb
  queue.drain: true
  pipeline.output.workers: 1
  path.config: "D:/ElasticStack-6.4.2/PipelineIngestionTest-1/IngestionTest_1.conf"

Config file:

input{
    stdin{
    }
}
output{
    stdout{
        codec => "rubydebug"
        id => "-RUBY-" #Unique ID assigned to the Output Channel for Monitoring
    }
    elasticsearch{
        hosts => ["localhost:9201"]
        index => "Test-%{+dd.MM.YYYY}" # Must be lowercase. ERROR: duplicate records caused by not specifying input=>file=>_id
        id => "-ES:9201:Test-" #Unique ID assigned to the Output Channel for Monitoring
    }
}

Logstash Error in cmd:

[2018-10-17T17:08:11,288][DEBUG][logstash.runner          ] --------------- Logstash Settings -------------------
[2018-10-17T17:08:11,340][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"D:/ElasticStack-6.4.2/logstash-6.4.2/config/pipelines.yml"}
[2018-10-17T17:08:11,395][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"D:/ElasticStack-6.4.2/logstash-6.4.2/config/pipelines.yml"}
[2018-10-17T17:08:11,462][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
Configuration OK
[2018-10-17T17:08:11,473][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

Please help!

[JKCode (JYK)]

Found a solution to the error i faced on another thread which works for this situation:

1. Config File:
   Index name to be lowercase as shown below:
   index => "test-%{+dd.MM.YYYY}"

2. File path in pipeline.yml:
   Drive letter removed as shown below:
   path.config: "/ElasticStack-6.4.2/PipelineIngestionTest-1/IngestionTest_1.conf"

Hope this helps and works for you, @adamdharris!