XML Webservice->http_poller->elastic


(Android Games) #1

Hello All,
I'm trying to pull data from xml webservice and load into elastic to show in kibana.
Using Client Certificate to authenticate webservice . My conf file looks as below .

input {
  http_poller {
    urls => {

      test2 => {
        # Supports all options supported by ruby's Manticore HTTP client
        method => get
        url => "https://abc.com/SWS/incidents.asmx/Incident?INumber=103"
        headers => {
          Accept => "application/xml"
        }
				ssl_certificate_validation => false
				client_cert => "C:\Users\Downloads\logstash-6.4.1\logstash-6.4.1\data\SII_3938.p12"
				#truststore_password => "Pass"
		
      }
    }
 
 
    metadata_target => "http_poller_metadata"
		schedule => { cron => "* * * * * UTC"}
  }
}

and i get the below error

[2018-09-27T12:15:57,150][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-09-27T12:15:57,791][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.1"}
[2018-09-27T12:16:00,379][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-09-27T12:16:02,521][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2018-09-27T12:16:02,537][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2018-09-27T12:16:02,787][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}
[2018-09-27T12:16:02,865][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-09-27T12:16:02,865][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-09-27T12:16:02,896][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1"]}
[2018-09-27T12:16:02,912][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-09-27T12:16:02,928][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-09-27T12:16:02,943][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"cron"=>"* * * * * UTC"}, :timeout=>nil}
[2018-09-27T12:16:03,006][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x302ece6f run>"}
[2018-09-27T12:16:03,068][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-09-27T12:16:03,475][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-09-27T12:17:00,622][ERROR][logstash.codecs.json     ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unrecognized token 'Fault': was expecting ('true', 'false' or 'null')
 at [Source: (String)"Fault occured
"; line: 1, column: 6]>, :data=>"Fault occured\r\n"}

(Magnus B├Ąck) #2

Since the HTTP payload returned isn't JSON you should set the http_poller's codec option to "plain".


(Android Games) #3

Thanks @magnusbaeck i updated the config file as below

input {
 http_poller {
   urls => {
     myurl => "https://abc.com/SWS/incidents.asmx/GetSev"
   }
   codec => "plain"
   cacert => "sampl.pem"
   truststore => "downloaded_truststore.jks"
   truststore_password => "abc"
   schedule => { cron => "* * * * * UTC"}
 }
 


}

output 
{
  elasticsearch 
  {
    index => "Severity"
  }
}

I've followed the steps given in https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html to import the certificate to trust store.

C:\Users\Downloads\logstash-6.4.1\logstash-6.4.1\bin>logstash -f Poller_CA.conf
Sending Logstash logs to C:/Users/Downloads/logstash-6.4.1/logstash-6.4.1/logs which is now configured via log4j2.properties
[2018-09-28T10:45:47,633][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-09-28T10:45:48,188][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.1"}
[2018-09-28T10:45:50,319][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-09-28T10:45:50,741][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2018-09-28T10:45:50,757][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2018-09-28T10:45:50,958][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}
[2018-09-28T10:45:51,011][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-09-28T10:45:51,011][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-09-28T10:45:51,042][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1"]}
[2018-09-28T10:45:51,058][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-09-28T10:45:51,073][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-09-28T10:45:51,089][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"cron"=>"* * * * * UTC"}, :timeout=>nil}
[2018-09-28T10:45:51,127][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x23756161 run>"}
[2018-09-28T10:45:51,174][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-09-28T10:45:51,523][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{ 923609448 rufus-scheduler intercepted an error:
  923609448   job:
  923609448     Rufus::Scheduler::CronJob "* * * * * UTC" {}
  923609448   error:
  923609448     923609448
  923609448     Java::JavaSecurityCert::CertificateParsingException
  923609448     signed fields invalid
  923609448       sun.security.x509.X509CertImpl.parse(sun/security/x509/X509CertImpl.java:1791)
  923609448       sun.security.x509.X509CertImpl.<init>(sun/security/x509/X509CertImpl.java:195)
  923609448       sun.security.provider.X509Factory.parseX509orPKCS7Cert(sun/security/provider/X509Factory.java:471)
  923609448       sun.security.provider.X509Factory.engineGenerateCertificates(sun/security/provider/X509Factory.java:356)
  923609448       java.security.cert.CertificateFactory.generateCertificates(java/security/cert/CertificateFactory.java:462)
  923609448       java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
  923609448       org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423)
  923609448       org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290)
  923609448       C_3a_.Users.Chezhian.Downloads.logstash_minus_6_dot_4_dot_1.logstash_minus_6_dot_4_dot_1.vendor.bundle.jruby.$2_dot_3_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.block in setup_trust_store

(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.