Filebeat with LogstashXML parser

leggrand · August 10, 2018, 12:50pm

I create a template, with field maps, but the fields is empty. My logstash config is:
input {
beats {
port => 5044
client_inactivity_timeout => 900
}
}

filter {
        xml {
                source => "message"
                target => "doc"
                store_xml => false
                xpath => [
                        "/error/@errorId", "errorId"
                        "/error/@type", "tipo"
                        "/error/@source", "fonte"
						"/error/@time", "registered"
                        "/error/@message", "mensagem"
                        "/error/@detail", "detalhe"
                        "//item[@name='REMOTE_ADDR']/value/@string", "REMOTE_ADDR"
                        "//item[@name='ALL_RAW']/value/@string", "ALL_RAW"
                        "//item[@name='APPL_MD_PATH']/value/@string", "APPL_MD_PATH"
                        "//item[@name='APPL_PHYSICAL_PATH']/value/@string", "APPL_PHYSICAL_PATH"
                        "//item[@name='AUTH_USER']/value/@string", "AUTH_USER"
                        "//item[@name='REQUEST_METHOD']/value/@string", "REQUEST_METHOD"
                        "//item[@name='URL']/value/@string", "URL"
                ]
        }
        date {
            match => ["registered", "yyyy-mm-dd'T'hh:mm:ss.SSSZ"]
            target => "registered"
            timezone => "America/Sao_Paulo"
        }
}

output {
        stdout {
                codec => rubydebug
        }
        elasticsearch {
                hosts => ["http://localhost:9200"]
                index => "elmah-log-%{+YYYY.MM.dd}"
                document_id => "%{errodId}"
        }
}

Please, can you help me?

magnusbaeck · August 10, 2018, 1:15pm

Show an example event produced by your stdout { codec => rubydebug } output.

leggrand · August 10, 2018, 2:17pm

Hello, magnusbaeck.
Stdout didn't work, the file is empty. But the data is input at elasticsearch, just don't fill the correct fields.

magnusbaeck · August 10, 2018, 2:30pm

Okay, fine, copy/paste from the JSON tab in Kibana. I just need to understand exactly what the document looks like.

leggrand · August 10, 2018, 2:51pm

I will split the JSON in two parts, is to big:

{
  "_index": "elmah-log-2018.08.10",
  "_type": "doc",
  "_id": "%{[errodId]}",
  "_version": 1,
  "_score": 1,
  "_source": {
    "offset": 0,
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "input": {
      "type": "log"
    },
    "@timestamp": "2018-08-10T14:37:06.774Z",
    "beat": {
      "version": "6.3.2",
      "name": "WGIP000655",
      "hostname": "WGIP000655"
    },
    "@version": "1",
    "host": {
      "name": "WGIP000655"
    },
    "prospector": {
      "type": "log"
    },
    "message": "",
    "source": "c:\\repositorio\\logs\\biolab\\elmah\\error-2018-02-18.xml"
  },
  "fields": {
    "@timestamp": [
      "2018-08-10T14:37:06.774Z"
    ]
  }
}

leggrand · August 10, 2018, 2:56pm

the field message, partial

"message": "<error errorId=\"7aa4d701-04a5-4173-bb08-9936aa7e8f7a\" application=\"/LM/W3SVC/1/ROOT/api-vialivre/biolab\" host=\"HPV01-VM29\" type=\"InterPlayers.Api.MarketPlace.Core.Infrastructure.Exceptions.ObjectNotFoundException\" message=\"Entidade não localizada.\" source=\"InterPlayers.Api.MarketPlace.Core\" detail=\"System.Web.HttpUnhandledException (0x80004005): Entidade não localizada. ---> InterPlayers.Api.MarketPlace.Core.Infrastructure.Exceptions.ObjectNotFoundException: Entidade não localizada.
 at InterPlayers.Api.MarketPlace.Core.Business.Services.Impl.PdvService.Get(String cnpj)
\" user=\"client:vialivre.app\" time=\"2018-08-02T19:11:06.9543231Z\" statusCode=\"500\" webHostHtmlMessage=\"\">\n <serverVariables>\n <item name=\"ALL_HTTP\">\n <value string=\"HTTP_AUTHORIZATION:Bearer gAAAAKIT06ICfXlWAbdVEInBkkCGoz1zay674ndZ4zJ72PtNkaPi5sLuTni7wjoTJPOOy_FXbfNXwI8yqYfcOxG7w4_qCNlMGxPB1GPThdaYG3;
HTTP_HOST:apigip01
\" />\n </item>\n <item name=\"ALL_RAW\">\n <value string=\"Authorization: Bearer gAAAAKIT06ICfXlWAbdVEInBkkCGoz1zay674ndZ4zJ72PtNkaPi5sLuTni7wjoTJ-wVFEPd4ZbjThRTCqYQ7JKI0MDE2UCUVPZPGsAcfQ-BY
Host: apigip01
\" />\n </item>\n <item name=\"APPL_MD_PATH\">\n <value string=\"/LM/W3SVC/1/ROOT/api-vialivre/biolab\" />\n </item>\n <item name=\"APPL_PHYSICAL_PATH\">\n <value string=\"E:\\Sites\\api.projetos\\api.marketplace\\biolab\\\" />\n </item>\n <item name=\"AUTH_TYPE\">\n <value string=\"OAuth\" />\n </item>\n <item name=\"AUTH_USER\">\n <value string=\"client:vialivre.app\" />\n </item>\n <item name=\"AUTH_PASSWORD\">\n <value string=\"*****\" />\n </item>\n <item name=\"LOGON_USER\">\n <value string=\"\" />\n </item>\n <item name=\"REMOTE_USER\">\n <value string=\"client:vialivre.app\" />\n </item>\n <item name=\"CERT_COOKIE\">\n <value string=\"\" />\n </item>\n <item name=\"CERT_FLAGS\">\n <value string=\"\" />\n </item>\n <item name=\"CERT_ISSUER\">\n <value string=\"\" />\n </item>\n <item name=\"CERT_KEYSIZE\">\n <value string=\"256\" />\n </item>\n <item name=\"CERT_SECRETKEYSIZE\">\n <value string=\"2048\" />\n </item>\n <item name=\"CERT_SERIALNUMBER\">\n <value string=\"\" />\n </item>\n <item name=\"CERT_SERVER_ISSUER\">\n <value string=\"DC=lan, DC=interplayers, CN=interplayers-PLKJBT-SRV01-CA\" />\n </item>\n <item name=\"CERT_SERVER_SUBJECT\">\n <value string=\"C=BR, S=Sao Paulo, L=Sao Paulo, O=Interplayers Solucoes Integradas SA, OU=TI, CN=apigip01\" />\n </item>\n <item name=\"CERT_SUBJECT\">\n <value string=\"\" />\n </item>\n <item name=\"CONTENT_LENGTH\">\n <value string=\"0\" />\n </item>\n <item name=\"CONTENT_TYPE\">\n <value string=\"\" />\n </item>\n <item name=\"GATEWAY_INTERFACE\">\n <value string=\"CGI/1.1\" />\n </item>\n <item name=\"HTTPS\">\n <value string=\"on\" />\n </item>\n <item name=\"HTTPS_KEYSIZE\">\n <value string=\"256\" />\n </item>\n <item name=\"HTTPS_SECRETKEYSIZE\">\n <value string=\"2048\" />\n </item>\n <item name=\"HTTPS_SERVER_ISSUER\">\n <value string=\"DC=lan, DC=interplayers, CN=interplayers-PLKJBT-SRV01-CA\" />\n </item>\n <item name=\"HTTPS_SERVER_SUBJECT\">\n <value string=\"C=BR, S=Sao Paulo, L=Sao Paulo, O=Interplayers Solucoes Integradas SA, OU=TI, CN=apigip01\" />\n </item>\n <item name=\"INSTANCE_ID\">\n <value string=\"1\" />\n </item>\n <item name=\"INSTANCE_META_PATH\">\n <value string=\"/LM/W3SVC/1\" />\n </item>\n <item name=\"LOCAL_ADDR\">\n <value string=\"192.168.0.156\" />\n </item>\n <item name=\"PATH_INFO\">\n <value string=\"/api-vialivre/biolab/pontos-de-venda/79430682030200\" />\n </item>\n <item name=\"PATH_TRANSLATED\">\n <value string=\"E:\\Sites\\api.projetos\\api.marketplace\\biolab\\pontos-de-venda\\79430682030200\" />\n </item>\n <item name=\"QUERY_STRING\">\n <value string=\"\" />\n </item>\n <item name=\"REMOTE_ADDR\">\n <value string=\"192.168.0.164\" />\n </item>\n <item name=\"REMOTE_HOST\">\n <value string=\"192.168.0.164\" />\n </item>\n <item name=\"REMOTE_PORT\">\n <value string=\"51031\" />\n </item>\n <item name=\"REQUEST_METHOD\">\n <value string=\"GET\" />\n </item>\n <item name=\"SCRIPT_NAME\">\n <value string=\"/api-vialivre/biolab/pontos-de-venda/79430682030200\" />\n </item>\n <item name=\"SERVER_NAME\">\n <value string=\"apigip01\" />\n </item>\n <item name=\"SERVER_PORT\">\n <value string=\"443\" />\n </item>\n <item name=\"SERVER_PORT_SECURE\">\n <value string=\"1\" />\n </item>\n <item name=\"SERVER_PROTOCOL\">\n <value string=\"HTTP/1.1\" />\n </item>\n <item name=\"SERVER_SOFTWARE\">\n <value string=\"Microsoft-IIS/8.5\" />\n </item>\n <item name=\"URL\">\n <value string=\"/api-vialivre/biolab/pontos-de-venda/79430682030200\" />\n </item>\n <item name=\"HTTP_AUTHORIZATION\">\n <value string=\"Bearer gAAAAKIT06ICfXlWAbdVEInBkkCGoz1zay674ndZ4zJ72PtNkaPi5sLuTni7wjoTJPOOy_FXbfNXwI8yqYfcOxG7w4_qCNlMGxPB1GPThdaYG3_IyWBQGljwVFEPd4ZbjThRTCqYQ7JKI0MDE2UCUVPZPGsAcfQ-BY\" />\n </item>\n <item name=\"HTTP_HOST\">\n <value string=\"apigip01\" />\n </item>\n </serverVariables>"

magnusbaeck · August 13, 2018, 5:48pm

I don't know what's going on here. Have you looked in the Logstash log for clues? Increasing the Logstash loglevel might reveal more information.

leggrand · August 14, 2018, 6:18pm

I solved this issue!

The xpath was wrong.
Before: "/error/@errorId", "errorId"
I changed to: "string(/error/@errorId)", "errorId"

Tks for your help!

system · September 11, 2018, 6:18pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
XML filter not parsing Logstash	6	481	November 4, 2018
Fields/tag set custom fields in filebeat, but logstash gives error Beats filebeat	1	330	December 20, 2019
Custom index with field from xpath Logstash	9	2031	May 24, 2017
Custom fields.yml Beats filebeat	3	1635	May 28, 2019
Filebeat to elasticsearch configuration(mapping) Beats filebeat	3	1466	May 27, 2021

Filebeat with LogstashXML parser

Related topics