Not able to see all fields when trying to create visualization

Dinesh_Potey · August 6, 2020, 4:12pm

Hi ,

I am trying to create error dashboard for which firstly I have to create a visualization out of my Index pattern. I am not able to see all the elements of my xml when I am trying to filter out to create the visualization. Not sure why all fields are not visible/available. Kindly help me here and let me know if you require more information from my end.

I am able to extract the below xml when written the grok code and out of this xml  I wanted to create a dashboard which is a stopper now.


2020 Jul 27 16:56:21:263 GMT +0200 BW.LogisticProcesses-2-LogisticProcesses-040101-1 Activity [Transport] EAI-04006 MSGID=tsCaw/h2YAHl8En/1Q-7eu3U6W2 Sending synchronous RV response
Subject: _INBOX.0A0A8001.5E969A7A38164.29506
<?xml version="1.0" encoding="UTF-8"?>
<ns0:PerformLogisticActivity.1 xmlns:ns0="http://xmlns.kpn.com/fixed/LogisticProcesses/Response/PerformLogisticActivity.1.xsd">
    <ns1:CMH xmlns:ns1="http://xmlns.kpn.com/common/cdm/Base.xsd">
        <ns1:ID>1687</ns1:ID>
        <ns1:NAME>PerformLogisticActivity</ns1:NAME>
        <ns1:MID>1687.10</ns1:MID>
        <ns1:PARADIGM>RESPONSE</ns1:PARADIGM>
        <ns1:OBJECT>PerformLogisticActivity</ns1:OBJECT>
        <ns1:VERSION>1.0</ns1:VERSION>
        <ns1:FROM ns1:COMPONENT_ID="LogisticProcesses" ns1:COMPONENT_VERSION="4.1.1"/>
        <ns1:TO ns1:DESTINATION="_INBOX.0A0A8001.5E969A7A38164.29506"/>
        <ns1:TRACKING>
            <ns1:MESSAGE_ID>tsCaw/h2YAHl8En/1Q-7eu3U6W2</ns1:MESSAGE_ID>
            <ns1:EXTERNAL_REF>0762a483-b71a-4466-a21e-74682f6fe6c3</ns1:EXTERNAL_REF>
            <ns1:ITEM ns1:APPINFO="LogisticProcesses" ns1:EVENT="PerformLogisticActivity" ns1:TIMESTAMP="1595861781255"/>
            <ns1:ITEM xmlns:ns0="http://xmlns.kpn.com/fixed/LogisticProcesses/Request/PerformLogisticActivity.1.xsd" xmlns:ns1="http://xmlns.kpn.com/common/cdm/Base.xsd" ns1:APPINFO="GenericB2BGateway" ns1:EVENT="PerformLogisticActivity" ns1:TIMESTAMP="1595861693208"/>
        </ns1:TRACKING>
        <ns1:ATTRIBUTE xmlns:ns0="http://xmlns.kpn.com/fixed/LogisticProcesses/Request/PerformLogisticActivity.1.xsd" xmlns:ns1="http://xmlns.kpn.com/common/cdm/Base.xsd" ns1:NAME="SOURCE" ns1:VALUE="REVIVA"/>
        <ns1:ATTRIBUTE xmlns:ns0="http://xmlns.kpn.com/fixed/LogisticProcesses/Request/PerformLogisticActivity.1.xsd" xmlns:ns1="http://xmlns.kpn.com/common/cdm/Base.xsd" ns1:NAME="DESTINATION" ns1:VALUE="Koninklijke PTT Nederland"/>
        <ns1:ATTRIBUTE xmlns:ns0="http://xmlns.kpn.com/fixed/LogisticProcesses/Request/PerformLogisticActivity.1.xsd" xmlns:ns1="http://xmlns.kpn.com/common/cdm/Base.xsd" ns1:NAME="TRANSACTION_ID" ns1:VALUE="0762a483-b71a-4466-a21e-74682f6fe6c3"/>
    </ns1:CMH>
    <ns0:RESULTSTATUS xmlns:ns1="http://xmlns.kpn.com/common/cdm/ResultStatus.xsd" ns1:STATUS="41" ns1:ERROR_CODE="LP-ERR-001" ns1:ERROR_DESCRIPTION="One or more errors occurred and the order could not be processed. See the result_details for a full report."/>
    <ns1:PerformLogisticActivityResponse1.0 xmlns:ns1="http://xmlns.kpn.com/EAI/fixed/0737_PerformLogisticActivity.1/1.0/Response">
        <BODY>
            <RESULT_DETAILS>
                <RESULT_DETAIL>
                    <TYPE>Activity</TYPE>
                    <STEP>PerformLogisticActivity</STEP>
                    <STATUS_CODE>1</STATUS_CODE>
                    <ERROR_CODE>10120</ERROR_CODE>
                    <ERROR_DESCRIPTION>An RV timeout occured in Sending/Receiving  sync response message</ERROR_DESCRIPTION>
                </RESULT_DETAIL>
            </RESULT_DETAILS>
        </BODY>
    </ns1:PerformLogisticActivityResponse1.0>
</ns0:PerformLogisticActivity.1>

Note that I will be making use of fields inside RESULT_DETAIL for my dashboard.

Thanks,
Dinesh Potey

Badger · August 6, 2020, 4:24pm

Please edit your post, select your data, and click on </> in the toolbar above the edit pane. That will preserve the formatting of your data and make it much easier to read. Please do not apply that markdown to ordinary text that needs to be re-flowed.

What have you tried so far, and what do you not like about the results?

Dinesh_Potey · August 6, 2020, 4:28pm

Is it fine now?

Dinesh_Potey · August 6, 2020, 4:31pm

This is the content in my .conf file.

input { file {
path => "/tibco/tra/domain/eai-v1/application/logs/LogisticProcesses-*"
start_position => "beginning"
type => "xml"
codec => multiline {
          pattern => "^%{YEAR} "
          negate => true
	  what => "previous"
 	  auto_flush_interval => 1
                  }

 }}

filter {

grok {   match => { "message" => ["%{YEAR:Year} %{MONTH:Month} %{MONTHDAY:Day} %{INT:Hour}:%{INT:Minutes}:%{INT:Seconds}:%{INT:MS} %{WORD:Zone} +%{INT:ZoneNum} BW.%{NOTSPACE:Adapter} %{NOTSPACE:Role} %{NOTSPACE:Category} %{NOTSPACE:MsgCode} MSGID=(?<MSGID>[[:ascii:]]{27}) Received synchronous RV request %{GREEDYDATA:Rest}","%{YEAR:Year} %{MONTH:Month} %{MONTHDAY:Day} %{INT:Hour}:%{INT:Minutes}:%{INT:Seconds}:%{INT:MS} %{WORD:Zone} +%{INT:ZoneNum} BW.%{NOTSPACE:Adapter} %{NOTSPACE:Role} %{NOTSPACE:Category} %{NOTSPACE:MsgCode} MSGID=(?<MSGID>[[:ascii:]]{27}) %{GREEDYDATA:Rest}"] }}

if "_grokparsefailure" in [tags] {
      drop { }
    }

mutate {
      remove_field => ["Zone", "ZoneNum", "MsgCode", "Job"]
    }


xml { source => "Rest" target => "lpXML" store_xml => true }

}

I defined index pattern and also can see expected xml in discover whereas while creating the visualisation , I don't see all fields

Badger · August 6, 2020, 6:15pm

It would be better if you did not </> the regular text. I have to scroll way to the right to read it. Remove the 4 blank lines at the start of sentences like "I am trying to create".

grok does not match multiline patterns by default. Try changing

grok {   match => { "message" => ["%{YEAR:Year}

to

grok {   match => { "message" => ["(?m)%{YEAR:Year}

However, there is a simpler solution. When using store_xml => true the xml filter is incredibly tolerant of junk preceding the XML. You can just use

xml { source => "message" target => "lpXML" store_xml => true }

Note that if you use the xpath option on an xml filter that has zero tolerance for any junk in the XML. It must be valid when using xpath.

Dinesh_Potey · August 7, 2020, 8:11am

Hi Badger,

With grok pattern I shared earlier, I am able to see the index created under index pattern and index management on kibana. The problem is , under kibana--> saved objects -> [mm-eai-logisticprocesses] I don't see the required fields for only this index [mm-eai-logisticprocesses]. For another index( see below snippet) which has same data , I get all the required error fields. My requirement is to fetch below xml and all the fields in the xml.

.

2020 Jul 27 16:56:21:263 GMT +0200 BW.LogisticProcesses-2-LogisticProcesses-040101-1 Activity [Transport] EAI-04006 MSGID=tsCaw/h2YAHl8En/1Q-7eu3U6W2 Sending synchronous RV response
Subject: _INBOX.0A0A8001.5E969A7A38164.29506
<?xml version="1.0" encoding="UTF-8"?>
<ns0:PerformLogisticActivity.1 xmlns:ns0="http://xmlns.kpn.com/fixed/LogisticProcesses/Response/PerformLogisticActivity.1.xsd">
    <ns1:CMH xmlns:ns1="http://xmlns.kpn.com/common/cdm/Base.xsd">
        <ns1:ID>1687</ns1:ID>
        <ns1:NAME>PerformLogisticActivity</ns1:NAME>
        <ns1:MID>1687.10</ns1:MID>
        <ns1:PARADIGM>RESPONSE</ns1:PARADIGM>
        <ns1:OBJECT>PerformLogisticActivity</ns1:OBJECT>
        <ns1:VERSION>1.0</ns1:VERSION>
        <ns1:FROM ns1:COMPONENT_ID="LogisticProcesses" ns1:COMPONENT_VERSION="4.1.1"/>
        <ns1:TO ns1:DESTINATION="_INBOX.0A0A8001.5E969A7A38164.29506"/>
        <ns1:TRACKING>
            <ns1:MESSAGE_ID>tsCaw/h2YAHl8En/1Q-7eu3U6W2</ns1:MESSAGE_ID>
            <ns1:EXTERNAL_REF>0762a483-b71a-4466-a21e-74682f6fe6c3</ns1:EXTERNAL_REF>
            <ns1:ITEM ns1:APPINFO="LogisticProcesses" ns1:EVENT="PerformLogisticActivity" ns1:TIMESTAMP="1595861781255"/>
            <ns1:ITEM xmlns:ns0="http://xmlns.kpn.com/fixed/LogisticProcesses/Request/PerformLogisticActivity.1.xsd" xmlns:ns1="http://xmlns.kpn.com/common/cdm/Base.xsd" ns1:APPINFO="GenericB2BGateway" ns1:EVENT="PerformLogisticActivity" ns1:TIMESTAMP="1595861693208"/>
        </ns1:TRACKING>
        <ns1:ATTRIBUTE xmlns:ns0="http://xmlns.kpn.com/fixed/LogisticProcesses/Request/PerformLogisticActivity.1.xsd" xmlns:ns1="http://xmlns.kpn.com/common/cdm/Base.xsd" ns1:NAME="SOURCE" ns1:VALUE="REVIVA"/>
        <ns1:ATTRIBUTE xmlns:ns0="http://xmlns.kpn.com/fixed/LogisticProcesses/Request/PerformLogisticActivity.1.xsd" xmlns:ns1="http://xmlns.kpn.com/common/cdm/Base.xsd" ns1:NAME="DESTINATION" ns1:VALUE="Koninklijke PTT Nederland"/>
        <ns1:ATTRIBUTE xmlns:ns0="http://xmlns.kpn.com/fixed/LogisticProcesses/Request/PerformLogisticActivity.1.xsd" xmlns:ns1="http://xmlns.kpn.com/common/cdm/Base.xsd" ns1:NAME="TRANSACTION_ID" ns1:VALUE="0762a483-b71a-4466-a21e-74682f6fe6c3"/>
    </ns1:CMH>
    <ns0:RESULTSTATUS xmlns:ns1="http://xmlns.kpn.com/common/cdm/ResultStatus.xsd" ns1:STATUS="41" ns1:ERROR_CODE="LP-ERR-001" ns1:ERROR_DESCRIPTION="One or more errors occurred and the order could not be processed. See the result_details for a full report."/>
    <ns1:PerformLogisticActivityResponse1.0 xmlns:ns1="http://xmlns.kpn.com/EAI/fixed/0737_PerformLogisticActivity.1/1.0/Response">
        <BODY>
            <RESULT_DETAILS>
                <RESULT_DETAIL>
                    <TYPE>Activity</TYPE>
                    <STEP>PerformLogisticActivity</STEP>
                    <STATUS_CODE>1</STATUS_CODE>
                    <ERROR_CODE>10120</ERROR_CODE>
                    <ERROR_DESCRIPTION>An RV timeout occured in Sending/Receiving  sync response message</ERROR_DESCRIPTION>
                </RESULT_DETAIL>
            </RESULT_DETAILS>
        </BODY>
    </ns1:PerformLogisticActivityResponse1.0>
</ns0:PerformLogisticActivity.1>

system · September 4, 2020, 8:11am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.