Kibana URL template with relative path

surajbarde · April 16, 2020, 11:17pm

I have a question regarding Kibana URL templates for fields in the index.

I have used relative path for the below URL template.
On local environment it is replaced with http://localhost:5601/s/reporting/app/kibana#/..... Which is expected.
On my DEV and QA environment also it is replaced with http://localhost:5601 instead of http://dev.somedomain.com:5601 which is not expected.
I have my DEV and QA environment Kibana on some domain (DNS name) and everything is working fine except this URL template.

Kibana Setup:
Version : 7.6.0

Kibana.yml

server.port: 5601
server.host: "dev.somedomain.com"

Please let me know what I am doing wrong, or point me in right direction.

Thanks

Luca_Belluccini · April 17, 2020, 12:12am

Hello @surajbarde

I've tested this on 7.6.0 and it works fine.

I've added a dummy relative path (e.g. /some/pathname/asset.png, with no prefix) and I get a link which points to https://mykibana.instance.com:5601/some/pathname/asset.png.

Did you export the index pattern from your DEV environment and you've imported it in your DEV/QA?

If it is the case, then the reason is the fact the URL scheme, hostname and port are stored in the index pattern.

Can you please try the following?

Go to the Index Pattern in Kibana which contains your URL field (in Management / Index Patterns
Click on the Index Pattern ID from the URL in the browser address bar (e.g.http://dev.somedomain.com:5601/app/kibana#/management/kibana/index_patterns/ff959d40-b880-11e8-a6d9-e546fe2bba5f?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-90d,to:now))&_a=(tab:indexedFields) ).
The ID in my case is ff959d40-b880-11e8-a6d9-e546fe2bba5f

Go to the Kibana Developer tool and execute

GET .kibana/_doc/index-pattern:<ID>?filter_path=_source.index-pattern.fieldFormatMap

Where <ID> is the ID we grabbed from the address.
E.g. in my case is:

GET .kibana/_doc/index-pattern:ff959d40-b880-11e8-a6d9-e546fe2bba5f?filter_path=_source.index-pattern.fieldFormatMap

Share the response with us.

I've found out a problem with the URL formatter in Kibana with relative links, which has been fixed on 7.6.0. This is not exactly your problem though.

Your case might deserve a bug or an enhancement as it seems it has never been reported.

surajbarde · April 17, 2020, 1:15am

Thank you for your quick reply @Luca_Belluccini .

Did you export the index pattern from your DEV environment and you've imported it in your DEV/QA?

Yes I have developed in local and use the snapshot and restore to promote it to DEV and QA environment.

I run below in Kibana Dev Tools for my DEV/QA/local environment, for all index patterns which have URL field

GET .kibana_1/_doc/index-pattern:448fb160-752c-11ea-874a-89c9b90911fc?filter_path=_source.index-pattern.fieldFormatMap

My all responses were empty :

{ }

Please let me know if there is anything else I can check.

Luca_Belluccini · April 17, 2020, 7:22am

Can you try the following?

GET .kibana_1/_doc/index-pattern:448fb160-752c-11ea-874a-89c9b90911fc
GET .kibana/_doc/index-pattern:448fb160-752c-11ea-874a-89c9b90911fc
GET .kibana*/_doc/index-pattern:448fb160-752c-11ea-874a-89c9b90911fc

surajbarde · April 17, 2020, 4:54pm

@Luca_Belluccini I have tried below queries they both give same results.

GET .kibana_1/_doc/index-pattern:448fb160-752c-11ea-874a-89c9b90911fc
GET .kibana/_doc/index-pattern:448fb160-752c-11ea-874a-89c9b90911fc
{ "_index" : ".kibana_1",
  "_type" : "_doc",
  "_id" : "index-pattern:448fb160-752c-11ea-874a-89c9b90911fc",
  "found" : false}

Also below gave me exception :There are no external requests known to support wildcards that don't support replacing their indices

  GET .kibana*/_doc/index-pattern:448fb160-752c-11ea-874a-89c9b90911fc

So I tried this in my DEV machine, as I have a space called reporting :

 GET .kibana_1/_doc/reporting:index-pattern:448fb160-752c-11ea-874a-89c9b90911fc?filter_path=_source.index-pattern.fieldFormatMap

Result is showing that origin value is coming multiple times as per the URL template use as localhost even if its DEV:

{  "_source" : {    "index-pattern" : {
"fieldFormatMap" : """{"clientinfoid":{"id":"number","params":
{"parsedUrl":{"origin":"http://localhost:5601"
    ,"pathname":"/s/reporting/app/kibana","basePath":"/s/reporting"},"pattern":"00.[000]"}},
    "corporationcode":{"id":"url","params":{"parsedUrl":
{"origin":"http://localhost:5601","pathname":"/s/reporting/app/kibana","basePath":"/s/reporting"},
        "urlTemplate":"/s/reporting/app/kibana#/dashboard/d4d630d0-6fa7-11ea-9103-d5c86fe87988?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-1y,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),panels:!((embeddableConfig:(title:'SFTP%20Time%20Series'),gridData:(h:10,i:'9d28ce5e-0aa4-45f1-9879-1a8533c896eb',w:21,x:0,y:0),id:'052bbc90-6fa9-11ea-9103-d5c86fe87988',panelIndex:'9d28ce5e-0aa4-45f1-9879-1a8533c896eb',title:'SFTP%20Time%20Series',type:visualization,version:'7.6.0'),(embeddableConfig:(title:SFTP-Corporation),gridData:(h:10,i:'71ac2ec6-682f-4d0e-934c-467ad50cc38f',w:15,x:21,y:0),id:f18060a0-6fa9-11ea-9103-d5c86fe87988,panelIndex:'71ac2ec6-682f-4d0e-934c-467ad50cc38f',title:SFTP-Corporation,type:visualization,version:'7.6.0'),(embeddableConfig:(title:''),gridData:(h:19,i:'551865a8-2bce-428b-bedc-c6f5c831a7c1',w:48,x:0,y:10),id:'32771cf0-6fa7-11ea-9103-d5c86fe87988',panelIndex:'551865a8-2bce-428b-bedc-c6f5c831a7c1',type:search,version:'7.6.0')),query:(language:kuery,query:'corporationcode%20:%20{{value}}'),timeRestore:!t,title:'IMPORT%20Summary%20Dashboard',viewMode:view)","labelTemplate":"{{value}}","openLinkInCurrentTab":true}}}"""}  }}

So is there any way that this Origin value can be taken from Kibana.yml file, so that in each environment we don't have to change it manually.

Also I have noticed, that in DEV environment after promotion using snapshot and restore, if I manually change the URL template to give full path with http://dev.somedomain.com:5601 , the links in dashboards works as expected, but the query in Dev Tools still returns the same origin value as localhost.

Luca_Belluccini · April 17, 2020, 5:28pm

Hello,
The problem is the parsedUrl":{"origin" is stored when you define the url field format in the index pattern in the original Kibana instance.

If DEV is the environment in which you restored the index pattern snapshotted from the local environment, then this is the reason you have {"origin":"http://localhost:5601".

I think one way to solve it would be to change the type of corporationcode to string in the index pattern via the user interface, save the index pattern.
Then set it back to URL with the relative path and save again.

Never edit the documents in .kibana indices manually.

surajbarde · April 17, 2020, 10:34pm

@Luca_Belluccini, The solution you gave worked.

I have changed the corporationcode type to string and saved it and then set it back to URL with the relative path and saved again -- and it worked

This also changed parsedUrl":{"origin" properly.

But this is still manual step.

Is there any way I can use any script/curl command/api to change the parsedUrl":{"origin" property?
So that I can just run that through some script. This way whenever I do promotion to any environment using snapshot and restore, I can just run the script as a part of deployment to solve this issue.

Please let me know any way to automate it.

Thanks.

Luca_Belluccini · April 17, 2020, 10:40pm

I see 2 options:

Exporting the index pattern, patching the file and importing it again
Performing a partial update via api

I am on mobile now, but the best is to open a bug/improvement issue in the Kibana repository.
I can do it if you wish.

I'll also suggest you how to patch the index pattern document.

surajbarde · April 17, 2020, 10:53pm

Thank you @Luca_Belluccini for quick response.

Can you open a bug/improvement issue in the Kibana repository, as I am not aware of how to do it.

Also The second option you gave :

Performing a partial update via api

Can you give me any sample for it, or any reference, like which api to use.

Exporting the index pattern, patching the file and importing it again

For the first option, If I use then do I have to do below steps for promotion:

Use Snapshot and restore.
Then import all indexes which have this URL field(after patching these files).
Do I need to this for every environment or just from local to DEV, as the promotion sequence we have is local-->DEV-->QA-->UAT-->IST-->PRODUCTION.

Second option of a partial update via api looks easy for my condition.

Please let me know.

Thanks

Luca_Belluccini · April 18, 2020, 12:51pm

One possible workaround might be:

Get the index-pattern using its title or the ID

# Search by index-pattern title
GET .kibana/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": {
              "value": "index-pattern"
            }
          }
        },
        {
          "term": {
            "index-pattern.title": {
              "value": "kibana_sample_data_logs"
            }
          }
        }
      ]
    }
  }
}
# Or if you know the ID
GET .kibana/_doc/index-pattern:90943e30-9a47-11e8-b64d-95841ca0b247

Get the content of the field fieldFormatMap.
E.g.

"fieldFormatMap": """{"url.keyword":{"id":"url","params":{"parsedUrl":{"origin":"https://localhost:9243","pathname":"/app/kibana","basePath":""},"urlTemplate":"/some/pathname/asset.png"}},"url":{"id":"url","params":{"parsedUrl":{"origin":"https://localhost:9243","pathname":"/app/kibana","basePath":""},"urlTemplate":"/some/pathname/asset.png"}}}"""

Get the _id of the index-pattern.
E.g. index-pattern:90943e30-9a47-11e8-b64d-95841ca0b247
Modify the content of the fieldFormatMap to replace the URL of your local environment with the correct one
Execute a partial update.
E.g.

POST .kibana/_update/index-pattern:90943e30-9a47-11e8-b64d-95841ca0b247
{
  "doc": {
    "index-pattern": {
      "fieldFormatMap": """{"url.keyword":{"id":"url","params":{"parsedUrl":{"origin":"https://yourdevenv:5601","pathname":"/app/kibana","basePath":""},"urlTemplate":"/some/pathname/asset.png"}},"url":{"id":"url","params":{"parsedUrl":{"origin":"yourdevenv:5601","pathname":"/app/kibana","basePath":""},"urlTemplate":"/some/pathname/asset.png"}}}"""
    }
  }
}

All those requests can be done using bash, curl and sed and automated in a script.

I've opened the following for reference:

surajbarde · April 21, 2020, 9:05pm

Thank you @Luca_Belluccini

This solution worked for me.
Also thank you for opening the issue for me, it will be great to have this feature by default in Kibana.

Regards,
Suraj

system · May 19, 2020, 9:05pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
URL Links from Index Patterns Redirect to Port 5601 Kibana	3	476	December 21, 2020
URL path in Index pattern > Edit path is not working for server file link Kibana	3	532	September 10, 2020
Kibana URL template dynamic url generation Kibana	1	182	April 27, 2023
Absolute instead of relative URL field Kibana	1	391	February 18, 2022
Reference hosted URL and Port from Kibana Kibana	7	450	July 16, 2020

Kibana URL template with relative path

Related topics