# Parsing confluence audit json into Elastic with logstash **URL:** \ **Category:** Logstash\ **Created:** [December 11, 2024, 1:45am UTC](https://discuss.elastic.co/t/parsing-confluence-audit-json-into-elastic-with-logstash/371798 "2024-12-11T01:45:29Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Logistic\_dilated](https://avatars.discourse-cdn.com/v4/letter/l/87869e/32.png) [@Logistic\_dilated](https://discuss.elastic.co/u/Logistic_dilated)\ **Post date:** [December 11, 2024, 1:45am UTC](https://discuss.elastic.co/t/parsing-confluence-audit-json-into-elastic-with-logstash/371798/1 "2024-12-11T01:45:29Z")
Hello, I'm testing scraping our company confluence (self hosted) audit api so permission changes etc. are in ELK (also self hosted). I've got an http\_poller pipeline that is successfully polling the api, however the event shows up in elastic as a giant blob with the fields in an unreadable order. If I curl the api and pipe to a file, the format looks like this (truncated for brevity)- ```auto { "results": [ { "author": { "type": "user", "displayName": "admin", "username": "admin", "userKey": "" }, "remoteAddress": "192.168.1.171", "creationDate": 1733800689935, "summary": "Audit Log search performed", "description": "", "category": "Auditing", "sysAdmin": false, "affectedObject": { "name": "", "objectType": "" }, "changedValues": [], "associatedObjects": [] }, { "author": { "type": "user", "displayName": "System", "userKey": "-1" }, "creationDate": 1700046445029, "summary": "User added to group", "description": "", "category": "Users and groups", "sysAdmin": false, "affectedObject": { "name": "My_Group_Name", "objectType": "Group" }, "changedValues": [], "associatedObjects": [ { "name": "john.smith", "objectType": "User" } ] } ], "start": 0, "limit": 1000, "size": 1000, "_links": { "self": "https://confluence.domain.net:8443/rest/api/audit", "base": "https://confluence.domain.net:8443", "context": "" } } ``` The pipeline config looks like this ```auto input { http_poller { urls => { confluence => { method => get url => "https://confluence.domain.net:8443/rest/api/audit" headers => { Authorization => "Bearer /p" Accept => "application/json" } } } truststore => "/etc/logstash/config/certs/trusted_certs.jks" truststore_password => "redacted" request_timeout => 60 # Supports "cron", "every", "at" and "in" schedules by rufus scheduler schedule => { cron => "*/5 * * * *"} codec => "json" # A hash of request metadata info (timing, response headers, etc.) will be sent here # metadata_target => "http_poller_metadata" } } output { elasticsearch { hosts => ["https://127.0.0.1:9200"] index => "confluence-%{+YYYY.MM.dd}" user => "logstash-writer" password => "redacted" ssl => "true" ssl_certificate_verification => "false" cacert => "/etc/logstash/config/certs/http_ca.crt" codec => "json" } } ``` When I search the index in kibana, the output looks like this ```auto { "_links.base": [ "https://confluence.domain.net:8443" ], "_links.base.keyword": [ "https://confluence.domain.net:8443" ], "_links.context": [ "" ], "_links.context.keyword": [ "" ], "_links.self": [ "https://confluence.domain.net:8443/rest/api/audit" ], "_links.self.keyword": [ "https://confluence.domain.net:8443/rest/api/audit" ], "@timestamp": [ "2024-12-11T01:35:00.785Z" ], "@version": [ "1" ], "@version.keyword": [ "1" ], "event.original": [ "{\"results\":[{\"author\":{\"type\":\"user\",\"displayName\":\"admin\",\"username\":\"admin\",\"userKey\":\""\"},\"remoteAddress\":\"192.168.1.172\",\"creationDate\":1733880760520,\"summary\":\"Audit Log search performed\",\"description\":\"\",\"category\":\"Auditing\",\"sysAdmin\":false,\"affectedObject\":{\"name\":\"\",\"objectType\":\"\"},\"changedValues\":[],\"associatedObjects\":[]},{\"author\":{\"type\":\"user\",\"displayName\":\"admin\",\"username\":\"admin\",\"userKey\":\"\"},\"remoteAddress\":\"192.168.1.172\",\"creationDate\":1733880460633,\"summary\":\"Audit Log search performed\",\"description\":\"\",\"category\":\"Auditing\",\"sysAdmin\":false,\"affectedObject\":{\"name\":\"\",\"objectType\":\"\"},\"changedValues\":[],\"associatedObjects\":[]}],\"start\":0,\"limit\":2,\"size\":2,\"_links\":{\"self\":\"https://confluence.domain.net:8443/rest/api/audit\",\"base\":\"https://confluence.domain.net:8443\",\"context\":\"\"}}" ], "event.original.keyword": [ "{\"results\":[{\"author\":{\"type\":\"user\",\"displayName\":\"admin\",\"username\":\"admin\",\"userKey\":\"\"},\"remoteAddress\":\"192.168.1.172\",\"creationDate\":1733880760520,\"summary\":\"Audit Log search performed\",\"description\":\"\",\"category\":\"Auditing\",\"sysAdmin\":false,\"affectedObject\":{\"name\":\"\",\"objectType\":\"\"},\"changedValues\":[],\"associatedObjects\":[]},{\"author\":{\"type\":\"user\",\"displayName\":\"admin\",\"username\":\"admin\",\"userKey\":\"\"},\"remoteAddress\":\"192.168.1.172\",\"creationDate\":1733880460633,\"summary\":\"Audit Log search performed\",\"description\":\"\",\"category\":\"Auditing\",\"sysAdmin\":false,\"affectedObject\":{\"name\":\"\",\"objectType\":\"\"},\"changedValues\":[],\"associatedObjects\":[]}],\"start\":0,\"limit\":2,\"size\":2,\"_links\":{\"self\":\"https://confluence.domain.net:8443/rest/api/audit\",\"base\":\"https://confluence.domain.net:8443\",\"context\":\"\"}}" ], "limit": [ 2 ], "results.affectedObject.name": [ "", "" ], "results.affectedObject.name.keyword": [ "", "" ], "results.affectedObject.objectType": [ "", "" ], "results.affectedObject.objectType.keyword": [ "", "" ], "results.author.displayName": [ "admin", "admin" ], "results.author.displayName.keyword": [ "admin", "admin" ], "results.author.type": [ "user", "user" ], "results.author.type.keyword": [ "user", "user" ], "results.author.userKey": [ "", "" ], "results.author.userKey.keyword": [ "", "" ], "results.author.username": [ "admin", "admin" ], "results.author.username.keyword": [ "admin", "admin" ], "results.category": [ "Auditing", "Auditing" ], "results.category.keyword": [ "Auditing", "Auditing" ], "results.creationDate": [ 1733880760520, 1733880460633 ], "results.description": [ "", "" ], "results.description.keyword": [ "", "" ], "results.remoteAddress": [ "192.168.1.172", "192.168.1.172" ], "results.remoteAddress.keyword": [ "192.168.1.172", "192.168.1.172" ], "results.summary": [ "Audit Log search performed", "Audit Log search performed" ], "results.summary.keyword": [ "Audit Log search performed", "Audit Log search performed" ], "results.sysAdmin": [ false, false ], "size": [ 2 ], "start": [ 0 ], "_id": "Impas5MBRqo3Lxj_7boe", "_ignored": [ "event.original.keyword" ], "_index": "confluence-2024.12.11", "_score": null } ``` Is there a way to get each result from the json as an individual event in elastic? Ie. is there a way to get ```auto { "author": { "type": "user", "displayName": "System", "userKey": "-1" }, "creationDate": 1700046445029, "summary": "User added to group", "description": "", "category": "Users and groups", "sysAdmin": false, "affectedObject": { "name": "My_Group_Name", "objectType": "Group" }, "changedValues": [], "associatedObjects": [ { "name": "john.smith", "objectType": "User" } ] } ``` as a single event? I tried adding a filter, but it didn't seem to make any sort of impact on how the data looked. ```auto filter { json { source => "results" } } ``` ---
**Author:** ![sholzhauer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sholzhauer/32/110282_2.png) [@sholzhauer](https://discuss.elastic.co/u/sholzhauer)\ **Post date:** [December 12, 2024, 1:52pm UTC](https://discuss.elastic.co/t/parsing-confluence-audit-json-into-elastic-with-logstash/371798/2 "2024-12-12T13:52:08Z")
So before I dive into your technical details. Are you aware of the [Atlassian Confluence](https://www.elastic.co/guide/en/integrations/current/atlassian_confluence.html) integration for Fleet which does all the work for you? Post that for logstash I am pretty sure you are looking for the [split](https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html) plugin. ```auto filter { split { field => "results" } } ``` ---
**Author:** ![Logistic\_dilated](https://avatars.discourse-cdn.com/v4/letter/l/87869e/32.png) [@Logistic\_dilated](https://discuss.elastic.co/u/Logistic_dilated)\ **Post date:** [December 19, 2024, 3:02am UTC](https://discuss.elastic.co/t/parsing-confluence-audit-json-into-elastic-with-logstash/371798/3 "2024-12-19T03:02:26Z")
Thank you this is exactly what I was looking for!