NOT ABLE TO INDEX DATA USING HTTP POLLER

rajatbhardwaj1393 · February 14, 2024, 10:07am

want to index data from the third party api.
trying to index data from response but the data is coming as array of results and ruby code below is not storing it as individual event. Can someone suggest what i am doing wrong:

filter {
ruby {
code => "
results = event.get('[results]')
if results.is_a?(Array)
results.each { |result|
cloned_event = LogStash::Event.new
cloned_event.set('result', result)
id = result['id']
cloned_event.set('[@metadata][document_id]', id) # Set document ID for Elasticsearch
pipeline.push(cloned_event)
}
event.cancel # Cancel the original event after processing
end
"
}
}

output {
elasticsearch {
hosts => "${ELASTICSEARCH.HOST}"
index => "logstash.glenigans.projects"
user => "${ELASTIC.USER}"
password => "${ELASTIC.PASSWORD}"
ca_trusted_fingerprint => "${CA.FINGERPRINT}"
action => "update"
doc_as_upsert => "true"
document_id => "%{[@metadata][document_id]}"
}
}

leandrojmp · February 14, 2024, 12:22pm

Hello,

Please share your entire logstash pipeline and also a sample of your log so it can be replicated.

It is not possible to know what is happening without those.

Also, if your events are in an array, you can simply use the split filter to create one event per each item on the array, there is no need to write a ruby code to replicate this.

rajatbhardwaj1393 · February 14, 2024, 12:51pm

Thanks for reply.

Making an http poller call here:

input {
http_poller {
urls => {
test1 => {
method => post
url => "https://www.gleniganapi.com/glenigan/project/_search?key=abcd"
body => '{
"from": 0,
"size": 2,
"query": {
"bool": {
"must": [
{
"nested": {
"path": "ProjectHistories",
"query": {
"bool": {
"must": [
{
"range": {
"ProjectHistories.ProjectHistoryModified": {
"gte": "2024-01-24T00:00:00",
"lte": "2024-01-26T00:00:00"
}
}
}
]
}
}
}
}
],
"filter": [
{
"bool": {
"should": [
{
"term": {
"Funding": "Private"
}
},
{
"term": {
"Funding": "Housing Association"
}
}
]
}
},
{
"bool": {
"must": [
{
"term": {
"IsArchived": "false"
}
}
]
}
}
]
}
}
}'
headers => {
"Content-Type" => "application/json"
}
}
}
schedule => { cron => "00 12 * * * UTC" }
codec => "json"
}
}

filter {
ruby {
code => "
results = event.get('[results]')
if results.is_a?(Array)
results.each { |result|
cloned_event = LogStash::Event.new
cloned_event.set('result', result)
id = result['id']
cloned_event.set('[@metadata][document_id]', id) # Set document ID for Elasticsearch
filter_matched(cloned_event) # Indicate that the cloned event has been processed successfully
}
event.cancel # Cancel the original event after processing
end
"
}
}

output {
elasticsearch {
hosts => "${ELASTICSEARCH.HOST}"
index => "logstash.glenigans.projects"
user => "${ELASTIC.USER}"
password => "${ELASTIC.PASSWORD}"
ca_trusted_fingerprint => "${CA.FINGERPRINT}"
action => "update"
doc_as_upsert => "true"
document_id => "%{[@metadata][document_id]}"
}
}

response looks like after input :
{"@version"=>"1",
"results"=>[
{"source"=>{"RoleLocationsLevel3WithRole"=>"South West#Wiltshire#Swindon#Client / Promoter#Client", "AddressLine1"=>nil, "MaterialsChildren"=>"", "RoleOfficeIds"=>"25020675~50146217", "FloorArea"=>nil, "ShlaaStartYear"=>nil, "Units"=>nil, "RoleLocationsLevel1WithRole"=>"South West#Client / Promoter#Client", "RecallDate"=>"2024-02-23T00:00:00", "RolesDetails"=>[{"RoleGroupDescription"=>"Client / Promoter", "Roles"=>[{"CompaniesInRole"=>[{"ContractStageDescription"=>"Applications to Tender", "BusinessType"=>"Government Dept./Agency", "OfficeId"=>"25020675", "Value"=>nil, "ContactsInCompanyInRole"=>[{"KeyCard_Gender"=>"M", "LinkedInUrl"=>nil, "KeyCard_FullContact"=>"Mr David Webb", "KeyCard_Phone1NoTps"=>nil, "FromDate"=>nil, "KeyCard_Phone1WithTps"=>"", "KeyCard_ContactId"=>"50038773", "KeyCard_Phone1"=>nil, "ToDate"=>nil, "DateAdded"=>"2024-01-24T14:06:14", "JobTitleForThisProject"=>"Estates Project Manager", "ContactId"=>"50038773"}], "ContractProcurementType"=>"Competitive Tender", "HeadOfficeId"=>"50146217", "StartDate"=>nil, "ContractType"=>"Framework Agreement", "NoLongerInvolved"=>"false"}], "DateAdded"=>"2024-01-24T14:06:07", "RoleDescription"=>"Client"}]}], "RoleList"=>[{"ContractStageDescription"=>"Applications to Tender", "KeyCard_ContractOfficeId"=>"25020675", "RoleParentRole"=>"Client / Promoter#Client", "RoleParent"=>"Client / Promoter", "LocationLevel2"=>"South West#Wiltshire", "LocationLevel1"=>"South West", "KeyCard_ContractTenderDueDate"=>nil, "ContractProcurementType"=>"Competitive Tender", "KeyCard_CountyCode"=>"65", "KeyCard_ContractPhone1WithTps"=>"01793 414 047", "NoLongerInvolved"=>"false", "OfficeIds"=>["25020675", "50146217"], "KeyCard_ContractStreetAddress"=>"The Engine House, Fire Fly Avenue", "KeyCard_ContractPhone1ShowTps"=>"01793 414 047", "Value"=>nil, "Role"=>"Client", "PostCodeDistrict"=>"SN2", "ContractType"=>"Framework Agreement", "KeyCard_ContractPhone1NoTps"=>"01793 414 047", "KeyCard_ContractRoleCode"=>"100", "OfficeType"=>"Office", "KeyCard_ContractId"=>"42083022", "HeadOfficeId"=>"50146217", "PostCode"=>"SN2 2EH", "KeyCard_ContractOfficeBusinessCode"=>"400", "KeyCard_ContractWinnerOfficeId"=>"25020675", "BusinessType"=>"Government Dept./Agency", "KeyCard_ContractPhone1"=>"01793 414 047", "OfficeId"=>"25020675", "KeyCard_ContractCountryCode"=>"UK", "KeyCard_ContractTownCode"=>"SN11", "DateAdded"=>"2024-01-24T14:06:07", "LocationLevel3"=>"South West#Wiltshire#Swindon", "StartDate"=>nil, "Eircode"=>nil}], "RoleLocationsLevel2"=>"South West#Wiltshire", "ProjectCounty"=>"Wiltshire", "ProjectRegion"=>"South West", "RoleLocationsLevel2WithRole"=>"South West#Wiltshire#Client / Promoter#Client", "VerticalHeight"=>nil, "Borough"=>nil, "NotificationText"=>nil, "OSEasting8"=>nil, "OfficeNames"=>"Historic England", "Client"=>"Historic England~25020675", "RoleLocationsLevel3WithRoleParents"=>"South West#Wiltshire#Swindon#Client / Promoter", "TenderInformation"=>"Applications to tender invited.", "ConstructionPeriodDescription"=>"36 months", "UndergroundStoreys"=>nil, "ProjectHistories"=>[{"ProjectHistoryModified"=>"2024-01-24T14:12:41", "ProjectHistoryFieldName"=>"PlanningStage", "ProjectHistoryModifiedBy"=>"Byggfakta Researcher 7", "ProjectHistoryOldValue"=>nil, "ProjectHistoryNewValue"=>"Planning Not Required", "ProjectHistoryChange"=>"Planning stage change", "ProjectHistoryEvent"=>"PSTG"}, {"ProjectHistoryModified"=>"2024-01-24T14:12:41", "ProjectHistoryFieldName"=>"ContractStage", "ProjectHistoryModifiedBy"=>"Byggfakta Researcher 7", "ProjectHistoryOldValue"=>nil, "ProjectHistoryNewValue"=>"Applications to Tender", "ProjectHistoryChange"=>"Contract stage change", "ProjectHistoryEvent"=>"CNSTG"}, {"ProjectHistoryModified"=>"2024-01-24T14:12:41", "ProjectHistoryFieldName"=>nil, "ProjectHistoryModifiedBy"=>"Byggfakta Researcher 7", "ProjectHistoryOldValue"=>nil, "ProjectHistoryNewValue"=>"24023160", "ProjectHistoryChange"=>"New Project", "ProjectHistoryEvent"=>"NewProject"}, {"ProjectHistoryModified"=>"2024-01-24T14:12:41", "ProjectHistoryFieldName"=>"Contract", "ProjectHistoryModifiedBy"=>"Byggfakta Researcher 7", "ProjectHistoryOldValue"=>nil, "ProjectHistoryNewValue"=>"Client", "ProjectHistoryChange"=>"New role added", "ProjectHistoryEvent"=>"NROL"}], "Funding"=>["Private"], "LatestInformation"=>"Applications to tender are currently invited for main contractors.

Expressions of interest should be made via Historic England Electronic Tendering Site - Home by 22nd of February 2024, 12:00 PM.", "ChangeOfUseFrom"=>nil, "Applications"=>[{"PermissionDate"=>nil, "CapturedDate"=>"2024-01-24T00:00:00", "FloorArea"=>nil, "RefusedDate"=>nil, "Units"=>nil, "HasPlansProposed"=>false, "CouncilName"=>"Swindon", "ReceivedDate"=>nil, "ApplicationType"=>"Standard building application", "HasPlansExisting"=>false, "Heading"=>"Planning not required", "CouncilUrl"=>"https://pa.swindon.gov.uk/publicaccess/", "URL"=>nil, "ApplicationDescription"=>"Scheme comprises of the services of an Architect/Chartered Building Surveyor/Chartered Architectural Technologist for works projects on the Historic England Estate. To include, when required, fulfilling the role of Principal Designer and Lead Professional. And when a project requires, services of a range of professional team roles.Location The location of the Works in respect of this contract is:Lot 1: North and Midlands Territory (York Office, Newcastle Office, Manchester Office, Baguley Hall [Manchester], Shrewsbury Flaxmill Maltings [Shrewsbury])Lot 2: South West, South East and London Territory (Bristol Office, Swindon Office, Cambridge Office, London Office, Fort Cumberland [Portsmouth], Bow Hill [Exeter]).Lot 1 - North & Midlands Territory (York Office, Newcastle Office, Manchester Office, Baguley Hall [Manchester], Shrewsbury Flaxmill Maltings [Shrewsbury])Lot 2 - South West, South East & London Territory (Bristol Office, Swindon Office, Cambridge Office, London Office, Fort Cumberland [Portsmouth], Bow Hill [Exeter])", "PermissionExpiryDate"=>nil, "ApplicationDate"=>nil, "HasDesignAndAccessStatements"=>false, "HasStatements"=>false, "TargetDecisionDate"=>nil, "AppealDate"=>nil, "PlanningStageDescription"=>nil, "IsLead"=>"true", "ProposalId"=>"20261953", "WithdrawnDate"=>nil, "OfficeId"=>"25020675", "IsLatest"=>false, "PlanningTypeDescription"=>"Planning not required", "ApplicationDocs"=>{"Statements"=>, "PlansProposed"=>, "DesignandAccessStatements"=>, "ApplicationForms"=>, "PlansExisting"=>}, "CommitteeDate"=>nil, "PlanningRecommendation"=>nil, "HasApplicationForms"=>false, "ApplicationNumber"=>"N/A"}], "Address"=>"Swindon, Wiltshire, SN2", "RoleLocationsLevel1WithRoleParents"=>"South West#Client / Promoter", "IsArchived"=>"false", "StartDate"=>"2024-06-01T00:00:00", "CshRating"=>nil, "OSNorthing8"=>nil, "ContractStageParent"=>"Tenders", "MasterLeadOf"=>, "ProjectLandTypes"=>, "BedroomsNumber"=>nil, "ProjectMaterials"=>"", "Sectors"=>[{"SectorValue"=>nil, "FloorArea"=>nil, "KeyCard_CategoryCode"=>"731", "Sector"=>"Offices", "Units"=>nil, "DevelopmentType"=>"Refurbishment", "BuildingType"=>"Office Buildings", "SectorType"=>"Secondary", "Storeys"=>nil, "PlanningUseClass"=>nil, "KeyCard_CategoryPriceGroupDesc"=>"Offices / Commercial"}, {"SectorValue"=>nil, "FloorArea"=>nil, "KeyCard_CategoryCode"=>"1011", "Sector"=>"Community & Amenity", "Units"=>nil, "DevelopmentType"=>"Refurbishment", "BuildingType"=>"Government Buildings", "SectorType"=>"Primary", "Storeys"=>nil, "PlanningUseClass"=>nil, "KeyCard_CategoryPriceGroupDesc"=>"Community & Amenity"}], "ProjectTown"=>"Swindon", "Structures"=>nil, "AdditionalText"=>"This contract has been advertised in the Find a Tender. The reference number is 2024/S 000-002403.

Start and end date are a guideline only.", "ProjectLocationLevel1Facet"=>"South West", "RoleLocationsLevel2WithRoleParents"=>"South West#Wiltshire#Client / Promoter", "Website"=>"", "ProjectLocationLevel3Facet"=>"South West#Wiltshire#Swindon", "IsMasterLead"=>true, "Length"=>nil, "Value"=>0.45, "ProjectId"=>"24023160", "PrimaryAndSecondarySectors"=>"Community & Amenity#Government Buildings~Offices#Office Buildings", "ParkingSpaces"=>nil, "PlanningDescription"=>"Planning approval is not required.", "SolarPanels"=>nil, "SecondarySectorParents"=>"Offices", "LiveHistoric"=>"L", "LatestEventDate"=>"2024-01-24T14:12:41", "SiteName"=>"Various Locations", "CivilDescription"=>nil, "RoleOfficeIdsWithRoleParents"=>"25020675#Client / Promoter~50146217#Client / Promoter", "LatestInformationModifiedDate"=>"2024-01-24T14:03:22", "PlanningStageParent"=>"Plans Approved", "DemolitionDescription"=>"", "ParentMasterLead"=>, "PrimarySectorParents"=>"Community & Amenity", "KeyCard_ContractStageDescription"=>"Applications to Tender", "BasementParkingSpaces"=>nil, "ValueType"=>"Estimated", "OJEU"=>, "SiteArea"=>nil, "SecondarySectors"=>"Offices#Office Buildings", "ProjectLocationLevel2Facet"=>"South West#Wiltshire", "PrimaryHeading"=>"Architectural, Surveying, Principal Designer and Professional Services Framework", "ProjectName"=>"Framework Commission for Architectural, Surveying, Principal Designer and Lead Professional Services on the Historic England Estate", "EndDate"=>"2027-06-01T00:00:00", "ProjectStatus"=>"In Progress", "ResearcherName"=>"Emma Hewitt", "ContractPeriod"=>36, "RoleLocationsLevel3"=>"South West#Wiltshire#Swindon", "ProjectOfficeIds"=>"25020675", "SectorExpertise"=>"Community & Amenity", "PreliminaryStartDate"=>nil, "PlanningTypes"=>"Planning not required", "ContractType"=>"", "NewsArticles"=>, "ProjectLocationFullFacetPath"=>"South West#Wiltshire#Swindon", "SitePhoneNumber"=>"", "FirstPublished"=>"2024-01-24T14:07:13", "ContractStage"=>"Tenders#Applications to Tender", "PrimaryAndSecondarySectorParents"=>"Community & Amenity~Offices", "SecondaryHeading"=>"Framework Commission for Architectural, Surveying, Principal Designer and Lead Professional Services on the Historic England Estate / Various Locations", "OSAccuracy"=>nil, "LatestEvent"=>"Planning stage change", "RoleOfficeIdsWithRole"=>"25020675#Client / Promoter#Client~50146217#Client / Promoter#Client", "AddressLine3"=>"", "Ceequal"=>{"InterimGrade"=>nil, "Code"=>nil, "ForecastFinalVerification"=>nil, "AwardType"=>nil, "FinalPercentage"=>nil, "InterimPercentage"=>nil, "Version"=>nil, "FinalGrade"=>nil}, "ProjectSize"=>"Large", "Updated"=>"2024-01-24T14:07:21", "PrimarySectors"=>"Community & Amenity#Government Buildings", "ShlaaEndYear"=>nil, "EndDateType"=>"Guideline", "KeyCard_MaterialsWithCodes"=>"", "StartDateType"=>"Guideline", "ElectricVehicleChargingPoints"=>nil, "ShlaaReference"=>nil, "Heading"=>"Architectural, Surveying, Principal Designer and Professional Services Framework", "Storeys"=>nil, "RoleParents"=>"Client / Promoter", "LastModifiedDate"=>"2024-01-24T14:07:22", "MaterialParents"=>"", "Breeam"=>{"Score"=>nil, "Reference"=>nil, "Stage"=>nil, "CertifiedDate"=>nil, "Rating"=>nil, "Version"=>nil, "CertifiedDateShort"=>nil}, "RoleLocationsLevel1"=>"South West", "SchemeDescription"=>"Scheme comprises of the services of an Architect/Chartered Building Surveyor/Chartered Architectural Technologist for works projects on the Historic England Estate. To include, when required, fulfilling the role of Principal Designer and Lead Professional. And when a project requires, services of a range of professional team roles.Location The location of the Works in respect of this contract is:Lot 1: North and Midlands Territory (York Office, Newcastle Office, Manchester Office, Baguley Hall [Manchester], Shrewsbury Flaxmill Maltings [Shrewsbury])Lot 2: South West, South East and London Territory (Bristol Office, Swindon Office, Cambridge Office, London Office, Fort Cumberland [Portsmouth], Bow Hill [Exeter]).

Lot 1 - North & Midlands Territory (York Office, Newcastle Office, Manchester Office, Baguley Hall [Manchester], Shrewsbury Flaxmill Maltings [Shrewsbury])

Lot 2 - South West, South East & London Territory (Bristol Office, Swindon Office, Cambridge Office, London Office, Fort Cumberland [Portsmouth], Bow Hill [Exeter])", "ProjectPlanningAuthorities"=>"Swindon", "DevelopmentType"=>"Refurbishment", "ProjectTenderReturnDate"=>nil, "PlanningStage"=>"Plans Approved#Planning Not Required", "ProjectPostcode"=>"SN2", "Bidders"=>, "ProjectRoles"=>"Client / Promoter#Client", "ProjectMaterialsTexts"=>, "AddressLine2"=>nil, "LocationSource"=>"Multiple Sites", "ProjectType"=>"Construction"}, "id"=>"24023160"},
{"source"=>{"RoleLocationsLevel3WithRole"=>"Yorkshire and the Humber#Humberside#Beverley#Client / Promoter#Client~Yorkshire and the Humber#Humberside#Beverley#Architect / Plans By#Architect", "AddressLine1"=>"Cherry Burton Leisure Park", "MaterialsChildren"=>"Metal cladding~Photovoltaic Panels~Aluminium framed~Timber~Soakaways~White Lining~Planting~Black Top Surfacing~Kitchen~Bathroom", "RoleOfficeIds"=>"50576650~20572599", "FloorArea"=>477, "ShlaaStartYear"=>nil, "Units"=>1, "RoleLocationsLevel1WithRole"=>"Yorkshire and the Humber#Client / Promoter#Client~Yorkshire and the Humber#Architect / Plans By#Architect", "RecallDate"=>"2024-04-22T00:00:00", "RolesDetails"=>[{"RoleGroupDescription"=>"Client / Promoter", "Roles"=>[{"CompaniesInRole"=>[{"ContractStageDescription"=>"Contract Awarded", "BusinessType"=>"Private Client", "OfficeId"=>"50576650", "Value"=>nil, "ContactsInCompanyInRole"=>[{"KeyCard_Gender"=>"U", "LinkedInUrl"=>nil, "KeyCard_FullContact"=>"Mr Chris Vaudin", "KeyCard_Phone1NoTps"=>nil, "FromDate"=>nil, "KeyCard_Phone1WithTps"=>"", "KeyCard_ContactId"=>"25248020", "KeyCard_Phone1"=>nil, "ToDate"=>nil, "DateAdded"=>"2024-01-24T13:03:02", "JobTitleForThisProject"=>"Director", "ContactId"=>"25248020"}], "ContractProcurementType"=>nil, "HeadOfficeId"=>"50576650", "StartDate"=>nil, "ContractType"=>nil, "NoLongerInvolved"=>"false"}], "DateAdded"=>"2022-09-21T07:19:48", "DateUpdated"=>"2024-01-24T13:03:03", "RoleDescription"=>"Client"}]}, {"RoleGroupDescription"=>"Architect / Plans By", "Roles"=>[{"CompaniesInRole"=>[{"ContractStageDescription"=>"Contract Awarded", "BusinessType"=>"Architect/Interior Desr", "OfficeId"=>"20572599", "Value"=>nil, "ContactsInCompanyInRole"=>[{"KeyCard_Gender"=>"F", "LinkedInUrl"=>"https://uk.linkedin.com/in/stella-squire-93a85838", "DateUpdated"=>"2023-11-24T14:22:02", "KeyCard_FullContact"=>"Ms Stella Squire", "KeyCard_Phone1NoTps"=>nil, "FromDate"=>"2022-09-21T00:00:00", "KeyCard_Phone1WithTps"=>"", "KeyCard_ContactId"=>"50298608", "KeyCard_Phone1"=>nil, "ToDate"=>nil, "DateAdded"=>"2022-09-21T07:20:40", "JobTitleForThisProject"=>"Office Manager", "ContactId"=>"50298608"}], "ContractProcurementType"=>nil, "HeadOfficeId"=>"20572599", "StartDate"=>nil, "ContractType"=>nil, "NoLongerInvolved"=>"false"}], "DateAdded"=>"2022-09-21T07:19:48", "DateUpdated"=>"2023-11-24T14:22:02", "RoleDescription"=>"Architect"}]}], "RoleList"=>[{"ContractStageDescription"=>"Contract Awarded", "KeyCard_ContractOfficeId"=>"50576650", "DateUpdated"=>"2024-01-24T13:03:03", "RoleParentRole"=>"Client / Promoter#Client", "RoleParent"=>"Client / Promoter", "LocationLevel2"=>"Yorkshire and the Humber#Humberside", "LocationLevel1"=>"Yorkshire and the Humber", "KeyCard_ContractTenderDueDate"=>nil, "ContractProcurementType"=>nil, "KeyCard_CountyCode"=>"41", "KeyCard_ContractPhone1WithTps"=>" TPS Screened", "NoLongerInvolved"=>"false", "OfficeIds"=>["50576650"], "KeyCard_ContractStreetAddress"=>"The Bungalow, Leconfield Road", "KeyCard_ContractPhone1ShowTps"=>" (CTPS)", "Value"=>nil, "Role"=>"Client", "PostCodeDistrict"=>"HU17", "ContractType"=>nil, "KeyCard_ContractPhone1NoTps"=>nil, "KeyCard_ContractRoleCode"=>"100", "OfficeType"=>"Head Office", "KeyCard_ContractId"=>"40870616", "HeadOfficeId"=>"50576650", "PostCode"=>"HU17 7LJ", "KeyCard_ContractOfficeBusinessCode"=>"631", "KeyCard_ContractWinnerOfficeId"=>"50576650", "BusinessType"=>"Private Client", "KeyCard_ContractPhone1"=>nil, "OfficeId"=>"50576650", "KeyCard_ContractCountryCode"=>"UK", "KeyCard_ContractTownCode"=>"HU01", "DateAdded"=>"2022-09-21T07:19:48", "LocationLevel3"=>"Yorkshire and the Humber#Hum2024-02-06T10:40:04.342456675Z berside#Beverley", "StartDate"=>nil, "Eircode"=>nil}, {"ContractStageDescription"=>"Contract Awarded", "KeyCard_ContractOfficeId"=>"FR6820030813012", "DateUpdated"=>"2023-11-24T14:22:02", "RoleParentRole"=>"Architect / Plans By#Architect", "RoleParent"=>"Architect / Plans By", "LocationLevel2"=>"Yorkshire and the Humber#Humberside", "LocationLevel1"=>"Yorkshire and the Humber", "KeyCard_ContractTenderDueDate"=>nil, "ContractProcurementType"=>nil, "KeyCard_CountyCode"=>"41", "KeyCard_ContractPhone1WithTps"=>"01482 888102", "NoLongerInvolved"=>"false", "OfficeIds"=>["20572599"], "KeyCard_ContractStreetAddress"=>"54 Lairgate", "KeyCard_ContractPhone1ShowTps"=>"01482 888102", "Value"=>nil, "Role"=>"Architect", "PostCodeDistrict"=>"HU17", "ContractType"=>nil, "KeyCard_ContractPhone1NoTps"=>"01482 888102", "KeyCard_ContractRoleCode"=>"500", "OfficeType"=>"Head Office", "KeyCard_ContractId"=>"40870615", "HeadOfficeId"=>"20572599", "PostCode"=>"HU17 8EU", "KeyCard_ContractOfficeBusinessCode"=>"60", "KeyCard_ContractWinnerOfficeId"=>"FR6820030813012", "BusinessType"=>"Architect/Interior Desr", "KeyCard_ContractPhone1"=>"01482 888102", "OfficeId"=>"20572599", "KeyCard_ContractCountryCode"=>"UK", "KeyCard_ContractTownCode"=>"HU01", "DateAdded"=>"2022-09-21T07:19:48", "LocationLevel3"=>"Yorkshire and the Humber#Humberside#Beverley", "StartDate"=>nil, "Eircode"=>nil}], "RoleLocationsLevel2"=>"Yorkshire and the Humber#Humberside", "ProjectCounty"=>"Humberside", "ProjectRegion"=>"Yorkshire and the Humber", "RoleLocationsLevel2WithRole"=>"Yorkshire and the Humber#Humberside#Client / Promoter#Client~Yorkshire and the Humber#Humberside#Architect / Plans By#Architect", "VerticalHeight"=>0, "Borough"=>nil, "NotificationText"=>nil, "ProjectLocation"=>{"lat"=>53.86644785, "lon"=>-0.4832689}, "OSEasting8"=>"04998360", "Client"=>"Mr. Chris Vaudin~50576650", "OfficeNames"=>"Mr. Chris Vaudin~Salt Architects", "TenderInformation"=>"Contract details are yet to be established.", "RoleLocationsLevel3WithRoleParents"=>"Yorkshire and the Humber#Humberside#Beverley#Client / Promoter~Yorkshire and the Humber#Humberside#Beverley#Architect / Plans By", "ConstructionPeriodDescription"=>"7 months.", "UndergroundStoreys"=>0, "ProjectHistories"=>[{"ProjectHistoryModified"=>"2024-01-24T13:10:57", "ProjectHistoryFieldName"=>nil, "ProjectHistoryModifiedBy"=>"Private Housing 1", "ProjectHistoryOldValue"=>nil, "ProjectHistoryNewValue"=>"22350508", "ProjectHistoryChange"=>"Updated Project", "ProjectHistoryEvent"=>"UpdatedProject"}, {"ProjectHistoryModified"=>"2024-01-24T13:10:57", "ProjectHistoryFieldName"=>nil, "ProjectHistoryModifiedBy"=>"Private Housing 1", "ProjectHistoryOldValue"=>nil, "ProjectHistoryNewValue"=>nil, "ProjectHistoryChange"=>"Project Info Updated", "ProjectHistoryEvent"=>"FORCED"}, {"ProjectHistoryModified"=>"2023-08-21T10:44:11", "ProjectHistoryFieldName"=>"Planning Stage", "ProjectHistoryModifiedBy"=>"Real Time Updates", "ProjectHistoryOldValue"=>"Detailed Plans Submitted", "ProjectHistoryNewValue"=>"Detail Plans Granted", "ProjectHistoryChange"=>"Planning stage change", "ProjectHistoryEvent"=>"PSTG"}, {"ProjectHistoryModified"=>"2022-09-21T08:44:04", "ProjectHistoryFieldName"=>"Contract", "ProjectHistoryModifiedBy"=>"Real Time Updates", "ProjectHistoryOldValue"=>nil, "ProjectHistoryNewValue"=>"Client", "ProjectHistoryChange"=>"New role added", "ProjectHistoryEvent"=>"NROL"}, {"ProjectHistoryModified"=>"2022-09-21T08:44:04", "ProjectHistoryFieldName"=>"Contract", "ProjectHistoryModifiedBy"=>"Real Time Updates", "ProjectHistoryOldValue"=>nil, "ProjectHistoryNewValue"=>"Architect", "ProjectHistoryChange"=>"New role added", "ProjectHistoryEvent"=>"NROL"}, {"ProjectHistoryModified"=>"2022-09-21T08:44:04", "ProjectHistoryFieldName"=>"Planning Stage", "ProjectHistoryModifiedBy"=>"Real Time Updates", "ProjectHistoryOldValue"=>nil, "ProjectHistoryNewValue"=>"Detailed Plans Submitted", "ProjectHistoryChange"=>"Planning stage change", "ProjectHistoryEvent"=>"PSTG"}, {"ProjectHistoryModified"=>"2022-09-21T08:44:04", "ProjectHistoryFieldName"=>"ProjectId", "ProjectHistoryModifiedBy"=>"Real Time Updates", "ProjectHistoryOldValue"=>nil, "ProjectHistoryNewValue"=>"22350508", "ProjectHistoryChange"=>"New Project", "ProjectHistoryEvent"=>"NewProject"}], "Funding"=>["Private"], "LatestInformation"=>"We have been advised that the project is at an early stage and tenders are yet to be invited.", "ChangeOfUseFrom"=>nil, "Applications"=>[{"PermissionDate"=>"2023-08-16T00:00:00", "CapturedDate"=>"2022-09-20T00:00:00", "FloorArea"=>nil, "RefusedDate"=>nil, "Units"=>1, "HasPlansProposed"=>true, "CouncilName"=>"East Riding of Yorkshire", "ReceivedDate"=>"2022-08-17T00:00:00", "ApplicationType"=>"Standard building application", "HasPlansExisting"=>true, "Heading"=>"Multifunctional Building", "CouncilUrl"=>"https://newplanningaccess.eastriding.gov.uk/newplanningaccess/", "URL"=>"https://newplanningaccess.eastriding.gov.uk/newplanningaccess/applicationDetails.do?keyVal=RGRK5PBJGEW00&activeTab=summary", "ApplicationDescription"=>"Scheme comprises construction of a multifunctional building for use as kitchen/servery/bar-cafe-wedding venue and entertainment stage, following removal of marque used for temporary cafe/bar amenities, with provision of additional planting to create natural buffer zones to accommodate event spaces. This project also includes associated infrastructure works and access roads.", "PermissionExpiryDate"=>nil, "ApplicationDate"=>"2022-09-08T00:00:00", "HasDesignAndAccessStatements"=>true, "HasStatements"=>true, "TargetDecisionDate"=>"2023-05-04T00:00:00", "AppealDate"=>nil, "PlanningStageDescription"=>nil, "IsLead"=>"true", "ProposalId"=>"19694952", "WithdrawnDate"=>nil, "OfficeId"=>"50576650", "IsLatest"=>false, "PlanningTypeDescription"=>"Detailed Planning", "ApplicationDocs"=>{"Statements"=>["TREATMENT_PLANT_BROCHURE-4452054.pdf"], "PlansProposed"=>["PROPOSED_SITE_ELEVATIONS-4450545.pdf", "PROPOSED_SITE_PLAN-4450544.pdf"], "DesignandAccessStatements"=>["DESIGN_AND_ACCESS_STATEMENT-4432357.pdf"], "ApplicationForms"=>, "PlansExisting"=>["EXISTING_SITE_PLAN-4450547.pdf"]}, "CommitteeDate"=>nil, "PlanningRecommendation"=>nil, "HasApplicationForms"=>false, "ApplicationNumber"=>"22/02757/PLF"}], "Address"=>"Cherry Burton Leisure Park, Leconfield Road, Cherry Burton East Riding Of Y, Beverley, Humberside, HU17 7LU", "RoleLocationsLevel1WithRoleParents"=>"Yorkshire and the Humber#Client / Promoter~Yorkshire and the Humber#Architect / Plans By", "IsArchived"=>"false", "StartDate"=>"2024-10-28T00:00:00", "CshRating"=>nil, "OSNorthing8"=>"04422030", "ContractStageParent"=>"Pre-Tender", "MasterLeadOf"=>, "ProjectLandTypes"=>["Brownfield"], "BedroomsNumber"=>nil, "ProjectMaterials"=>"Roof#Metal cladding~Roof#Photovoltaic Panels~Windows#Aluminium framed~Doors#Timber~Site Works#Soakaways~Site Works#White Lining~Site Works#Planting~Site Works#Black Top Surfacing~Fittings#Kitchen~Fittings#Bathroom", "Sectors"=>[{"SectorValue"=>nil, "FloorArea"=>nil, "KeyCard_CategoryCode"=>"821", "Sector"=>"Hotel & Leisure", "Units"=>nil, "DevelopmentType"=>"New", "BuildingType"=>"Pubs/Wine Bars/Clubs", "SectorType"=>"Primary", "Storeys"=>nil, "PlanningUseClass"=>nil, "KeyCard_CategoryPriceGroupDesc"=>"Hotel & Leisure"}, {"SectorValue"=>nil, "FloorArea"=>nil, "KeyCard_CategoryCode"=>"826", "Sector"=>"Hotel & Leisure", "Units"=>nil, "DevelopmentType"=>"New", "BuildingType"=>"Cafés", "SectorType"=>"Secondary", "Storeys"=>nil, "PlanningUseClass"=>nil, "KeyCard_CategoryPriceGroupDesc"=>"Hotel & Leisure"}, {"SectorValue"=>nil, "FloorArea"=>nil, "KeyCard_CategoryCode"=>"914", "Sector"=>"Hotel & Leisure", "Units"=>nil, "DevelopmentType"=>"New", "BuildingType"=>"Conference/Exhibition", "SectorType"=>"Secondary", "Storeys"=>nil, "PlanningUseClass"=>nil, "KeyCard_CategoryPriceGroupDesc"=>"Hotel & Leisure"}, {"SectorValue"=>nil, "FloorArea"=>nil, "KeyCard_CategoryCode"=>"1625", "Sector"=>"Infrastructure", "Units"=>nil, "DevelopmentType"=>"New", "BuildingType"=>"Surface Car Parks", "SectorType"=>"Secondary", "Storeys"=>nil, "PlanningUseClass"=>nil, "KeyCard_CategoryPriceGroupDesc"=>"Civil (Infrastructure)"}, {"SectorValue"=>nil, "FloorArea"=>nil, "KeyCard_CategoryCode"=>"1675", "Sector"=>"Infrastructure", "Units"=>nil, "DevelopmentType"=>"New", "BuildingType"=>"Landscaping Works", "SectorType"=>"Secondary", "Storeys"=>nil, "PlanningUseClass"=>nil, "KeyCard_CategoryPriceGroupDesc"=>"Civil (Infrastructure)"}, {"SectorValue"=>nil, "FloorArea"=>nil, "KeyCard_CategoryCode"=>"1622", "Sector"=>"Infrastructure", "Units"=>nil, "DevelopmentType"=>"New", "BuildingType"=>"Estate Roads", "SectorType"=>"Secondary", "Storeys"=>nil, "PlanningUseClass"=>nil, "KeyCard_CategoryPriceGroupDesc"=>"Civil (Infrastructure)"}, {"SectorValue"=>nil, "FloorArea"=>nil, "KeyCard_CategoryCode"=>"1681", "Sector"=>"Infrastructure", "Units"=>nil, "DevelopmentType"=>"New", "BuildingType"=>"Infrastructure", "SectorType"=>"Secondary", "Storeys"=>nil, "PlanningUseClass"=>nil, "KeyCard_CategoryPriceGroupDesc"=>"Civil (Infrastructure)"}], "ProjectTown"=>"Beverley", "Structures"=>nil, "AdditionalText"=>"We are unable to verify the company and contact details for the Client.", "ProjectLocationLevel1Facet"=>"Yorkshire and the Humber", "RoleLocationsLevel2WithRoleParents"=>"Yorkshire and the Humber#Humberside#Client / Promoter~Yorkshire and the Humber#Humberside#Architect / Plans By", "Website"=>"", "ProjectLocationLevel3Facet"=>"Yorkshire and the Humber#Humberside#Beverley", "IsMasterLead"=>false, "Length"=>nil, "Value"=>1.32, "ProjectId"=>"22350508", "PrimaryAndSecondarySectors"=>"Hotel & Leisure#Pubs/Wine Bars/Clubs~Hotel & Leisure#Cafés~Hotel & Leisure#Conference/Exhibition~Infrastructure#Surface Car Parks~Infrastructure#Landscaping Works~Infrastructure#Estate Roads~Infrastructure#Infrastructure", "ParkingSpaces"=>100, "PlanningDescription"=>"Detailed plans have been granted.", "SolarPanels"=>nil, "SecondarySectorParents"=>"Hotel & Leisure~Infrastructure", "LiveHistoric"=>"L", "LatestEventDate"=>"2024-01-24T13:10:57", "SiteName"=>"Leconfield Road, Beverley", "CivilDescription"=>nil, "RoleOfficeIdsWithRoleParents"=>"50576650#Client / Promoter~20572599#Architect / Plans By", "LatestInformationModifiedDate"=>"2024-01-24T13:04:04", "PlanningStageParent"=>"Plans Approved", "DemolitionDescription"=>"", "ParentMasterLead"=>, "PrimarySectorParents"=>"Hotel & Leisure", "KeyCard_ContractStageDescription"=>"Pre-Tender", "BasementParkingSpaces"=>nil, "ValueType"=>"Calculated", "OJEU"=>, "SiteArea"=>2.3, "SecondarySectors"=>"Hotel & Leisure#Cafés~Hotel & Leisure#Conference/Exhibition~Infrastructure#Surface Car Parks~Infrastructure#Landscaping Works~Infrastructure#Estate Roads~Infrastructure#Infrastructure", "ProjectLocationLevel2Facet"=>"Yorkshire and the Humber#Humberside", "PrimaryHeading"=>"Multifunctional Building", "ProjectName"=>nil, "EndDate"=>"2025-06-09T00:00:00", "ProjectStatus"=>"In Progress", "ResearcherName"=>"Suzy Riding", "ContractPeriod"=>7, "RoleLocationsLevel3"=>"Yorkshire and the Humber#Humberside#Beverley", "ProjectOfficeIds"=>"50576650~20572599", "SectorExpertise"=>"Hotel & Leisure", "PreliminaryStartDate"=>nil, "PlanningTypes"=>"Detailed Planning", "ContractType"=>"", "NewsArticles"=>, "ProjectLocationFullFacetPath"=>"Yorkshire and the Humber#Humberside#Beverley", "SitePhoneNumber"=>"", "FirstPublished"=>"2022-09-21T08:16:01", "ContractStage"=>"Pre-Tender#Pre-Tender", "PrimaryAndSecondarySectorParents"=>"Hotel & Leisure~Hotel & Leisure~Infrastructure", "SecondaryHeading"=>"Leconfield Road, Beverley", "OSAccuracy"=>"A", "LatestEvent"=>"Project Info Updated", "RoleOfficeIdsWithRole"=>"50576650#Client / Promoter#Client~20572599#Architect / Plans By#Architect", "AddressLine3"=>"Cherry Burton East Riding Of Y", "Ceequal"=>{"InterimGrade"=>nil, "Code"=>nil, "ForecastFinalVerification"=>nil, "AwardType"=>nil, "FinalPercentage"=>nil, "InterimPercentage"=>nil, "Version"=>nil, "FinalGrade"=>nil}, "ProjectSize"=>"Large", "Updated"=>"2024-01-24T13:04:42", "PrimarySectors"=>"Hotel & Leisure#Pubs/Wine Bars/Clubs", "ShlaaEndYear"=>nil, "EndDateType"=>"Calculated", "KeyCard_MaterialsWithCodes"=>"Metal cladding#315~Photovoltaic Panels#333~Aluminium framed#404~Timber#501~Soakaways#1117~White Lining#1119~Planting#1154~Black Top Surfacing#1169~Kitchen#1204~Bathroom#1205", "StartDateType"=>"Calculated", "ElectricVehicleChargingPoints"=>nil, "ShlaaReference"=>nil, "Heading"=>"Multifunctional Building", "Storeys"=>0, "RoleParents"=>"Client / Promoter~Architect / Plans By", "LastModifiedDate"=>"2024-01-24T13:05:43", "MaterialParents"=>"Roof~Windows~Doors~Site Works~Fittings", "Breeam"=>{"Score"=>nil, "Reference"=>nil, "Stage"=>nil, "CertifiedDate"=>nil, "Rating"=>nil, "Version"=>nil, "CertifiedDateShort"=>nil}, "RoleLocationsLevel1"=>"Yorkshire and the Humber", "SchemeDescription"=>"Scheme comprises construction of a multifunctional building for use as kitchen/servery/bar-cafe-wedding venue and entertainment stage, following removal of marque used for temporary cafe/bar amenities, with provision of additional planting to create natural buffer zones to accommodate event spaces. This project also includes associated infrastructure works and access roads.", "ProjectPlanningAuthorities"=>"East Riding of Yorkshire", "DevelopmentType"=>"New", "ProjectTenderReturnDate"=>nil, "PlanningStage"=>"Plans Approved#Detail Plans Granted", "ProjectPostcode"=>"HU17 7LU", "Bidders"=>, "ProjectRoles"=>"Client / Promoter#Client~Architect / Plans By#Architect", "ProjectMaterialsTexts"=>["Roof#Metal cladding#Grey", "Roof#Photovoltaic Panels#", "Windows#Aluminium framed#Dark Grey", "Doors#Timber#", "Site Works#Soakaways#", "Site Works#White Lining#", "Site Works#Planting#", "Site Works#Black Top Surfacing#", "Fittings#Kitchen#", "Fittings#Bathroom#"], "AddressLine2"=>"Leconfield Road", "LocationSource"=>"Council - Application Form", "ProjectType"=>"Construction and Civil"}, "id"=>"22350508"},
]}

leandrojmp · February 14, 2024, 1:03pm

Yeah, I can't replicate because the response you shared is not in a valid json format.

But it seems that you want each item in the results array to be an independent event, right?

Remove your ruby filter, there is no need for it.

Use the split filter.

Replace your ruby filter with this:

filter {
    if [results][0] {
        split {
            field => "results"
        }
        mutate {
            add_field => {
                "[@metadata][document_id]" => "%{[results][id]}"
            }
        }
    }
}

The conditional will test if the results field is an array by checking if it has at least one item.

rajatbhardwaj1393 · February 14, 2024, 2:18pm

tried the above code but it index the first id but didnot index the complete result and for rest it is giving blank row

leandrojmp · February 14, 2024, 2:24pm

You need to share the output you are getting in Kibana.

Or provide a valid json of your input, it is not possible to test your pipeline without a valid json output to simulate.

rajatbhardwaj1393 · February 14, 2024, 2:41pm

thats the above output on kibana.

leandrojmp · February 14, 2024, 3:14pm

You need to share the valid json of the response of this request, the results document.

rajatbhardwaj1393 · February 14, 2024, 3:21pm

valid json response:
{
"total": 14004,
"results": [
{
"id": "24024743",
"source": {
"LastModifiedDate": "2024-01-26T04:21:59",
"Address": "Digberry Lane, Near Park Corner, Henley-On-Thames, Oxfordshire, RG9 6DP",
"Funding": [
"Private"
],
"Sectors": [
{
"FloorArea": null,
"BuildingType": "Farm Buildings",
"Storeys": null,
"Sector": "Industrial",
"DevelopmentType": "New",
"KeyCard_CategoryCode": "471",
"PlanningUseClass": null,
"KeyCard_CategoryPriceGroupDesc": "Industrial",
"Units": null,
"SectorType": "Primary",
"SectorValue": null
},
{
"FloorArea": null,
"BuildingType": "Workshops",
"Storeys": null,
"Sector": "Industrial",
"DevelopmentType": "New",
"KeyCard_CategoryCode": "413",
"PlanningUseClass": null,
"KeyCard_CategoryPriceGroupDesc": "Industrial",
"Units": null,
"SectorType": "Secondary",
"SectorValue": null
}
],
"FirstPublished": "2024-01-26T04:15:53",
"ContractStageParent": "Pre-Tender",
"ProjectLocationFullFacetPath": "South East#Oxfordshire#Henley-On-Thames",
"NotificationText": null,
"ProjectCounty": "Oxfordshire",
"SectorExpertise": "Industrial",
"ResearcherName": "Suzy Riding",
"RoleLocationsLevel2WithRole": "South East#Berkshire#Client / Promoter#Client",
"Breeam": {
"Score": null,
"CertifiedDateShort": null,
"Version": null,
"Reference": null,
"CertifiedDate": null,
"Rating": null,
"Stage": null
},
"ProjectMaterialsTexts": ,
"AdditionalText": null,
"RoleParents": "Client / Promoter",
"PrimaryAndSecondarySectors": "Industrial#Farm Buildings~Industrial#Workshops",
"ParentMasterLead": ,
"AddressLine3": "Near Park Corner",
"AddressLine2": "Digberry Lane",
"AddressLine1": null,
"SolarPanels": null,
"PreliminaryStartDate": null,
"PlanningStageParent": "Plans Approved",
"EndDateType": "Guideline",
"Heading": "Agricultural Storage Shed",
"ProjectName": null,
"PlanningDescription": "Planning approval is not required.",
"Website": "",
"RecallDate": null,
"ProjectLocation": {
"lon": -1.00348321,
"lat": 51.58969988
},
"ProjectTown": "Henley-On-Thames",
"Borough": null,
"LatestInformationModifiedDate": null,
"ProjectRoles": "Client / Promoter#Client",
"RoleOfficeIdsWithRole": "50657747#Client / Promoter#Client",
"OSAccuracy": "A",
"TenderInformation": "Contract details are yet to be established.",
"ProjectTenderReturnDate": null,
"Structures": null,
"StartDateType": "Guideline",
"DemolitionDescription": "",
"SchemeDescription": "Scheme comprises wooden workshop for potting, storage and shelter (6 x 3m). Secure metal tool shed (3 x 1.5m). Non-glass green house (3 x 3m). All for woodland management, propagation and increasing biodiversity.",
"RoleLocationsLevel3WithRole": "South East#Berkshire#Reading#Client / Promoter#Client",
"EndDate": "2024-11-22T00:00:00",
"LatestEventDate": "2024-01-26T04:52:24",
"IsMasterLead": false,
"IsArchived": "false",
"LiveHistoric": "L",
"RoleLocationsLevel2WithRoleParents": "South East#Berkshire#Client / Promoter",
"ProjectStatus": "In Progress",
"Storeys": 0,
"Length": null,
"ElectricVehicleChargingPoints": null,
"SecondaryHeading": "Digberry Lane, Henley-On-Thames",
"PrimaryHeading": "Agricultural Storage Shed",
"SiteName": "Digberry Lane, Henley-On-Thames",
"SiteArea": 0.003,
"ProjectOfficeIds": "50657747",
"CshRating": null,
"ContractStage": "Pre-Tender#Pre-Tender",
"NewsArticles": ,
"DevelopmentType": "New",
"ProjectLocationLevel2Facet": "South East#Oxfordshire",
"MaterialsChildren": "",
"KeyCard_MaterialsWithCodes": "",
"OSEasting8": "04691330",
"StartDate": "2024-08-29T00:00:00",
"MaterialParents": "",
"ShlaaStartYear": null,
"PrimarySectorParents": "Industrial",
"MasterLeadOf": ,
"PlanningTypes": "Planning not required",
"CivilDescription": null,
"Client": "Mr. Stuart Bell~50657747",
"OfficeNames": "Mr. Stuart Bell",
"RolesDetails": [
{
"RoleGroupDescription": "Client / Promoter",
"Roles": [
{
"RoleDescription": "Client",
"DateAdded": "2024-01-26T04:19:27",
"CompaniesInRole": [
{
"StartDate": null,
"ContractStageDescription": "Contract Awarded",
"ContractProcurementType": null,
"NoLongerInvolved": "false",
"ContactsInCompanyInRole": ,
"Value": null,
"BusinessType": "Private Client",
"ContractType": null,
"HeadOfficeId": "50657747",
"OfficeId": "50657747"
}
]
}
]
}
],
"Updated": null,
"SecondarySectors": "Industrial#Workshops",
"RoleList": [
{
"LocationLevel2": "South East#Berkshire",
"LocationLevel1": "South East",
"ContractProcurementType": null,
"LocationLevel3": "South East#Berkshire#Reading",
"NoLongerInvolved": "false",
"BusinessType": "Private Client",
"KeyCard_ContractTenderDueDate": null,
"OfficeType": "Head Office",
"ContractStageDescription": "Contract Awarded",
"StartDate": null,
"KeyCard_ContractCountryCode": "UK",
"KeyCard_ContractTownCode": "RG07",
"OfficeIds": [
"50657747"
],
"KeyCard_ContractOfficeBusinessCode": "631",
"KeyCard_ContractPhone1": null,
"PostCodeDistrict": "RG4",
"KeyCard_ContractPhone1ShowTps": " (CTPS)",
"KeyCard_ContractWinnerOfficeId": "50657747",
"KeyCard_ContractPhone1WithTps": " TPS Screened",
"HeadOfficeId": "50657747",
"OfficeId": "50657747",
"Role": "Client",
"KeyCard_ContractId": "42086852",
"KeyCard_ContractStreetAddress": "Orchard Cottage, Tagg Lane",
"RoleParentRole": "Client / Promoter#Client",
"KeyCard_ContractPhone1NoTps": null,
"Eircode": null,
"RoleParent": "Client / Promoter",
"KeyCard_ContractRoleCode": "100",
"Value": null,
"ContractType": null,
"KeyCard_ContractOfficeId": "50657747",
"KeyCard_CountyCode": "22",
"PostCode": "RG4 9PR",
"DateAdded": "2024-01-26T04:19:27"
}
],
"Ceequal": {
"InterimGrade": null,
"Version": null,
"InterimPercentage": null,
"FinalGrade": null,
"Code": null,
"AwardType": null,
"ForecastFinalVerification": null,
"FinalPercentage": null
},
"SitePhoneNumber": "",
"Units": 1,
"ContractPeriod": 2,
"LocationSource": "Council - Application Form",
"PrimaryAndSecondarySectorParents": "Industrial~Industrial",
"RoleOfficeIds": "50657747",
"ProjectType": "Construction and Civil",
"Value": 0,
"ContractType": "",
"ProjectMaterials": "",
"ProjectLocationLevel3Facet": "South East#Oxfordshire#Henley-On-Thames",
"ChangeOfUseFrom": null,
"KeyCard_ContractStageDescription": "Pre-Tender",
"LatestInformation": null,
"FloorArea": 0,
"PrimarySectors": "Industrial#Farm Buildings",
"ProjectLandTypes": ,
"LatestEvent": "New role added",
"ShlaaReference": null,
"BedroomsNumber": null,
"VerticalHeight": 0,
"BasementParkingSpaces": null,
"RoleLocationsLevel3WithRoleParents": "South East#Berkshire#Reading#Client / Promoter",
"RoleLocationsLevel1": "South East",
"RoleLocationsLevel2": "South East#Berkshire",
"RoleLocationsLevel3": "South East#Berkshire#Reading",
"ConstructionPeriodDescription": "2 months.",
"PlanningStage": "Plans Approved#Planning Not Required",
"OSNorthing8": "01883420",
"ParkingSpaces": 0,
"ProjectLocationLevel1Facet": "South East",
"RoleLocationsLevel1WithRole": "South East#Client / Promoter#Client",
"SecondarySectorParents": "Industrial",
"Applications": [
{
"FloorArea": null,
"Heading": "Agricultural Storage Shed",
"WithdrawnDate": null,
"HasPlansProposed": false,
"CommitteeDate": null,
"CouncilUrl": "Applications Reference Search",
"IsLead": "true",
"HasPlansExisting": false,
"ApplicationDate": "2024-01-11T00:00:00",
"URL": "http://www.southoxon.gov.uk/ccm/support/Main.jsp?MODULE=ApplicationDetails&REF=P24/S0140/AG",
"PlanningStageDescription": null,
"PermissionExpiryDate": null,
"ApplicationNumber": "P24/S0140/AG",
"PlanningTypeDescription": "Planning not required",
"HasStatements": false,
"TargetDecisionDate": "2024-02-16T00:00:00",
"HasDesignAndAccessStatements": false,
"CouncilName": "South Oxfordshire",
"OfficeId": "50657747",
"IsLatest": false,
"RefusedDate": null,
"ApplicationDescription": "Scheme comprises wooden workshop for potting, storage and shelter (6 x 3m). Secure metal tool shed (3 x 1.5m). Non-glass green house (3 x 3m). All for woodland management, propagation and increasing biodiversity.",
"Units": 1,
"ApplicationType": "Prior approval",
"ReceivedDate": "2024-01-11T00:00:00",
"PlanningRecommendation": null,
"HasApplicationForms": false,
"CapturedDate": "2024-01-23T00:00:00",
"ProposalId": "20263591",
"PermissionDate": null,
"AppealDate": null,
"ApplicationDocs": {
"PlansExisting": ,
"ApplicationForms": ,
"Statements": ,
"PlansProposed": ,
"DesignandAccessStatements":
}
}
],
"ValueType": "Calculated",
"Bidders": ,
"ProjectId": "24024743",
"ShlaaEndYear": null,
"UndergroundStoreys": 0,
"RoleLocationsLevel1WithRoleParents": "South East#Client / Promoter",
"OJEU": ,
"ProjectPostcode": "RG9 6DP",
"RoleOfficeIdsWithRoleParents": "50657747#Client / Promoter",
"ProjectHistories": [
{
"ProjectHistoryChange": "New role added",
"ProjectHistoryModified": "2024-01-26T04:52:24",
"ProjectHistoryOldValue": null,
"ProjectHistoryFieldName": "Contract",
"ProjectHistoryModifiedBy": "Update Opus",
"ProjectHistoryNewValue": "Client",
"ProjectHistoryEvent": "NROL"
},
{
"ProjectHistoryChange": "Planning stage change",
"ProjectHistoryModified": "2024-01-26T04:52:24",
"ProjectHistoryOldValue": null,
"ProjectHistoryFieldName": "Planning Stage",
"ProjectHistoryModifiedBy": "Update Opus",
"ProjectHistoryNewValue": "Planning Not Required",
"ProjectHistoryEvent": "PSTG"
},
{
"ProjectHistoryChange": "New Project",
"ProjectHistoryModified": "2024-01-26T04:52:24",
"ProjectHistoryOldValue": null,
"ProjectHistoryFieldName": "ProjectId",
"ProjectHistoryModifiedBy": "Update Opus",
"ProjectHistoryNewValue": "24024743",
"ProjectHistoryEvent": "NewProject"
},
{
"ProjectHistoryChange": "Contract stage change",
"ProjectHistoryModified": "2024-01-26T04:52:24",
"ProjectHistoryOldValue": null,
"ProjectHistoryFieldName": "ContractStage",
"ProjectHistoryModifiedBy": "Update Opus",
"ProjectHistoryNewValue": "Pre-Tender",
"ProjectHistoryEvent": "CNSTG"
}
],
"ProjectRegion": "South East",
"ProjectSize": "Small",
"ProjectPlanningAuthorities": "South Oxfordshire"
}
}
]
}

leandrojmp · February 14, 2024, 4:08pm

Not sure what is the issue here, in the response you shared the results array has only one item.

What do you want to have as individual documents? Maybe you will need to split on another field, since your document has multiple array fields.

Can you provide more context?

rajatbhardwaj1393 · February 14, 2024, 4:13pm

The response i share have only one record but you can take it as array of records in results. Due to message constraint here was not able to send the multiple record one.

leandrojmp · February 14, 2024, 4:21pm

Without a working example is not possible to know what is the issue, you may post it on a gist on github if you want.

You can to troubleshoot it yourself.

The filter I shared will do the following things:

Validate if the results filter is a array by looking if it has at least one element ([results][0])
If the conditional is true it will split the event on the results fields, this means that for each item on the results array a new document will be created.
It will create a field named [@metadata][document_id] with the value of the field [results][id], so every item needs to have this field.

rajatbhardwaj1393 · February 15, 2024, 10:37am

HI Leo,

Does try it but its coming blank:

This is a valid output JSON:

{
"total": 14939,
"results": [
{
"id": "20154282",
"source": {
"LastModifiedDate": "2024-01-26T13:50:42",
"Address": "Caherwalter, Loughrea, Co Galway",
"Funding": [
"Private"
]
}},
{
"id": "24026360",
"source": {
"LastModifiedDate": "2024-01-26T13:25:51",
"Address": "Nottingham, Nottinghamshire, NG5 1AP",
"Funding": [
"Housing Association",
"Public"
]
}
},
{
"id": "24026364",
"source": {
"LastModifiedDate": "2024-01-26T13:29:51",
"Address": "Nottingham, Nottinghamshire, NG5 1AP",
"Funding": [
"Housing Association",
"Public"
]
}
}
]
}

system · March 14, 2024, 10:37am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Retrieve data from API, split response array and handle duplicates in Logstash Logstash	1	599	September 11, 2019
Transforming array of objects Logstash	3	1867	June 29, 2020
Logstash to Elastic search, Could not index event to Elasticsearch , "reason"=>"mapper [] of different type, current_type [long], merged_type [text] Logstash	3	2779	March 17, 2019
Ruby filter - How to access some elastic index data from a ruby filter Logstash	23	3015	July 24, 2018
Logstash could not index event to Elasticsearch Logstash	3	1206	May 18, 2020

NOT ABLE TO INDEX DATA USING HTTP POLLER

Related topics