Servers Australia Engineers have identified that specific database queries executed by the vCloud Director system during inventory collection were causing the sync status between vCloud director and vCenter to enter an invalid state, causing the vCenter to continually try to resync and never complete. This caused specific objects from vCenter to be unable to be collected into the VCD database, as well as affecting creating of new VMs while the sync status was not healthy. This issue was traced back to a bug in the error handling code within the vCloud Director application itself, which made the root cause difficult to pinpoint.
To address this, we have implemented changes to mitigate the issue and restore functionality. We will continue to monitor the situation closely to ensure the stability of the system before formally closing the status.
We appreciate your patience and will provide further updates as needed.
Thank you, SAU Engineering Team
Posted Feb 11, 2025 - 09:32 AEDT
Update
We are continuing to work with the Vendor on this issue.
As part of the process, we need to export a log bundle from the vCloud Director server.
Doing this may cause the VDC interface to become unresponsive while the log is generated.
There will be no impact to customer workloads while this takes place.
Posted Feb 10, 2025 - 14:31 AEDT
Investigating
The VCD platform is back online. We are continuing to work with the software vendor to resolve intermittent issues in the UI. We will keep this status updated as work progresses and with further maintenance.
Posted Feb 06, 2025 - 04:59 AEDT
Identified
The VCD user interface will be temporarily unavailable while we perform emergency maintenance and updates. Engineers will update this status once complete.
Posted Feb 06, 2025 - 03:39 AEDT
Investigating
We are once again investigating an issue with vCloud Director that causes some operations within the user interface to fail.
Our Systems Engineering and Networking teams are actively investigating this issue in collaboration with the software vendor and their support teams, maintaining constant communication throughout the process.
We will provide updates as they become available.
Posted Feb 05, 2025 - 09:50 AEDT
Monitoring
Maintenance is complete and all services are back online. We will continue monitoring and will update this status further if there are any more changes.
Posted Feb 05, 2025 - 01:50 AEDT
Identified
Engineers have found that certain API calls to the backend from the vCloud Director interface are timing out. Emergency maintenance will start shortly and the vCloud Director user interface will be unavailable during this time. Estimated time is 30 minutes.
Posted Feb 05, 2025 - 00:30 AEDT
Monitoring
Our Systems Engineers have rectified issues that were causing certain operations to fail within the VCD platform.
We have tested functionality and can no longer replicate previous issues seen. Our Engineers will continue to monitor the VCD platform throughout the night for any irregularities.
Please reach out to our support team if you encounter any issues with your VDC or Private Cloud environment.
Posted Feb 04, 2025 - 16:58 AEDT
Identified
We have identified an issue with the hardware hosting the vCenter appliance. The appliance is being moved to a new host and will be unavailable for approximately 10 minutes. This will only impact the availability of the vCenter WebUI and not customer workloads.
Posted Feb 04, 2025 - 16:12 AEDT
Update
We are once again investigating an issue with vCloud Director that causes some operations within the user interface to fail.
We will provide updates as they become available.
Posted Feb 04, 2025 - 09:57 AEDT
Monitoring
After a reboot of the VCD environment, operations have returned to normal.
We are monitoring the services and will update as needed.
Posted Feb 04, 2025 - 01:04 AEDT
Investigating
We are currently investigating an intermittent issue with vCloud Director that causes some operations within the vCloud Director user interface to fail.
We need to restart the vCenter Management Server and the vCloud Director Server, There will be no customer workload impact, but the VCS and VCD User Interfaces will be unavailable for 15 minutes.
This work is commencing now.
Posted Feb 04, 2025 - 00:49 AEDT
Monitoring
After a reboot of the VCS environment, operations have returned to normal.
We are monitoring the services and will update as needed.
Posted Feb 03, 2025 - 16:57 AEDT
Update
As part of the normal troubleshooting procedure, we need to restart the vCenter Management Server.
There will be no customer workload impact, but the vCenter UI will be unavailable for approximately 10 minutes.
This work is being performed at 16:25 AEDT.
Posted Feb 03, 2025 - 16:24 AEDT
Investigating
We are currently investigating an intermittent issue with vCloud Director that causes some operations within the user interface to fail.
We will provide updates as they become available.
Posted Feb 03, 2025 - 15:22 AEDT
This incident affected: Services (Virtual Data Centre (VDC)).