Active Incident

Updated a few seconds ago

Incident Status

Component Disruption - SEV 3

Components

Workfront Fusion

Locations

Workfront Fusion



June 2, 2020 12:29PM MDT
[Investigating] FLO and execution history are intermittently not loading. Execution does not appear to be impacted on any FLOs at this point, even where the history in either case is unavailable. Engineering is investigating the cause of the issue.

June 2, 2020 3:00PM MDT
[Investigating] Engineering is still investigating the issue.

June 2, 2020 5:20PM MDT
[Monitoring] Engineers have restarted a background processes to restore Fusion FLO history entries. Please reach out to support if you are seeing the issue persist.

June 3, 2020 8:30AM MDT
[Investigating] Engineering is still investigating the issue.

June 4, 2020 7:05PM MDT
[Monitoring] Engineers have restarted a service to resolve the missing FLO history. Please reach out to support if you are still experiencing the issue.

Workfront Application Platform




Operational

Workfront API




Operational

Workfront Event Subscription




Operational

Workfront Fusion




Component Disruption - SEV 3

Workfront Library




Operational

WorkfrontDAM




Operational

Workfront Proof




Operational

Workfront Non-Production




Operational

Customer Experience Site




Operational

External Services

Workfront Uptime

100.00%




Workfront-Proof Uptime

70.83%




Avg. Round Trip

68.00ms




Workfront Uptime

100.00%




Workfront-Proof Uptime

100.00%




Avg. Round Trip

78.72ms




Workfront Uptime

99.99%




Workfront-Proof Uptime

100.00%




Avg. Round Trip

78.00ms




Maintenance Windows North America: CL01 | CL02 | CL03 | CL05 | CL06 2nd and 4th Saturday 8:00 pm to 12:00 am MT 1st and 3rd Thursday 8:00 pm to 12:00 am MT EMEA: CL04 1st and 3rd Thursday 8:00 pm to 12:00 am GMT CL01, CL02, CL03, CL05, and CL06 Preview environments refresh on Saturday CL04 Preview environment refreshes on Friday **We infrequently schedule maintenance in the windows listed above. If down time is necessary, it will be scheduled in the corresponding window and announced on this page.**

1

Active Incidents

0

Active Maintenances

0

Upcoming Maintenances

History (Last 30 days)

Proofs failing to generate on cluster 5Component Disruption - SEV 3

Incident Status

Component Disruption - SEV 3


Components

Workfront Proof


Locations

Workfront Proof




June 4, 2020 9:43AM MDT
[Investigating] We have customer reports of proofs failing to generate on cluster 5 when the New Workfront Experience is enabled.

June 4, 2020 11:01AM MDT
[Resolved] After further investigation this issue has been identified as a software defect and will be handled through the normal defect process. A workaround is available: Navigate to Account settings in the Proof account. Find the "Proof Defaults" section and change the "Deadline" option to "-". If you have any additional concerns, please contact support.

Incident Status

Service Disruption - SEV 1


Components

Workfront Non-Production


Locations

CL04 - Preview/Sandbox




June 4, 2020 6:07AM MDT
[Investigating] Our Preview environment in cluster04 (EMEA region) is currently unavailable. Our engineers are investigating the cause of the issue.

June 4, 2020 6:32AM MDT
[Monitoring] Cl04 preview environment is available now. Our engineers currently investigating the root cause of the issue.

June 4, 2020 7:29AM MDT
[Resolved] At this time the issues involving cluster 4 preview environment have been resolved. Please contact support if you experience any additional issues.

Incident Status

Degraded Performance - SEV 2


Components

Workfront Fusion


Locations

Workfront Fusion




June 4, 2020 6:12AM MDT
[Investigating] We have discovered that Workfront Fusion is currently unavailable. Our engineers now investigating the cause of the issue.

June 4, 2020 6:31AM MDT
[Monitoring] Fusion is back now. Our engineers now investigating the root cause of the issue.

June 4, 2020 7:27AM MDT
[Resolved] At this time Fusion functionality is performing within expected parameters. If you continue to experience any issues, please contact support.

Description

We will be performing database maintenance the evening of Saturday, May 30th. The Production Workfront and Proof environments may be unavailable for up to 60 minutes while maintenance takes place. Expected Time of Maintenance: Saturday, May 30th from 10:00 PM to 12:00 Midnight Mountain Time.


Components

Workfront Application Platform, Workfront Proof


Locations

CL06, Workfront Proof GCP


Schedule

May 30, 2020 10:00PM - May 31, 2020 12:00AM MDT



May 30, 2020 10:11PM MDT
[Update] Maintenance is started.

May 30, 2020 11:13PM MDT
[Update] Maintenance completed.
SFTP Connectors failing.Component Disruption - SEV 3

Incident Status

Component Disruption - SEV 3


Components

Workfront Fusion


Locations

Workfront Fusion




May 29, 2020 8:39AM MDT
[Investigating] At this time engineers are investigating an issue with SFTP connectors within Fusion. Existing connectors fail to connect and new connectors cannot be created.

May 29, 2020 11:47AM MDT
[Identified] Engineers have identified a potential root cause and are working on the details at this time in order to resolve impact. Once complete, some SFTP connectors will need to be re-authorized. Workfront is currently working to validate the list of impacted connectors in order to resolve the issue as efficiently as possible.

May 29, 2020 1:45PM MDT
[Identified] Engineers are preparing to removed problematic data within the SFTP connectors to allow them to reconnect.

May 29, 2020 3:48PM MDT
[Resolved] At this time engineers have completed their work. SFTP connections are once again functional. If you continue to experience any issues or have any further questions please contact support.

Incident Status

Operational


Components

Workfront Application Platform, Workfront Proof


Locations

CL01, CL02, CL03, CL05, Workfront Proof




May 27, 2020 8:38AM MDT
[Investigating] Proofing functionality in Workfront is experiencing slower than normal performance. We are working to resolve the situation as quickly as possible.

May 27, 2020 8:42AM MDT
[Investigating] Proofing functionality in Workfront is experiencing slower than normal performance. We are working to resolve the situation as quickly as possible.

May 27, 2020 10:20AM MDT
[Resolved] Changes have been implemented to improve performance, and services related to proofing are back to optimal. Please reach out to our customer support teams if further issues become visible.

Description

We will be performing database maintenance the evening of Thursday, May 21st. The Production environment may be unavailable for up to 60 minutes while maintenance takes place. US Proof for clusters CL01, CL02, CL03, and CL05 will also be unavailable during maintenance for both Prod and Sandbox Proof. Expected Time of Maintenance: Thursday, May 21st from 9:00 PM to 12:00 Midnight Mountain Time.


Components

Workfront Application Platform, Workfront Proof


Locations

CL01, CL02, CL03, Workfront Proof


Schedule

May 21, 2020 9:00PM - May 22, 2020 12:00AM MDT



May 21, 2020 10:52PM MDT
[Update] The Maintenance is started.

May 21, 2020 11:35PM MDT
[Update] The Maintenance has been completed.

Description

The deployment of the 20.2 release for Clusters 1, 3, & 5 will start on Monday May 18 at 8:00 p.m., MDT. The deployment is expected to take no longer than 4 hours. No downtime is expected and all clusters and components will be available during deployment. For more detailed information about the functionality included with this release, see the 20.2 release page (https://one.workfront.com/s/article/2020-2-release-overview-439776658?language=en_US)


Components

Workfront Application Platform


Locations

CL01, CL03, CL05


Schedule

May 18, 2020 8:00PM - May 18, 2020 11:59PM MDT



May 14, 2020 6:43PM MDT
[Update] The 20.2 release originally scheduled for tonight has been delayed. Further details and an updated status will be posted here by 1:00 PM Mountain Time tomorrow, May 15th.

May 15, 2020 12:58PM MDT
[Update] The problem we encountered yesterday is still being evaluated. We will post a further update by 5:00 PM MDT

May 15, 2020 4:29PM MDT
[Update] The problem we encountered that caused us to delay the 20.2 release has been resolved. The release will now be deployed on Monday 5/18 starting at 8:00 pm MDT. The release deployment is expected to take no longer than 4 hours, and no downtime is expected during the release deployment. All features will be released as planned except for the option to manage user allocations in the Workload Balancer, which will be rolled out in a phased approach starting on Tuesday 5/26. See the 20.2 release page for more details (https://one.workfront.com/s/article/2020-2-release-overview-439776658?language=en_US).

May 18, 2020 8:03PM MDT
[Update] Beginning maintenance.

May 18, 2020 11:53PM MDT
[Update] Maintenance complete.

May 18, 2020 11:53PM MDT
[Update] Maintenance complete.

Description

The deployment of the 20.2 release for Workfront Proof be on Monday May 11. The Proof EU instance will begin at 2:00 p.m., MDT/8:00 p.m., GMT. The Proof US instance will begin at 11:00 p.m., MDT. The deployment of each cluster is expected to take no longer than 4 hours. No downtime is expected and all clusters and components will be available during deployment. The 20.2 Workfront Proof release includes only back-end improvements and fixes, and no new customer-facing features. For more detailed information about the functionality included with this release, see the 20.2 release page (https://one.workfront.com/s/article/2020-2-release-overview-439776658?language=en_US).


Components

Workfront Proof


Locations

Workfront Proof, Workfront Proof GCP


Schedule

May 11, 2020 2:00PM - May 12, 2020 3:00AM MDT



May 14, 2020 5:49PM MDT
[Update] Proof maintenance has been completed.

May 14, 2020 5:49PM MDT
[Update] Proof maintenance has been completed.

Description

The deployment of the 20.2 release for Cluster 4 will start on Wednesday May 13 at 2:00 p.m., MDT/8:00 p,m., GMT. The deployment is expected to take no longer than 4 hours. No downtime is expected and all clusters and components will be available during deployment. For more detailed information about the functionality included with this release, see the 20.2 release page (https://one.workfront.com/s/article/2020-2-release-overview-439776658?language=en_US)


Components

Workfront Application Platform


Locations

CL04


Schedule

May 13, 2020 2:00PM - May 13, 2020 6:00PM MDT



May 14, 2020 5:49PM MDT
[Update] Maintenance for 20.2 has been completed.

May 14, 2020 5:49PM MDT
[Update] Maintenance for 20.2 has been completed.
Cluster 2 Is Inaccessible Service Disruption - SEV 1

Incident Status

Service Disruption - SEV 1


Components

Workfront Application Platform


Locations

CL02




May 14, 2020 1:04PM MDT
[Investigating] We have internally detected slowness in cluster 2 and we are investigating

May 14, 2020 1:10PM MDT
[Investigating] We are taking maintenance activity to try and solve the issue. Cluster 2 may be unavailable for 5 to 10 minutes.

May 14, 2020 1:23PM MDT
[Monitoring] We are continuing to investigate the issue and have engaged engineering to identify and solve the root problem.

May 14, 2020 1:37PM MDT
[Monitoring] Engineers have been engaged and are investigating the underlying cause of cluster 2 slowness. At this time, Workfront requests seem to be working regularly.

May 14, 2020 1:50PM MDT
[Monitoring] Engineering has disabled the Work Per Day functionality. We are continuing to monitor the application for slowness.

May 14, 2020 1:53PM MDT
[Investigating] We are seeing slowness return to the cluster. Engineering is performing a cluster restart, applications may be unreachable for 5-10 minutes.

May 14, 2020 2:04PM MDT
[Investigating] The cluster restart has completed, engineering is continuing to investigate the issues that customers are experiencing.

May 14, 2020 2:21PM MDT
[Monitoring] Engineers are continuing to investigate the underlying cause of slowness.

May 14, 2020 2:39PM MDT
[Monitoring] We are continuing to monitor the situation as we investigate and fix the root cause of this issue.

May 14, 2020 3:39PM MDT
[Monitoring] User sessions are being terminated on cl02 prod to assist in the troubleshooting process. Users with active sessions will need to log back in as of now, but any new login attempts from this point should be unaffected.

May 14, 2020 5:02PM MDT
[Resolved] Monitoring over the last few hours confirms that the incident has been resolved and that performance has been returned to optimal. Please reach out to our support teams if performance on cl02 prod experiences additional impact.
Event Subscriptions experiencing delays. Component Disruption - SEV 3

Incident Status

Component Disruption - SEV 3


Components

Workfront Application Platform, Workfront Event Subscription


Locations

CL01, CL02, CL03, CL05, Workfront Event Subscription




May 14, 2020 8:44AM MDT
[Investigating] Some portions of the event subscription service are delayed resulting in some events not being received in a timely fashion. This is being investigated by engineering at this time.

May 14, 2020 11:15AM MDT
[Monitoring] The issues with event subscription delays have been resolved. As a part of this resolution, historical messages are being replayed to ensure no data loss by event subscription users. This process has begun but will take additional time to process. We expect real time performance to resume by 2:00 PM MST.

May 14, 2020 11:42AM MDT
[Monitoring] Additional investigation has identified a bottleneck in processing which will require an additional maintenance action to correct. Engineers will be correcting this bottleneck, however this will require beginning the reprocessing behavior once more and.

May 14, 2020 4:58PM MDT
[Resolved] Event Subscriptions is currently functioning as expected. If you continue to see any issues, please contact support.

Description

The deployment of the 20.2 release for Clusters 2 & 6 will start on Wednesday May 13 at 8:00 p.m., MDT. The deployment is expected to take no longer than 4 hours. No downtime is expected and all clusters and components will be available during deployment. For more detailed information about the functionality included with this release, see the 20.2 release page (https://one.workfront.com/s/article/2020-2-release-overview-439776658?language=en_US)


Components

Workfront Application Platform


Locations

CL02, CL06


Schedule

May 13, 2020 8:00PM - May 13, 2020 11:59PM MDT



May 13, 2020 8:00PM MDT
[Update] Maintenance is starting for CL02 and CL06

May 14, 2020 8:42AM MDT
[Update] This maintenance has been concluded.
Fusion FLO performance degradation Component Disruption - SEV 3

Incident Status

Component Disruption - SEV 3


Components

Workfront Application Platform, Workfront Fusion


Locations

CL01, CL02, CL03, CL04, CL05, CL06, Workfront Fusion




May 7, 2020 8:38PM MDT
[Investigating] We are experiencing a slowness in executing Fusion FLOs and are investigating the root case. FLO executions may take an additional 2 minutes to fully execute.

May 8, 2020 3:31AM MDT
[Monitoring] The problem is solved at 11:50 MDT All FLOs running in normal speed. We will continue monitoring the system.

May 8, 2020 3:56AM MDT
[Resolved] The problem is completely solved. All services up and operational.

May 8, 2020 4:02AM MDT
[Resolved] The problem is completely solved. All services up and operational.

May 8, 2020 12:20PM MDT
[Investigating] Upon further investigation of server logs and existing Fusion FLOs, the incident is still ongoing, although inconsistently. Engineering teams are working to identify the cause of degraded performance, which may add up to 10 minutes to execution time in some cases.

May 13, 2020 2:25PM MDT
[Resolved] At this time Fusion functionality is performing within expected parameters. If you continue to experience any issues, please contact support.
Problem on proofing environmentDegraded Performance - SEV 2

Incident Status

Degraded Performance - SEV 2


Components

Workfront Application Platform, Workfront Proof


Locations

CL03, CL04, CL05, Workfront Proof




May 13, 2020 6:09AM MDT
[Identified] W've discovered that proofs not available on some prod environments. We found the root cause and our engineers are now applying solution.

May 13, 2020 6:24AM MDT
[Identified] US production environments are restored. We are working now to restore Europe environment.

May 13, 2020 6:32AM MDT
[Resolved] The fix is applied and all environments are now operational.

Incident Status

Component Disruption - SEV 3


Components

Workfront Event Subscription, Workfront Non-Production


Locations

Workfront Event Subscription, CL01 - Preview/Sandbox, CL02 - Preview/Sandbox, CL03 - Preview/Sandbox, CL05 - Preview/Sandbox




May 11, 2020 8:52AM MDT
[Investigating] Engineering teams have discovered that event subscriptions for preview environments on clusters 1, 2, 3 and 5 are down. We are investigating the cause and will provide updates along the way.

May 11, 2020 10:32AM MDT
[Monitoring] Event subscription services have been restored for US preview environments. We are resending messages from 12:00 AM MST of May 9, 2020 to make sure no data is lost, and will be monitoring the process for several hours to make sure it continues to make progress. It will likely be several hours before event subs in these environments are caught up again.

May 11, 2020 1:06PM MDT
[Monitoring] Messages are caught up now, and new requests are flowing optimally again for US preview environments. We'll continue monitoring for a few more hours to make sure performance is consistent.

May 11, 2020 4:18PM MDT
[Resolved] Event sub performance for US preview environments has been consistently optimal since the issue was addressed earlier today, so the issue is considered resolved. Please reach out to our support teams if any lingering issues with this functionality are found.
Proofing for HTML5.zip files errorsComponent Disruption - SEV 3

Incident Status

Component Disruption - SEV 3


Components

Workfront Proof


Locations

Workfront Proof




May 11, 2020 11:57AM MDT
[Investigating] We have received reports of proofs not generating for HTML5 zip files. We have engaged engineering and are investigating.

May 11, 2020 12:39PM MDT
[Monitoring] At this time the issues involving HTML5 zip uploads have been resolved. Please contact support if you experience any additional issues.

May 11, 2020 1:01PM MDT
[Resolved] After further investigation this issue has been identified as an infrastructure defect and will be handled through the normal defect process. If you have any additional concerns, please contact support.

Incident Status

Operational


Components

Workfront Application Platform


Locations

CL02




May 8, 2020 1:37PM MDT
[Investigating] We have received reports of slowness on cluster 2

May 8, 2020 1:38PM MDT
[Investigating] Application slowness is being reported

May 8, 2020 1:55PM MDT
[Investigating] Engineers have been engaged and are investigating the underlying cause of slowness on cluster 2.

May 8, 2020 2:10PM MDT
[Monitoring] We will continue to monitor the situation.

May 8, 2020 2:30PM MDT
[Resolved] At this time the issues involving cluster 2 have been resolved. Please contact support if you experience any additional issues.

Description

We will be performing database maintenance the evening of Thursday, May 7th. The CL04 /EU Proof environment may be unavailable for up to 60 minutes while maintenance takes place. US Proof will not be affected. Expected Time of Maintenance: Thursday, May 7th from 8:00 PM to 12:00 Midnight GMT (2:00 PM to 6:00 PM Mountain)


Components

Workfront Proof


Locations

Workfront Proof


Schedule

May 7, 2020 2:00PM - May 7, 2020 6:00PM MDT



May 7, 2020 2:02PM MDT
[Update] Beginning Maintenance. Application will be unavailable for the next few minutes.

May 7, 2020 2:16PM MDT
[Update] Maintenance complete.