Service Incidents

Issues sending emails to Microsoft domains - 5th May 2023

Updated 8th. May 2023.

It has come to our attention that Microsoft mail servers are delaying, and in some cases failing to deliver, email messages (including survey invitations) sent from our REDCap systems to Microsoft owned domains such as hotmail.com, live.com, and outlook.com. We are working with the University of Alberta's central IT services in order to resolve the issue but at this time it is largely outside of our control.

Today, 8th May we were informed that emails to Microsoft domain's are now being sent correctly.

Critical Security Vulnerabilities Discovered and Resolved - 19th April to 5th May 2023

Mid-afternoon on April 19th we received a notification from the REDCap team at Vanderbilt that critical security vulnerabilities had been discovered in all current releases of REDCap. In order to protect the integrity of the data and the privacy of our study participants we took the decision to take our REDCap servers offline and to perform emergency upgrades to install security patches provided by Vanderbilt. We took both of our REDCap systems offline at approximately 3:30 pm. Both systems were upgraded to the latest REDCap release. The MICYRN install was back online at 7:40 pm and our Standard install was back online at 8:15 pm.

In the following two weeks, additional vulnerabilities have been discovered and we have continued to patch our systems as each fix becomes available.

What this means for your study

As always when REDCap is offline, survey invitations may be delayed and survey participants cannot complete surveys during that time. Generally, any delayed survey invitations are sent as soon as REDCap is back online. However, this is dependent on the logic used to configure each individual survey invitation.

As always with a new release, there may be changes to REDCap features and functionality. In this specific release there are bug-fixes that may change the way survey invitations may be handled if a survey has been completed before the invitation is sent. If you are using automated survey invitations in your study and are also completing one or more survey instruments as a logged-in user (rather than as a survey participant) you should check how invitations and reminders are handled in your project.

We do not believe that the security or integrity of any study data has been compromised.

For Health Canada and other regulated clinical trials

The MICYRN installation of REDCap is validated for use in Health Canada regulated clinical trials. However, validation is a long process that requires both planning and effort. Given the sudden appearance and the severity of the vulnerabilities, and on consultation with senior staff members, we made the decision that the security of the data would take priority over and above the validated state of the system. Indeed, failure to deal with a possible security issue could also be considered a compliance issue. We consider this upgrade to be low risk and will be validating this release retrospectively.

Documentation for this emergency release is available here.
We recommend that study teams confirm (validate) that their REDCap projects continue to operate as expected. You may do this by entering test participants and completing a database testing form. This form is available here. (Select File->Download from the menu.)
Study teams that use the WCHRI DCC may contact the helpdesk for additional information and support.

REDCap Outage - 18th April 2023

At approximately 3:40 PM on April 18th. we suffered a failure in a storage device in the Faculty's secure data centre. This resulted in the database servers for both of our REDCap systems being unavailable. The problem was diagnosed and rectified by the University's central IT group. Normal REDCap service was resumed at approximately 10:00 pm.

What this means for your study

Apart from regular users not being able to login and enter data, the system was unavailable to survey participants. Additionally, depending on individual project configurations, some automated survey invitations may have been delayed or missed. If your study employs automated survey invitations you may wish to check your survey invitation log. Any surveys that were missed can be sent manually.

The integrity of existing data was not affected.

REDCap Service Incident - 22nd January 2023

Early in the evening of January 22nd. the server running the database that serves our main REDCap installation suffered from a storage failure. As a result, the system was offline from approximately 5:10 pm until midnight, when the fault was resolved.

What this means for your study

REDCap Service Incident - 28th October 2020

It has come to our attention that overnight on October 28th 2020 the University of Alberta's central mail servers encountered a problem that led to them rejecting some emails from our REDCap servers. The rejected emails were those to recipients with external (not ualberta.ca) email addresses. Due to the nature of the error and the way REDCap delivers emails, these emails were lost. Having identified the problem we notified IST (our central computer services group) and the problem was resolved at approximately 1:45 pm on October 29th.

What this means for your study

Survey invitations, sent overnight on the 28th and before 1:45 pm on the 29th, may not have been delivered to external email addresses.
As far as REDCap is concerned the message was sent. Unfortunately, due to the way email systems and REDCap work, REDCap has no way of knowing if an email message has been delivered.
If you are using automated survey invitations (ASIs) that include reminders REDCap will send the reminders as scheduled.
If you are using ASIs that do not include reminders, then you may need to resend survey invitations manually.
If your surveys were sent using the Survey Distribution Tool, then you can manually send a new invitation to those participants who have not yet responded. (This can be achieved in a single operation and you can, if you want, select only those participants with an external email address.)

Questions?

If you have further questions please email our helpdesk at redcap@ualberta.ca