Best Practices for Notifying Customers
Sending the right message to your customers is important. Having clear concise messages created in advance of an incident that simplify complex messaging, improves communication during chaos. This is useful before, during, and after an incident as well as for routine and on-going communication. If you have to cause an outage due to a maintenance upgrade, explain the positives of why (new features x,y,z) and not solely the fact that services will be unavailable during the outage. Now is your time to market to your customers. Don't leave it to your engineers to craft a message at the last minute while they are under pressure. Instead, you should be prepared well in advance and Constant Status is here to help you do that.
Guidelines to Follow
In general, IT is a no-news-is-good-news operation. Use the little bit of contact you have with your customer base to show them that you’re being proactive and provide them enough detail (in layman’s terms) to keep them from feeling like you’re hiding something or keeping them in the dark.
Why Notify Customers?
While it is never a positive when something goes wrong, especially when it results in a software application or service not being available, there is a way to reduce the damage. We all know the proverbial phrase “To err is human; to forgive, divine.” Isn’t it easier to forgive someone if they notified you in advance that something could happen during a specific time period? Therefore, it is much better if the users of the software or service know ahead of time that the service could have issues when a software/hardware release or maintenance window is planned. A recent study has found that over 60% of organizations have found major software errors on production. Therefore, the odds are that something will eventually go wrong and you will be happy you notified your users/customers ahead of time, because everyone knows being “forewarned is forearmed”. Here are a few other things to consider about notifying customers:
- By notifying your customers of changes to the production system, you are working with them as a partner and not as a vendor. If something does go wrong, they are on heightened alert and can report issues sooner, as well have an idea why something changed or is no longer functioning properly. If they are made aware of the system changes that occur, this leads to a bond of trust with your customers over time.
- Did you ever think that this is also a good time to 'market' the changes or new features? Customers feel like they are getting more for their money if they are kept updated all throughout their experience with the company. This shows an evolution and improvement of the product over time, that the system is being properly maintained, and it gives reassurance to your customers that they are getting their money's worth. This is especially true for software as a service (SaaS) websites.
- If the changes also require adjustments/changes on the customers side, then the communication between you and the customer becomes even more important. Customers need to be made aware that changes are being planned and how the schedule will affect the customer.
- While communication must take place within the implementation team to agree on plans, schedule dates, and so on, it is also important to communicate externally and inform the user community of the new process as well as its benefits to them. The implementation of a process can be seen as being a change just like the upgrading of a server, and the impact on the user community should be communicated to them clearly in advance of the change. If not taken by surprise, the customer will more readily embrace it.
- Often, software as a service (SaaS) websites and software providers don't realize that customers create special training materials and operating procedures for their teams to follow. So when the user interface changes, all their documentation and workflows must be updated as well. Give your customers time to prepare for these changes by letting them know in advance.
- While some companies have embraced the approach of continuous deployments and fully automated deployments, the reasons above are still valid.
Key Elements for Notifications
The diagram below begins to break down the key elements for notifications. The following sections will explain each of these areas and give further insight into the notification process. While these items may vary by the business or product, there are many common themes that can be applied regardless of the notification recipients.
Who to Notify?
Now that we know why we are sending notifications, it is important to send them to the correct recipients. Often times it is not obvious who to notify about a change. Sometimes those doing the deployments or updates have a limited understanding of the changes, the ripple effect they can have on the various system components, and ultimately the end user effected by the change. As a result the individual sending the notification is usually too conservative and afraid to notify users of changes that may not be needed. Furthermore, they often treat internal “ customers” differently than paying customers and are less likely to notify them. All of this is counter to what you really want to see happen. For all of the aforementioned reasons about why it's important to send notifications, there really are not many negatives in notifying a customer, unless you are sending too many notifications. This is highly doubtful unless you are sending notices more frequently than on a daily basis. Keep in mind that all customers have a cost when an outage occurs, so make sure everyone is notified including the following:
- If a customer is paying for your service, it is to your benefit to inform them of changes and they should be at the top of your list.
- Anyone that is a stakeholder in the system should be notified of a change, maintenance, or an outage. This should include officers of the company, sales or account managers, or anyone that could hear from customers if there is an issue.
- Customer service teams that will receive calls from users and be asked questions about the changes or a new feature.
- Anyone that has specifically requested changes or new features, or that will benefit from the changes being made should be considered for notifications.
- While many systems and applications are just used by internal users, it is important to let them know when there is a maintenance window planned for a hardware or software change.
- Any user that can experience a “momentary” outage should be notified of a possible outage, even if it is for a few seconds. While many engineers think it is ok to only affect a few users, it can actually cost company time, money in lost work, data loss, and frustrates end users. This is often the case when a device or system needs to be failed over to its backup device or system because there is a momentary interruption during the failover. The same can happen with the simple reboot of a device as well.
- After hours changes when “nobody is around” are not always the case today when society has become a 24/7 connected and global society. There is no longer an acceptable down time for users and therefore it is more important to send out notices to all users regardless of the time of day of the maintenance.
- If your product requires an administrator, at a minimum they may prefer if communication is sent only to them and not the end users. For products that have many subscribed users, the users may want to take advantage of any new features or bug fixes after a release. This all depends on your product offering and how it is administered.
When to Notify?
By sending users notices ahead of time it allows your users to prepare to be doing something else during this time. They will know the situation is under control and the outage will be complete within the given window. In order to do this you need to send out a notification ahead of time, and have it be a one of a series of reminders as the event gets closer in addition to sending notices during the event. What is recommended and required by many large companies is a reminder schedule for planned changes or maintenance windows that looks like the following:
- 5 business days in advance send out a notification
- 1 business day in advance send out a reminder notification
- 15-30 minutes before the start send out a final reminder notice
- At the start of the scheduled time
- At the completion of the changes
The above assumes that everything goes perfectly on schedule, but the following are the most important notices:
- A common issue is not knowing you are going to be on schedule for a release and it sometimes comes down to the night before or even hours before it starts. Regardless notices should be sent out on schedule and a notice can always be sent out later that the release or change has been delayed or canceled. It is more important that users have the prior notice to plan ahead versus not sending a notice and the release or change happens and users are caught off-guard.
- During the deployment of the release or change, notify users if the update is delayed or cancelled so that it gives people time to prepare or do something else with their time. The more advanced the notice, the less the impact on the users. It also makes sense to provide progress updates if it is an extended outage window. This can be based on completed tasks or steps, based on time elapsed, or based on the percentage of the time window such as 25%, 50%, 75%.
- If you know the deployment is going to be longer than planned, even if it is a couple of minutes, let people know as soon as possible. Keep in mind you never want someone asking the status; you need to always notify them of the status before then.
- If a previous notice was sent indicating that the completion date/time was extended or delayed, you should send out another notice and reset expectations once again.
- Giving advanced notice is important, but having to do a quick fix to resolve an issue that was caused in a deployment or just recently discovered can’t wait until the next release. In these cases a notice should be sent as the issue is discovered to notify users of the issue, include that it is being worked on, and that a change or fix will be made as soon as it is ready and a notification will be sent out at that time. The goal is to proactively communicate the issue before the customer reports it to you, so they are assured you have the situation under control.
- Some companies will ask for as much as 8 weeks advance notification for connectivity changes, datacenter moves, and major changes. Keep this in mind if the changes could have a large impact on your customers or require the customers to make changes on their side.
- Major changes such as discontinuing support for a major feature or an entire product could require up to a year of advanced notice. The more notice given the easier it is to accept and make the appropriate plans.
- Publish a rough schedule of upcoming releases or system changes ahead of time so your customers know in a general sense of how often to expect changes and when they need to prepare to make any changes themselves including changes to operating procedures and any related user training documentation.
- Having a regularly scheduled maintenance window whether weekly, biweekly, or monthly should be agreed upon in advance for changes. These still require notifications that need to include if the window will be used and the changes that will occur for that window.
Thus far, we have been discussing sending notices for planned changes or updates, but a critical time for notifications is for unplanned outages or issues. The goal in all unplanned outages is to get the notice out as quickly as possible and notify the customer or user before they notify you. Keep sending updates regularly until it is resolved as your customers will appreciate it. The natural tendency is to wait until the situation is fully assessed and a cause is known. However this usually takes much longer than expected and customers often report the issue and expect an answer of why. Sending out the notice first actually buys time to get the answers and understand why this occurred. Follow these guidelines for unplanned notices:
- The first notice should go out immediately. This first notice does not need to include detailed information, all that is needed is simply stating what some users are seeing or experiencing and that it is being investigated. It should include when the next update notification will be sent, which can usually be set to an hour. Depending how critical the outage or issue, an hour could be too long or too short. This length of time will also depend on the capabilities of the team to investigate such an issue.
- The second notice must be sent within the time frame that was stated in the previous message to reinforce the situation is under control. If a message is not sent out when it is expected, then customers start to lose confidence in the situation. Even if there are no additional updates the message can simply restate what users are continuing to experience and that it is continued to be worked on. Further state another update will be sent when there is additional information or in another hour. If there is additional information or an expected time frame for completion, that should be included.
An important aspect of unplanned outage notices is to ensure engineers working on the problem are not the individuals sending the notification updates. This ensures that someone can keep track of the clock of when the next update needs to be sent, get it out before the promised time, and can take the time to formulate a message. Ideally there should be a single person designated to send these updates.
What to Say?
You have the option to be impersonal and send automated messages, but they will most likely get ignored or leave your customers disillusioned. For example:
- Bad: 'The server is unavailable.'
- Good: 'The server is unavailable due to a hard drive failure. We are in the process of replacing the faulty drive and expect it back online at 12:30 PM. Until then, you may experience intermittent connectivity issues. We apologize for the inconvenience.'
Clearly, the latter message along with Constant Status updates would be much more valuable to a customer. Here are some additional items to consider:
- Acknowledge the problem and take ownership of it.
- If you're doing preventative maintenance, say so. Customers understand and appreciate the idea of 'an ounce of prevention is worth a pound of cure.'
- Inform your customer what they could experience during the outage. Most importantly, include how long the outage is expected to last or at a minimum inform them when they can expect another status update.
- If you can’t say anything nice, don’t say anything at all, but if you can muster a humorous comment in the outage notice, then do it. I like joking about how peaceful the commute is at 4 AM while I'm driving to the office to resolve an outage. Provide frequent updates, and always remember to send a final follow-up status update when the outage/issue is resolved.
- Be honest. Lying, whether explicit or through omission, is not a good way to win friends and influence people. Never play the blame game. Get the problem solved, and get the service back online as soon as possible. Blame assignments are for internal discussions and even then it should be about not making the same mistake rather than who dropped the ball. Keep it concise, but no jargon.
Subjects lines and headings are the first thing that will get the attention of the recipient. It is important to make them efficient and effective. Long subject lines are often not read, because either they are too lengthy for the email client to render or users don't have the patience to read it fully and will just click into the actual email message or even worse, just delete the message. Here are a few approaches that can be used to ensure your users pay attention to the email.
- Company XYZ Proactive Notification
- Company XYZ Release M.N.O Announcement
- Incident Name -- (include the system, service or component)
- Incident Name Status
There are many different ways to send out notifications and some are better than others depending on the recipients.
- Provide Website Notifications - allows recipients to view at any time. Requires them to remember where to go for it.
- Post on a Chat including using Webhooks, Slack, or other chat methods - This is a push mechanism that can notify recipients in the desired tool of their choice and can provide history over time that is easily viewable. Often times a blinking icon can tell users a message is there to be read. However chats can often times be ignored or user must be online.
- Send an Email - While this is also a push mechanism it can be mixed with 100s of other messages and can be lost in the shuffle
- Send a Text message (SMS) - This can be good. but does not provide much detail given how short messages are. Suggest a link to a site with more detail be included in the message.
- Phone - This will get the most attention, however many users may be unable to answer the call (I.e. busy or in a meeting), and the receipt of the message could be delayed.
While there are many different factors that go into best practices for notifying customers it is important to note that each business, service or application has its own variation of what can be considered a best practice. Companies and users move at different speeds and utilize different technologies to get the message across. Regardless of the methods and practices, Constant Status is an excellent tool to facilitate this process. It was designed to meet the needs of your business, and ensures your team is ready to quickly notify your customers when the next planned or unplanned outage occurs.
Remember to notify your customers before they notify you, and that an informed customer is a happy customer.
I was amazed at the positive feedback I got from customers when I acknowledged the outage as soon as it occurred. I was able to send them a message with Constant Status before their support team was even aware of the issue.
They asked their own site reliability engineering team why they were unable to detect and notify everyone about the issue prior to us. We became a good partner after that incident.
System Engineering Manager