Did Your SaaS Provider’s Ear Fall Off?

It was the mid-1990s. Y2K was beginning to get worldwide attention. The first graphical internet browser, Mosaic, had paved the way for Netscape and Internet Explorer. And Yahoo was in its heyday as a search engine.

One night, back then, I remember waking up shortly after midnight with a terrible earache. Ouch!

Considering the power of the Internet at the time – with its hypertext markup language being the standard – I fired up my trusty modem and dialed into the Internet to look up ear infection on Yahoo. A list of standard results popped up. I clicked on the one that said ear infection symptoms and causes. I was taken to what appeared to be a medical journal since I didn’t fully understand the words they were using. But I knew I was on the right track. There were two close-up pictures: one of a healthy ear cell that was brightly colored and plump; the other of an unhealthy cell that was black and brown and deteriorated.

I may not have comprehended everything, but I was completely stunned by a sentence that appeared in bold within the text on the page: “If the infection persists, the ear will fall off.”

This sent me into a panic! Alarmed, I screeched to my wife, who was sound asleep in the next room, “Hurry, take me to the ER, my ear is going to fall off!”

Startled from a sound slumber, I heard her quickly pounce out of bed and get dressed. In her tired state, she struggled to find her car keys and slip on her shoes. While she was doing that, I held my left ear with my left hand and clicked on the next page; that’s when I realized my mistake. Before me were two pictures of corn. Once picture was a healthy ear of corn on a stalk brightly colored in yellow, the other a sickly ear of corn about to fall off its stalk.

“Never mind,” I yelled to her. “My ear’s not going to fall off. I was looking at an ear of corn! It’s ok, you can go back to sleep.”

I knew at this point that an earache was the least of my problems. As I heard my wife storm down the hall, I knew I was in trouble. I felt her ominous presence behind me when she asked “which ear is in pain?”

I timidly pointed to my left ear. She took a tight grip of it, twisted and gave it a good yank. “How does it feel now?” she retorted.

Actually, it did feel better.

Since the days of my earache, the power of today’s internet browsers has opened up a whole new business model known as Software as a Service (SaaS). Instead of the static pages of the 1990s with very little user interaction, we have more powerful dynamic browser technology.

The technology back in the ’90s did not lend itself well to robust applications on the Internet. However, that all changed with the evolution of the browser. Technologies such as JavaScript, Dynamic HTML and XML are now available to provide users with robust interfaces that include such elements as menu selections, edit boxes, tabs and tooltips. However, from a user point of view, that would seem to be all that is needed to run applications on the browser, such as Customer Resource Management, Financial systems and Enterprise Resource Management. But just as there was more to the “ear” than what I saw on that Web page 15 years ago, there is more work that goes on behind the scenes of a SaaS application than meets the eye.

To really provide a successful SaaS model, applications need to be engineered in a certain way, with three distinct tenets in mind:

1. Statelessness
2. Load Balancing
3. Multi-tenancy

Statelessness

Back in the old days of software development and server platforms, systems were written to run on a server dedicated to the application; they were known as stateful applications. If a particular task hung the system, all users on the server suffered response time issues. In my opinion, one of the failures of companies porting their legacy systems to the SaaS model was the inability of their architecture to handle such a scenario. Building an application from scratch to run on multiple servers requires the application be stateless. This means the interaction between the user’s browser and the application is disconnected multiple times during the session. Session information is maintained within the user’s computer to allow (recall the cookie concept) the server responding to the request to understand who the user is and what functions were being performed. This all happens with the user unaware of which server is responding to her request. The bottom line is that if a server suffers response time problems, a user can be directed to another server to handle future requests, once again with no impact to the user experience. This group of servers running the same applications is sometimes referred to as a sever farm. The SaaS provider maintains the server farm at their hosting site.

This leads to the next tenet.

Load Balancing

So when a request from a browser enters the SaaS hosting site, how does the system know which server will honor the request? This is accomplished through another server that is known as a load balancer. The function of this server is to check the health of all the servers in the farm. By knowing the state of each server the load balancing can pass the request to the server running with the least amount of work load. This ensures excellent response time for all users, regardless if one server is not feeling well at the moment. There are also additional benefits to this model. One is that the SaaS provider can add new servers to the farm during peak demand. Another is the ability to remove a server that may need maintenance with no impact to the user.

We next move on to the data component.

Multi-tenancy

Once again back to the olden days, if you needed to add another database you typically had to have another instance of the application attached to the new database. So in the simple scenario of a single database on a single server, if you have 500 customers each with their own database you would need 500 servers! Now comes multi-tenancy to the rescue. In this model applications are designed in such a way as to share common code with multiple databases. A single instance of the application can send and receive database requests to the appropriate customer database. In the example above, the 500 databases can reside on one server attached to a farm of application servers, monitored by a load balancing server. Pretty neat stuff!

The point is, the growing sophistication of Internet browsers has indeed facilitated the SaaS model; but also, the underlying technologies that happen at the SaaS hosting site – unbeknownst to the end user – also makes the SaaS model possible. When it comes to Software as a Service, there truly is more than meets the end user’s eyes.

The Lord of the Audits, Part 2: The Two Engagements

In 2008, as a successful SaaS provider of supply chain software for the healthcare industry, MediClick’s current and prospective customers began requesting a SAS70 Type 2 audit from us. We heard warnings that our developers would revolt if we launched the auditing process. We started to expect red flags due to our customer support methodology. We even anticipated the audit would cause trouble for our agile development process. But the value of a SAS70 Type 2 audit was too great. Once we decided to dive into the process, there was no turning back.

To head off any backlash from an audit, we wanted to ensure we had the right team in place. We first set out to find an auditing firm that understood the importance of preserving the successful process we had built during our seven years in business.  Since a certified public accountant must perform each SAS 70 audit, we found a firm that understood SaaS and could effectively work with us to build control objectives that reflected our business model.

After we had the auditing firm in place, we needed to undertake two engagements to achieve a successful SAS70 Type 2 audit: a readiness assessment and the final audit.

The first engagement, a readiness assessment, took place in March 2008. Much like the PSATs that high school students take to gauge their performance prior to taking the full SATs for college entrance, the readiness assessment would allow us to get a first-hand grasp of the entire process and the types of activities the auditors performed. We worked with the auditors to define our control objectives, which we designed so that an auditor could test and report on the effectiveness of the objective. Objectives ranged from the security of the MediClick network environment to the documentation of our development, testing and deployment processes.

Prior to the readiness assessment the auditors requested that we provide them with detailed documents, such as an organization chart, recent employee handbooks, listings of firewall changes, data backup schedules, visitor logs, hot fixes and service packs applied to production systems.

The auditors were on site for a week to perform the assessment; at the end they produced their findings and recommendations for remediation. The auditors assigned each recommendation with a risk level – from low to medium to high – and included a resolution difficulty rating with each.

We felt good about the results of the assessment because it revealed no “gotchas.” The recommended resolutions were doable within our 3-month remediation period. During this time we corrected any shortcomings discovered, such as using generic userids, tightening up our employee access rights and hardening our network and application passwords.

With a better idea of what to expect during the audit, our next step was to talk to the employees about SAS70. Because employee buy-in was essential for success, we needed to ensure all MediClick employees thoroughly understood the purposes of the audit process and what their roles would be. We held internal meetings with all the employees to explain the process and the importance of the audit to MediClick. This was a significant moment in our auditing process because we could see how the employees would react to the audit. It was a success! No one revolted as we had been warned would happen! In fact, just the opposite occurred: acceptance and enthusiasm.

After completing the remediation tasks and employee orientation, we were ready for the second engagement. Our actual audit period was from July 1, 2008, through Dec. 31, 2008. Our first official SAS70 audit would cover a 6-month time period. Then we would have an audit done for each subsequent 12-month cycle.

Two auditors were on site for two weeks during the month of December 2008, pouring through our system logs, documents and development processes. They held interviews with each department manager to review controls and operational procedures. The auditors observed MediClick personnel in their daily activities, looking at such detail as how well they enforced visitor sign-in protocol at the front entrance to our office. The auditors left with tons of data to review.

Even though we successfully navigated the readiness assessment, we held our collective breaths as we waited several weeks for the preliminary findings. Would we experience any of the doomsday scenarios my industry colleagues had painted in my head?

The results were in and we did it! Our first audit was clean and all tests of control objectives showed no relevant exceptions. The final report was delivered in January 2009 and immediately sent out to a backlog of customers who had requested the report.

Despite the tales of woe that we had heard, we had learned for ourselves how to successfully navigate the SAS70 audit process.

Next is The Lord of the Audits Trilogy, Part 3: The Return of Art. In this segment I will reflect on my views on what it takes for a successful audit, what are the critical areas you need to review when receiving a SAS70 Type 2 audit and the future of the SAS70 audit.

The Lord of the Audits Trilogy, Part 1: The Fellowship of the IT Executives

“Be prepared to lose half of your developers! They’re not gonna tolerate all the controls, checks and balances getting in the way of coding and implementing your applications. I know this for a fact because it happened at my own shop.”

That was the advice I received from a fellow IT executive while discussing a SAS70 Type 2 audit at a meeting on IT governances. At the time, the company he worked for had already undergone the audit, which MediClick was considering. The process involved a complete, independent review of his company’s controls and operations, ending in a published report assessing the effectiveness of those controls.

The executive offered his cautionary tale about SAS70 in late 2007 during a Triangle Technology Executive Council (TTEC) meeting in North Carolina’s Research Triangle Park. Now with a membership of more than 100 senior IT executives from over 100 companies, TTEC included only a dozen or so executives at the time MediClick was considering SAS70.

I sought the wisdom of these IT executives because we were experiencing an increased demand for MediClick to become SAS70 compliant, something we purposely put on the back burner. Why? Because we were concerned that our own SAS70 audit would be cost prohibitive for a company of our size and that it would slow our development process to a crawl.

A bit of company history may put our SAS70 dilemma into perspective.

Formed in 2000, executives wanted MediClick to be an Application Service Provider (now referred to as SaaS) for healthcare. We saw a need to advance the industry beyond the traditional software solutions our larger competitors were putting on the market.

Our success depended on our ability to deploy our application quickly and effectively. We had to be lean and mean, which meant employing rapid development methodologies and a hosting site for our software.

Fast forward seven years, we met these obligations and became a successful SaaS company. It was then when our hospital customers began requesting that we become SAS70 compliant. Before we could do that, though, we had to be careful not to change the positive impact we were having on our customers.

“MediClick’s customer support is very helpful. If we have problems, they log on and solve them. They get back to us fairly quickly.”

“MediClick has a sprint process for enhancements instead of doing a rebuild each year. Every eight weeks they do one of these small builds and get things fixed. I have never seen a company be so efficient.”

Published by an independent organization that evaluates healthcare companies, these quotes reaffirm our commitment to efficiency and customer service. We were – and still are – a more nimble software company. We were afraid a SAS70 audit would change that.

For one, we expected that an audit would flag us when our support team signed on to the customer’s application to resolve issues. The SaaS model made this easy because we could access the application via the Internet. Unfortunately, all MediClick personnel used a single sign on to the databases. That was a no-no for SAS70 as it left no audit trail that documented what a particular MediClick support person did.

While those sign-on issues were a top priority, protecting our development methodology became a main concern. Since we had introduced the tenets of agile development and abandoned the standard 12 to 18 month release model, our customers were used to seeing our enhancements every 6 to 8 weeks.

For seven years, we had been fine tuning our process so we could effectively deliver value to our customers. We questioned whether a SAS70 audit would set up road blocks and cause us to lose our competitive edge. Would it frustrate developers resulting in a mass migration to the exit doors? Would we have to revert back to a long release cycle? These questions resulted in many sleepless nights.

So why did we choose the route that my peer advised me against?

First, we wanted to give our customers what they wanted. Plus, as we met more and more industry prospects asking for our SAS70 validation, it became apparent that MediClick would have to take the SAS70 plunge sooner rather than later.

So late in 2007 the decision was made; there was no turning back. We would attack this problem as MediClick has always responded to a challenge: jump into the SAS70 waters with enthusiasm to get the job done completely and correctly. Failure was not an option.

The most difficult part was determining where you start the SAS70 process. How prepared were we? Were there other vulnerabilities we were not aware of? Would there be a potential for a poor report and how would that affect our business?

The storm clouds formed over the horizon and the Dark Lords of the invidious auditors began their approach.

Next is The Lord of the Audits Trilogy, Part 2 – The Two Engagements.