Did Your SaaS Provider’s Ear Fall Off?

It was the mid-1990s. Y2K was beginning to get worldwide attention. The first graphical internet browser, Mosaic, had paved the way for Netscape and Internet Explorer. And Yahoo was in its heyday as a search engine.

One night, back then, I remember waking up shortly after midnight with a terrible earache. Ouch!

Considering the power of the Internet at the time – with its hypertext markup language being the standard – I fired up my trusty modem and dialed into the Internet to look up ear infection on Yahoo. A list of standard results popped up. I clicked on the one that said ear infection symptoms and causes. I was taken to what appeared to be a medical journal since I didn’t fully understand the words they were using. But I knew I was on the right track. There were two close-up pictures: one of a healthy ear cell that was brightly colored and plump; the other of an unhealthy cell that was black and brown and deteriorated.

I may not have comprehended everything, but I was completely stunned by a sentence that appeared in bold within the text on the page: “If the infection persists, the ear will fall off.”

This sent me into a panic! Alarmed, I screeched to my wife, who was sound asleep in the next room, “Hurry, take me to the ER, my ear is going to fall off!”

Startled from a sound slumber, I heard her quickly pounce out of bed and get dressed. In her tired state, she struggled to find her car keys and slip on her shoes. While she was doing that, I held my left ear with my left hand and clicked on the next page; that’s when I realized my mistake. Before me were two pictures of corn. Once picture was a healthy ear of corn on a stalk brightly colored in yellow, the other a sickly ear of corn about to fall off its stalk.

“Never mind,” I yelled to her. “My ear’s not going to fall off. I was looking at an ear of corn! It’s ok, you can go back to sleep.”

I knew at this point that an earache was the least of my problems. As I heard my wife storm down the hall, I knew I was in trouble. I felt her ominous presence behind me when she asked “which ear is in pain?”

I timidly pointed to my left ear. She took a tight grip of it, twisted and gave it a good yank. “How does it feel now?” she retorted.

Actually, it did feel better.

Since the days of my earache, the power of today’s internet browsers has opened up a whole new business model known as Software as a Service (SaaS). Instead of the static pages of the 1990s with very little user interaction, we have more powerful dynamic browser technology.

The technology back in the ’90s did not lend itself well to robust applications on the Internet. However, that all changed with the evolution of the browser. Technologies such as JavaScript, Dynamic HTML and XML are now available to provide users with robust interfaces that include such elements as menu selections, edit boxes, tabs and tooltips. However, from a user point of view, that would seem to be all that is needed to run applications on the browser, such as Customer Resource Management, Financial systems and Enterprise Resource Management. But just as there was more to the “ear” than what I saw on that Web page 15 years ago, there is more work that goes on behind the scenes of a SaaS application than meets the eye.

To really provide a successful SaaS model, applications need to be engineered in a certain way, with three distinct tenets in mind:

1. Statelessness
2. Load Balancing
3. Multi-tenancy

Statelessness

Back in the old days of software development and server platforms, systems were written to run on a server dedicated to the application; they were known as stateful applications. If a particular task hung the system, all users on the server suffered response time issues. In my opinion, one of the failures of companies porting their legacy systems to the SaaS model was the inability of their architecture to handle such a scenario. Building an application from scratch to run on multiple servers requires the application be stateless. This means the interaction between the user’s browser and the application is disconnected multiple times during the session. Session information is maintained within the user’s computer to allow (recall the cookie concept) the server responding to the request to understand who the user is and what functions were being performed. This all happens with the user unaware of which server is responding to her request. The bottom line is that if a server suffers response time problems, a user can be directed to another server to handle future requests, once again with no impact to the user experience. This group of servers running the same applications is sometimes referred to as a sever farm. The SaaS provider maintains the server farm at their hosting site.

This leads to the next tenet.

Load Balancing

So when a request from a browser enters the SaaS hosting site, how does the system know which server will honor the request? This is accomplished through another server that is known as a load balancer. The function of this server is to check the health of all the servers in the farm. By knowing the state of each server the load balancing can pass the request to the server running with the least amount of work load. This ensures excellent response time for all users, regardless if one server is not feeling well at the moment. There are also additional benefits to this model. One is that the SaaS provider can add new servers to the farm during peak demand. Another is the ability to remove a server that may need maintenance with no impact to the user.

We next move on to the data component.

Multi-tenancy

Once again back to the olden days, if you needed to add another database you typically had to have another instance of the application attached to the new database. So in the simple scenario of a single database on a single server, if you have 500 customers each with their own database you would need 500 servers! Now comes multi-tenancy to the rescue. In this model applications are designed in such a way as to share common code with multiple databases. A single instance of the application can send and receive database requests to the appropriate customer database. In the example above, the 500 databases can reside on one server attached to a farm of application servers, monitored by a load balancing server. Pretty neat stuff!

The point is, the growing sophistication of Internet browsers has indeed facilitated the SaaS model; but also, the underlying technologies that happen at the SaaS hosting site – unbeknownst to the end user – also makes the SaaS model possible. When it comes to Software as a Service, there truly is more than meets the end user’s eyes.

Waste Management, Teenagers and User Interface Design

Several years ago when my son, Alex, was a sophomore in high school we were having our morning breakfast with his best buddy, Dillon, at a bagel shop prior to the start of their school day. The shop bustled with folks rushing to grab a coffee before work while more leisurely groups enjoyed a bite as they discussed the day ahead. In the midst of this scene, Alex says “Dad, we’re going on a field trip today with my science class.” Intrigued, I naturally inquired where he was going. “To the waste treatment plant on Falls Lake,” he replied. “Oh! Do you need some money for the souvenir shop?” I quipped. He casually replied “No” as the adults sitting around us laughed out loud. Alex and Dillon continued to eat as if nothing humorous had happened.

Adding a bit more levity to the story, Alex approached me that night and said, “You know, there really is no souvenir shop at the waste treatment plant.” To which my wife and I had another good chuckle.

So why did the adults find humor at the situation while the teenagers didn’t get it? It has to do with how we view the world. People see things in different ways based on many factors: age, culture, education, etc. These all affect the way people perceive their environment. It particularly impacts how different people interact with technology.

Time and again, I’ve seen scenarios where a CFO purchases a financial system because he loves the sophisticated user interface. But the chosen system falls flat at implementation because his operational staff – those who sit in front of their workstations for 8 hours each day – hate the interface because it is too laborious; they can’t get their work done effectively.

As software developers, we face the difficult task of designing applications for a myriad of users, each with different value systems. Our daily challenge is to build a user interface that allows the casual user and the expert to perform their tasks on the same system. It isn’t always easy!

The casual user wants a simple, intuitive interface that walks them through the process and enables them to complete their task without any confusing or ambiguous icons, links or instructions. The expert, or power user, deals with the application for hours each day and wants quick shortcuts and easy access to functions.

There is no middle ground; the novice expects to be led step by step through each function and feature while too many clicks and too much detail will frustrate the expert. As we start a new decade (another topic for debate: is 1/1/2010 or 1/1/2011 the start of the new decade) the challenge we face as developers is designing an interface that satisfies all.

To illustrate how difficult this can be, I went to Barnes & Noble and looked at books about Microsoft Excel. What I found was staggering. Excel 2010: The Missing Manual is over 800 pages and Excel 2010 Inside and Out is more than 1,000! And Excel is only a software tool to help you get your real job done. Has anyone ever read the entire text? Original versions of Excel, I’m sure, were simpler and required shorter manuals. But as the software evolved and more people started using it for different functions, it became more complex. Today, as the CFO uses it to balance a company’s budget, the project manager uses it to track a multitude of tasks. Different types of professionals need to know how to navigate the same software; therefore the 1,000 + page How To guide is born.

Our society today is filled with technology that has great benefits. We can get more done with our limited time each day. The cost, however, is frustration and, at times, the feeling of stupidity with the surfeit of user interface models we encounter in our daily activities.

My experience with a self checkout machine at a local mega hardware store last month is an example. Accustomed to the self checkout interface at the grocery store closest to my house, imagine my surprise when I accidentally pressed the French button on this unfamiliar system. Since I don’t know a lick of French, I had no idea how to cancel the transaction and start again. Five people were waiting impatiently behind me in line and I had this thing speaking French to me. I eventually got through the transaction, and all the people waiting were likely expecting me to speak in my most alluring French accent. They were a bit surprised when I turned to them and, in my best New York inflection, said “Sorry for taking so long.”

This embarrassment brings to mind the story of a new computer user who had just received a PC-DOS computer (remember them?) with a rather vaguely written installation guide. As he attempted to assemble the components, his frustration at the manual got the better of him. He called customer support for help, which only reinforced his feelings of ignorance. After several fruitless attempts to solve the problem, the support rep – who was obviously on a different experience level than the caller – claimed to have identified the problem.

“Great,” replied the customer.

The support rep said “Stand up in front of the computer parts.”

“Okay, I am,” replied the customer.

“Is your front door to the right or left?”

“To the left,” the customer replied.

“Good, start walking to the left and tell me when you are by your front door.”

“I’m there now.”

“Open the front door,” the support rep commanded.

“Door is now opened.”

“Good,” the computer expert on the other end of the phone paused. “Now step outside and shout: DOES ANYONE OUT THERE KNOW HOW TO USE A COMPUTER?”

What’s the point? During our different ventures in software development, we will encounter both pro users and novices. The trick is to find a way to make both feel like they’re getting a great experience, one that’ll make them want to come back again and again. That’s what makes software development challenging – in a good way. Technology continually evolves, making new techniques and methods available so we can design a user experience that satisfies all participants.

The Lord of the Audits, Part 2: The Two Engagements

In 2008, as a successful SaaS provider of supply chain software for the healthcare industry, MediClick’s current and prospective customers began requesting a SAS70 Type 2 audit from us. We heard warnings that our developers would revolt if we launched the auditing process. We started to expect red flags due to our customer support methodology. We even anticipated the audit would cause trouble for our agile development process. But the value of a SAS70 Type 2 audit was too great. Once we decided to dive into the process, there was no turning back.

To head off any backlash from an audit, we wanted to ensure we had the right team in place. We first set out to find an auditing firm that understood the importance of preserving the successful process we had built during our seven years in business.  Since a certified public accountant must perform each SAS 70 audit, we found a firm that understood SaaS and could effectively work with us to build control objectives that reflected our business model.

After we had the auditing firm in place, we needed to undertake two engagements to achieve a successful SAS70 Type 2 audit: a readiness assessment and the final audit.

The first engagement, a readiness assessment, took place in March 2008. Much like the PSATs that high school students take to gauge their performance prior to taking the full SATs for college entrance, the readiness assessment would allow us to get a first-hand grasp of the entire process and the types of activities the auditors performed. We worked with the auditors to define our control objectives, which we designed so that an auditor could test and report on the effectiveness of the objective. Objectives ranged from the security of the MediClick network environment to the documentation of our development, testing and deployment processes.

Prior to the readiness assessment the auditors requested that we provide them with detailed documents, such as an organization chart, recent employee handbooks, listings of firewall changes, data backup schedules, visitor logs, hot fixes and service packs applied to production systems.

The auditors were on site for a week to perform the assessment; at the end they produced their findings and recommendations for remediation. The auditors assigned each recommendation with a risk level – from low to medium to high – and included a resolution difficulty rating with each.

We felt good about the results of the assessment because it revealed no “gotchas.” The recommended resolutions were doable within our 3-month remediation period. During this time we corrected any shortcomings discovered, such as using generic userids, tightening up our employee access rights and hardening our network and application passwords.

With a better idea of what to expect during the audit, our next step was to talk to the employees about SAS70. Because employee buy-in was essential for success, we needed to ensure all MediClick employees thoroughly understood the purposes of the audit process and what their roles would be. We held internal meetings with all the employees to explain the process and the importance of the audit to MediClick. This was a significant moment in our auditing process because we could see how the employees would react to the audit. It was a success! No one revolted as we had been warned would happen! In fact, just the opposite occurred: acceptance and enthusiasm.

After completing the remediation tasks and employee orientation, we were ready for the second engagement. Our actual audit period was from July 1, 2008, through Dec. 31, 2008. Our first official SAS70 audit would cover a 6-month time period. Then we would have an audit done for each subsequent 12-month cycle.

Two auditors were on site for two weeks during the month of December 2008, pouring through our system logs, documents and development processes. They held interviews with each department manager to review controls and operational procedures. The auditors observed MediClick personnel in their daily activities, looking at such detail as how well they enforced visitor sign-in protocol at the front entrance to our office. The auditors left with tons of data to review.

Even though we successfully navigated the readiness assessment, we held our collective breaths as we waited several weeks for the preliminary findings. Would we experience any of the doomsday scenarios my industry colleagues had painted in my head?

The results were in and we did it! Our first audit was clean and all tests of control objectives showed no relevant exceptions. The final report was delivered in January 2009 and immediately sent out to a backlog of customers who had requested the report.

Despite the tales of woe that we had heard, we had learned for ourselves how to successfully navigate the SAS70 audit process.

Next is The Lord of the Audits Trilogy, Part 3: The Return of Art. In this segment I will reflect on my views on what it takes for a successful audit, what are the critical areas you need to review when receiving a SAS70 Type 2 audit and the future of the SAS70 audit.

The Lord of the Audits Trilogy, Part 1: The Fellowship of the IT Executives

“Be prepared to lose half of your developers! They’re not gonna tolerate all the controls, checks and balances getting in the way of coding and implementing your applications. I know this for a fact because it happened at my own shop.”

That was the advice I received from a fellow IT executive while discussing a SAS70 Type 2 audit at a meeting on IT governances. At the time, the company he worked for had already undergone the audit, which MediClick was considering. The process involved a complete, independent review of his company’s controls and operations, ending in a published report assessing the effectiveness of those controls.

The executive offered his cautionary tale about SAS70 in late 2007 during a Triangle Technology Executive Council (TTEC) meeting in North Carolina’s Research Triangle Park. Now with a membership of more than 100 senior IT executives from over 100 companies, TTEC included only a dozen or so executives at the time MediClick was considering SAS70.

I sought the wisdom of these IT executives because we were experiencing an increased demand for MediClick to become SAS70 compliant, something we purposely put on the back burner. Why? Because we were concerned that our own SAS70 audit would be cost prohibitive for a company of our size and that it would slow our development process to a crawl.

A bit of company history may put our SAS70 dilemma into perspective.

Formed in 2000, executives wanted MediClick to be an Application Service Provider (now referred to as SaaS) for healthcare. We saw a need to advance the industry beyond the traditional software solutions our larger competitors were putting on the market.

Our success depended on our ability to deploy our application quickly and effectively. We had to be lean and mean, which meant employing rapid development methodologies and a hosting site for our software.

Fast forward seven years, we met these obligations and became a successful SaaS company. It was then when our hospital customers began requesting that we become SAS70 compliant. Before we could do that, though, we had to be careful not to change the positive impact we were having on our customers.

“MediClick’s customer support is very helpful. If we have problems, they log on and solve them. They get back to us fairly quickly.”

“MediClick has a sprint process for enhancements instead of doing a rebuild each year. Every eight weeks they do one of these small builds and get things fixed. I have never seen a company be so efficient.”

Published by an independent organization that evaluates healthcare companies, these quotes reaffirm our commitment to efficiency and customer service. We were – and still are – a more nimble software company. We were afraid a SAS70 audit would change that.

For one, we expected that an audit would flag us when our support team signed on to the customer’s application to resolve issues. The SaaS model made this easy because we could access the application via the Internet. Unfortunately, all MediClick personnel used a single sign on to the databases. That was a no-no for SAS70 as it left no audit trail that documented what a particular MediClick support person did.

While those sign-on issues were a top priority, protecting our development methodology became a main concern. Since we had introduced the tenets of agile development and abandoned the standard 12 to 18 month release model, our customers were used to seeing our enhancements every 6 to 8 weeks.

For seven years, we had been fine tuning our process so we could effectively deliver value to our customers. We questioned whether a SAS70 audit would set up road blocks and cause us to lose our competitive edge. Would it frustrate developers resulting in a mass migration to the exit doors? Would we have to revert back to a long release cycle? These questions resulted in many sleepless nights.

So why did we choose the route that my peer advised me against?

First, we wanted to give our customers what they wanted. Plus, as we met more and more industry prospects asking for our SAS70 validation, it became apparent that MediClick would have to take the SAS70 plunge sooner rather than later.

So late in 2007 the decision was made; there was no turning back. We would attack this problem as MediClick has always responded to a challenge: jump into the SAS70 waters with enthusiasm to get the job done completely and correctly. Failure was not an option.

The most difficult part was determining where you start the SAS70 process. How prepared were we? Were there other vulnerabilities we were not aware of? Would there be a potential for a poor report and how would that affect our business?

The storm clouds formed over the horizon and the Dark Lords of the invidious auditors began their approach.

Next is The Lord of the Audits Trilogy, Part 2 – The Two Engagements.

Punch Cards vs. Airline Food

I remember sitting in my manager’s office in 1980 after just having completed a successful consulting assignment at the State Department in Washington, D.C. As I eagerly awaited my next job, Dave, my manager, canvassed pages of open assignments. He mentioned a consulting opportunity at Pan Am (for you younger folks, it was a major airline back in the ’80s); I thought it was a dream come true.

As a consultant, you anticipate that one assignment would eventually turn into a permanent position. I imagined the perks of working for an airline: unlimited travel to exotic locales; a bounty of delectable airline food (which was quite good in those days). I was already charting a course and ordering my first meal when the sound of Dave’s voice yanked me back into reality. He told me there were two negatives to this assignment:

1.       They were still using punch cards

2.       Since the operations were unionized, I had to play by union rules

The second negative was a nonissue. Once a proud member of the New York City teacher’s union, I could deal with a union shop. So the decision came down to punch cards vs. airline food. “Okay, I’m in,” I enthusiastically told Dave, ready to begin my career in this exciting industry.

Unfortunately, working as a contract programmer at a large company like Pan Am turned out to be more of a challenge than I thought. I lasted only two weeks.

My first incident involved a 20 punch card limit. Programmers were only allowed to punch a maximum of 20 cards per day, after that you were required to submit your coding sheets to a key punch operator. I had reached my limit by noon and all key punch operator personnel were out to lunch. I kept punching. End result: first violation.

The second violation happened after work hours when just about everyone had gone home. I sent a listing to the printer and waited for someone to grab it. You see, only operations personnel were allowed to burst listings and put them in cubby holes. But seeing my listing on the printer, just staring me in the face, was too much for me to bear. Knowing it was after hours and operations personnel were on skeleton crew assignments it could be hours before someone would get to it, I went for the printout. Yet again, another rule caught up with me.

That was it; my career at Pan Am was over with two, relatively minor, violations! So much for unlimited travel and tasty cuisine.

In working at Pan Am and a few other large corporations, I started to feel their rules and procedures cuffed my productivity. For me, it is far better to work for a small- to mid-sized company for several reasons.

                 1.       Direct Impact on the Bottom Line

In a small-to mid-sized business, each employee contributes to the success of the company and, most importantly, feels that way. Whether you are a developer writing code, QA analyst testing releases or a customer support representative helping clients, you know your work is critical and everyone in the organization notices it.

                 2.       Everyone is a Star

There really is no place to hide in a smaller shop. If you cannot carry your weight, your underperformance will be noticed. Whether it is a lack of ability or a poor attitude, your career will be a short one. On the positive side, as a viable employee you know the person to your right and to your left is competent, dedicated and will get the job done.

                 3.       Agility

I don’t mean the software development methodology we use at MediClick. When I say agility, I mean the ability to quickly adjust to customer demands and market conditions. Decisions to change product direction can be made quickly without layers upon layers of corporate structure to get through.

A case in point occurred at MediClick several years ago. Marketing approached senior management with a need to develop a contract management application for our SaaS suite that complemented our financial and supply chain products, which had been successfully developed and deployed over the last seven years. Senior management, architects and developers held an offsite meeting. That full day meeting resulted in a decision to pursue the development of this new product line. The next day we laid the foundation and started the project.

These reasons highlight why I feel small- to mid-size companies are often more effective. On the other hand, one might argue that a large corporation offers security and longevity. So what? That security comes with a few corporate rules you have to live with. It’s worth it, right?

My rejoinder to that argument is a single word: Enron.

An Exclusive Club That Anyone Can Join

Welcome to the A.R.T. of IT, a blog in which Tony Verdone shares his witty, entertaining and useful insights from his 30 plus years in the IT industry. It’s a must read for anyone involved in the business of information systems and information technology.

In the 1970s, as a laid off high school math teacher, I was sitting in my living room watching TV when a commercial caught my eye. Being the days well before the DVR or remote control, I couldn’t fast forward or instantly change the channel. So I watched.

The ad showed a slick, red Maserati sports car pull up in front of a steel and glass building. The car stops directly in front of the entrance and a debonair man of intrigue stepped out and entered the building. Once inside, a beautiful young lady in a long white lab coat greeted him. She helped him into his own lab coat.

The two walk by a set of computers – in the ’70s, a set of computers took up several rooms – with lights blinking and buttons flashing. They pressed buttons on the machines and then huddled over some data they had extracted and printed via their state-of-the-art dot matrix printer.

It seemed the two were part of an exclusive, high-tech club; one that I wanted to join. Their computer wonderland – while rudimentary by today’s microprocessor-based, palm-sized computer standards – intrigued me.

The commercial turned out to be an advertisement for a computer school. I knew what I wanted to do for the rest of my life.

I had no illusions when I enlisted in computer training. I knew the IT lifestyle – this elite society – would be nothing like the glamorous one portrayed in the commercial. In reality, it was much different.

In the early years, I had developed an application in Cobol for a Phase Four minicomputer at CBS, written CICS macro instructions to program the IBM 2260s, and scratched my head in front of a TRS-80 trying to decipher an error message.

It’s no wonder why, several decades later, the professional world gave us the unfortunate moniker of tech geeks! But, like many of you, I wore the label proudly, especially since we tech geeks can make an immense difference in the business world.

One such example begins in September 2004 when my Director of IT approached me with a news photo of a data center seven miles down the road from MediClick’s hosting site at Savvis. The shot showed a data center with the roof torn off, an unfortunate result of its encounter with a tornado.

The damage to the data center was clearly severe. Water had leaked on servers causing several customers to be offline for three days. What good is diesel backup for seven days if there was nothing to power –all the equipment was flooded!

I immediately asked myself what if this had been our hosting site? What if our servers were flooded and no longer operational?

We already knew how imperative it was for MediClick to be prepared for a similar emergency. Just as they do today, our customers depended on our software to order medical supplies for their departments – often using just-in-time inventory and placing orders days or hours before actual use.

We had a well-documented back up plan to get our clients back up and running after a disaster. We backed up the data and sent it off site to Iron Mountain for protection. In the event of an emergency, we could put together another co-location and restore the data.

With our recovery plan in place, we knew we would be well ahead of a hospital’s ability to get its own data centers operational after a disaster. In fact, our users understand when a short period of down time is necessary after a disaster, but we still weren’t satisfied. Moving the back-ups to a new server location required a long downtime. Unfortunately, there were no good alternatives available yet.

That’s when we decided that we needed to form our own exclusive club within the IT world – one where data downtimes were short and we were the heroes of the entire enterprise.

Enter the hot-backup solution. Technology had finally caught up to the long downtime issue. The first step was server virtualization management, which was just making its headway into IT. Then came the Utility Computing Platform.

Our downtime after a disaster shrunk from a couple of days to a couple of hours.

In future blogs, I will discuss how migrating from co-location to the utility platform satisfied our immediate hot failover needs until MediClick could develop its separate DR site. I’ll also talk more about disaster recovery and utility computing as well as Software-as-a-Service, cloud computing, agile development and more.

It’s a new club, and you’re invited.