Tag Archives: Data Management

Viable System Model and Data Management: Practical First Steps

Approaching data management through the lens of the Viable System Model for the first time may feel overwhelming. It’s an entirely reasonable feeling to have (if you’re unfamiliar read Trevor Hilder’s excellent introduction). Learning the Viable System Model is like a learning new language. Learning it and then applying it to your work isn’t too different from taking a French class and then being asked to write a pen-pal in Paris.

I’d like to put together a scenario that we can use to walk through this.  Let’s play pretend. Microsoft publishes a sample Visio database diagram that models a fictional bicycle businesss named AdventureWorks that we can use. You may want to download it to play along.

You’re the new IT manager for a moderately successful bike manufacturer, AdventureWorks. After meeting with your staff to learn what challenges IT is having, you’ve decided that one of your biggest problems is data sprawl. You have massive amounts of uncataloged and unmanaged data that may or may not be serving the business. You have data sprawl challenges in terms of depth and breadth. You suspect that years of unmanaged data in your database has led to your applications performing slowly. Furthermore, it’s reported that different business units each have their own applications, each containing some amount of duplicate data. There is no agreement on what information is canonical and there are  problems with updated information not reaching everyone. The good news is that Sales, Human Resources, Purchasing, and Production have all been sharing an enterprise class application that uses the AdventureWorks database for some of their work. Starting with the AdventureWorks database seems like a good place to start. Without a lot of information at hand and desperate for something, you ask your database administrator if she has a database diagram and it turns out that we have one and it’s up to date. At a high level, the diagram looks like this:


Scanning the diagram, you’re able to piece together a bit about at least what the database reflects about the business. You also look for boundary changes. Boundary changes, or foreign key relations between the schemas that reflect business domains, such as between Purchasing and Production, implies channels between systems. You’ve decided that Production will be your starting point. Your team, having been versed in the VSM, starts with Production. Production appears to be a type 1 system, that is, it’s firmly in the role of operations.

Your analysts set up a couple of meetings with the folks in Production. Of course, the staff, having had bad relations with IT in the past with poorly implemented tools that either don’t do the job or do it inefficiently, are a little reticent but they accept the invitation. The first thing they bring up is the poor performance of the AdventureWorks database. It was only fast during the first couple of years but over time, it’s turned into a real dog and now sometimes they will get up and get a cup of coffee while they wait for a screen to load. Your team, knowing that the database is running on premium hardware and knowing that no data has ever been archived out of the system, starts asking questions about how long bike part information is needed for production. Production, whose focus is on absorbing variety from Sales and delivering on customer expectations, really uses most of the bike part information for tracking inventory and forecasting for parts purchases. Digging deeper, your analysts find that not all the inventory is tracked in the database. Because of poor relations with IT, newer business attributes associated with changes in how they do work are being tracked in an Excel spreadsheet. It was easier to use Office than try and ask for changes to the enterprise application. The work order module of the application isn’t being used either. Instead Production uses email to let assembly teams know when work is ready for them. People sometimes forget to check their email or they get buried under other emails. This leads to unpredictable delays and other challenges. As your team starts to wind down the meeting, Production mentions that they have challenges meeting the special requests from Sales. The Sales department frequently oversells limited issue models, ignores recently discontinued models, or special requests. Your team certainly has their work cut out for them.

Let’s go over what your team discovered in that meeting. The AdventureWorks database and application has issues attenuating data, absorbing variety, and amplifying effort. Attenuating data could be pretty straight forward. Your team will need to have a couple of meetings about how long certain data needs to persist and in what state with the coordination, control, intelligence, and policy systems that have channels with Production. Some information likely can be archived in a report somewhere and purged. Or it may be transformed in some other way that meets the variety needs of other systems.

The problem of the application absorbing the appropriate variety is a little stickier. Shadow solutions, like the Excel spreadsheet they’re using, are in place because of inefficiencies between Production and IT. Investigating how those processes work and fixing them will take some later brainstorming but, for the time being, you can document and implement the new tables and import the spreadsheet into the database. This information too, may or may not be necessary for other systems to do their jobs. Your team will need to investigate, create, and document as necessary.

The biggest area of issue will be how to amplify Production’s efforts. The application’s work order module is so painful to use that the Production team would rather roll the dice on email. In those inboxes, you have “unstructured data” relating to work orders intermingling with many other types of information. Sorting and prioritization are challenges. Assembly teams are out of sync as communication and teamwork is out of sync. This is a coordination nightmare and is likely being felt in other parts of the business. For example, Sales may be having to deal with failure demand, “Where’s my stupid bike!?” This could beget direct commands from a control system, an unsavory but potentially necessary experience. The easy question for your team to ask is how to make the application’s work order module work better so they don’t have to use email. The better question is, in general, what does Production need to improve the flow of work? The application’s work order module may end up just getting deep sixed. Saving a poorly built piece of software does no one any favors. In this context, flow is what matters.

There’s also the specter of the residual variety that Production continues to absorb from Sales in terms of difficult or impossible to fulfill orders being placed. This likely also impacts Purchasing as well. There are data and communication issues here that need to be investigated.

Your team, informed of a systems approach, will have come away with more questions than answers. This is a good thing because systems thinking is a holistic approach that isn’t satisfied with addressing symptoms. It will compel one to find root cause.

What the Viable System Model does in this case is provide the appropriate model and concepts for data management. By understanding who or what in your organization is responsible for operations, coordination, control, intelligence, and policy you can start to ask the right questions on what they need. You will also need to employ the concepts of variety, amplification, attenuation, as well as algedonic signals and the Law of Requisite Variety. In this way, you can carry on a conversation with them about their needs instead of continuing the time-honored IT tradition of mere order placement and fulfillment. Having those conversations is critical because the false dichotomy between business and IT is vanishing (some places faster than others).

Data management isn’t just about finding and deleting data that isn’t used anymore. Data management is about ensuring that the data needs of the organization are met. To be effective, data management needs to partner with business process management. It’s very difficult to discuss one without encountering the other.  Employing systems thinking and the Viable System Model will go a long way toward doing the hardest work of all, asking the right questions instead of merely looking for right answers.

Tagged , ,

Data Management, Waste, and You

Data catastrophes usually come in the form of clumsy fingers. Sometimes it’s a user. Sometimes it’s an IT administrator. The recovery process is straight forward, right? Find the backup. Restore it. Resume eating Bonbons. Of course not. Technically, backups are easy. In practice, they’re amazingly difficult. And it has nothing to do with technology.

Backup sites tend to behave like digital versions of the junk drawer at your home. Just on a massive scale. The business, unwilling to part with data, just wants to back up everything. IT is left trying to sort out massive amounts of data, much of which it knows nothing very little about in terms of importance and relevance. Time to recovery? Point of recovery? Forget about it! And the money spent on storing it? Dizzying. Organizations that don’t treat their data as an asset (complete with value depreciation) often fall into either of two camps: those that continue to pour money into saving everything or those that are reluctant to spend money on storage. If you’re in former camp, good luck finding anything when you’re saving everything. If you’re in the latter camp, you’re going to eventually inadvertently push important data out of the system, losing it forever and not knowing it until time comes to use it. Both approaches are costly, one just defers the cost until later. It’s in every business’ best interest to manage their data as an enterprise class asset.

If our offices reflected the way we managed data, we’d find ourselves unable to navigate to the restroom with all the clutter. The individual practice of saving everything you’ve ever done on a computer rolls up into massive and gnarly enterprise problems. We also institutionalize it within our databases. Some of the best minds in the world have worked on trying to assist in resolving this problem of the human condition, this need to hoard data and save it forever, ignoring the diminishing return on value that much of it represents. Since we all struggle with data management, there’s an entire industry built around helping businesses with it. So, with costs starting somewhere in the low six figures, you can acquire yourself a data deduplication system where you can deploy some of the best mathematics in the world to assist you in managing this problem. This buys you only time at the cost of extra money as data deduplication is costly. The dike is built but the waters continue to rise.

Like backups, the value of deduplication is marginalized without a coherent plan to address on how to manage data as an asset. And managing data as an asset is a challenge unless one takes a systems approach to data management. The imperative of a systems approach to data management is that IT is laterally positioned within the business. We’ve had a number of years of experience within the industry with service-oriented architectures and even longer with extract-transform-load operations. We’ve built data warehouses. We’ve written an uncountable number of reports. We’ve long recognized the flow of data throughout the organization. There’s no one better positioned than us to lead our businesses into these brave blue waters. What is required of us is that we evolve that model to understand the flow of work, or even better, the flow of value creation throughout the enterprise. We need to understand what is required by each of the subunits to do precisely the job they need to do. As any analyst can attest, what is required likely doesn’t match what is done. This is how IT must provide the leadership necessary to put together a comprehensive plan of data management that treats data as an enterprise asset replete with backup schedules, archival schedules, records management, and more.

To that end, it’s time we resurfaced the Viable System Model and applied it to data management. Stafford Beer’s model dissects the organizational model from a systems perspective and describes the necessary subsystems required for an organization to be viable; that is, to remain effective within its particular ecosystem (or market). An excellent introduction to VSM can be found here.

As we progress into our respective data management programs, we will need to align each organizational unit with their appropriate system or systems, discover what states each system requires to maintain control (referred to as variety) and how each system interacts with the others. That leads right to discovering what data is needed to support control and what data needs to be ignored (referred to as attenuation). Understanding what data can be attenuated is key. Coordination and control systems can consume only so much information at a time. Oversupply of information leads to costly delays and waste. Most questions about data aren’t necessarily binary. That is, it’s not usually just a question of keep versus discard but, rather, how long to keep it and in what state.

If you’ve built a data warehouse, you’ve likely asked many of these questions already. The goal would be to expand that work so that the data warehouse isn’t the fixed point that everything traces back to. One of the tricky tasks of IT leadership is to decouple programs from the technologies that support them. Failure to do so leads to very costly evolutionary programs or over-reliance on obscure systems. Think about California and the amount of money they’re spending on COBOL programmers to support their legacy systems today. Take the work you’ve done in building data warehouses and expand upon it with the VSM. Generalize it.

What is particularly attractive about applying the Viable System Model to data management is its neutrality in language and flexibility in application. It has a lexicon that is neither rooted in technology nor business language. It is intuitive to understand and is malleable enough to be utilized in most any situation.

There’s more to this than just backup schedules. Backups are just the start. Your programmers and analysts, once involved, are going to start revealing data waste all over the organization. Like a compass, that data waste will point you to institutional waste baked into stagnant business processes. You’ll be a catalyst for business process improvement. Or just business process management if you’re starting humbly. What it takes is a common framework and language that sets the stage for asking the right questions.

It’s critical for you to start this now. Why? Because we can no longer afford to blindly throw money at problems that require planning and management. We’ll let Peter Sondergaard from Gartner be the final punctuation mark.

Tagged , , ,