I love it when a gnarly technical concept can be elegantly explained via a good analogy. In fact, I have found myself searching for these perfect analogies throughout my career.

I’ve been working on NoSQL projects of late, and found the following two articles – both with brilliant analogies — to be very helpful:

1. “Starbucks Does Not Use Two-Phase Commit” – This article is not about NoSQL databases per se, but it does illustrate the point that sometimes you can afford to lose a transaction or have your database in an inconsistent state temporarily.
2. “A Plain English Introduction to the CAP Theorem” – this is a brilliant explanation of the trade-offs you need to make when scaling a database:  consistency, availability, and partition tolerance.  If you’ve ever had a hard time wrapping your head around these concepts, this article will help.

Economics are a big deal when deciding whether to go with a NoSQL, NewSQL, MPP DBMS, or traditional RDBMS.

Many of the products  (in all four of these categories) claim that economics are on their side, because either…

1. Their products are ultimately cheaper than the alternatives, or
2. Their products produce greater benefits, that dramatically overshadow their cost.

The first is basically a Total Cost of Ownership (“TCO”) argument.  The second is about  higher ROI (return on investment) or shorter payback periods, versus the alternatives.

The TCO, ROI, and payback period for any particular data store (be it of the NoSQL, NewSQL, MPP, or RDBMS variety) will vary greatly according to the application.  It is important not to take vendor claims of  ”X% improvement” for granted.

Instead, create your own economic model for quantifying expected costs and benefits. Here are some factors to consider:

1.Hardware costs – Many vendors claim their products require cheaper hardware, either by using fewer machines or by using cheaper machines.  Hardware costs affects both up-front project costs (CapEx) as well as yearly maintenance fees (operational costs) paid to the hardware vendors.  Hardware costs can usually be further subdivided into processors, memory, disk storage, network interfaces, racks,  load balancers, dedicated storage appliances, etc.  The cost of memory can be a significant driver of costs, as well as the type of storage used (SSDs, SANs, RAID arrays of regular old hard disks, ….)

2. Software license costs & maintenance – The cost here can vary wildly, from $0 for one of the many open source NoSQL solutions, to millions of dollars.  This item includes up-front CapEx costs – as well as the annual maintenance bill, typically about 20% of the purchase price.

3. Software support costs – If you buy a commercial data store, customer support is probably included in your annual software maintenance bill (see the previous bullet). But if you are looking at open source, you will want to find a vendor that can provide ongoing software support.  And that costs, even if the software itself is free.  This is an ongoing operational expense.

4. Power costs – If you use fewer boxes, or use more energy efficient hardware, then your power bill will be lower — sometimes significantly so.  If you want to get fancy and impress others with your “green-ness”, consider calculating your carbon footprint, and take into account any Carbon Offsets you’d purchase to reduce the system’s environmental impact.

5. Administrative costs – This bucket includes the cost of the staff needed to keep your system running and healthy — usually the full-loaded salaries of the DBAs and other administrators.  In general, the more hardware you are running, and the more instances of software you are running, the more administrative staff you need.  However, different products have different demands of administrators.  For example, “sharding” (sensibly dividing up data across multiple nodes) can consume a lot of administrator time if  done manually, but many NoSQL and NewSQL systems do this automatically.  Recovering from downtime – both planned and unplanned – can consume large amounts of administrative time, so the availability of the system impacts these costs.  Other tasks that can take up a lot of time, but vary considerably from system to system, include: time spent on upgrades (especially if updates to the software come out very frequently),  time spent on performance tuning, and time spent on monitoring, backups, etc.  Different products, and different rules governing data and IT, require varying levels of attention from human administrators.

6. Developer productivity - If the structure and type of data in the datastore changes, often the applications that use the data store need to be changed as well. So, you should account for the fully loaded salaries (or consulting fees) for the programmer time needed to make these changes.

7. “Hard” revenue impact - Different configurations of products will achieve different levels of availability.  This ultimately translates into some amount of system downtime or time when system is overloaded.  If your system is essential to your company actually making money (for example, it is an e-commerce store), than every minute of downtime results in lost revenue.  If the system is still “up” but is overloaded, then you lose the ability to serve some customers who would have bought.  Again, the result is lost revenue.

8. “Soft” revenue impact – This includes things like “greater customer satisfaction”, “higher accuracy”, “better response times”  – things your choice of system might affect that will impact customers, and will thereby affect the amount of revenue they give your company.  ”Soft” costs are very real, but are often difficult to quantify.

Note that the above factors assume that you will own your infrastructure – the hardware, software, etc.  ’The model needs tweaking / revamping if you are “renting” infrastructure in the cloud, by using something like Amazon Web Services, for example.  (In the future, perhaps I’ll post about the economics of cloud deployments).

Hadoop is just starting to come into mainstream consciousness.  As a result, a lot of people are grappling with understanding the relationship between Hadoop and traditional data warehouses, and how ETL (Extract, Transform, Load)  fits into the picture.  On one of the “Big Data” forums on LinkedIn, someone asked the below question. Scroll down for my answer.

Enterprises are looking at Hadoop as an ETL  processing engine that will feed unstructured data into an Enterprise Data Warehouse to do traditional BI (Business Intelligence). Are companies looking beyond this for more value-added uses of Hadoop?

I work with a a few vendors of Hadoop-related technology; their end customers are indeed looking to use Hadoop as more than a mere ETL engine that feeds data into a data warehouse.

Instead, these customers are looking for ways to do some analytics directly on the data stored in Hadoop, in order to explore avenues of analysis against the raw data (stored in Hadoop) that were completely not anticipated when the data warehouse was designed.

A (highly imperfect) analogy I sometimes use is that Hadoop is kind of like having a garage (or maybe even a garbage dump) of infinite size. You can throw everything in there, just in case you might need it one day. No need to clean up anything before you throw it in your infinite garage – just toss it in!  This means, of course, that your garage will get messy and dirty quickly, and it will eventually get difficult to find what you need. You might even lose track of what’s in there.  But at least you will have everything in case you need it later.

A data warehouse, in contrast, is like, well, a warehouse:  finite size (due to the expense), with nicely ordered shelves, everything labeled, and the way things are stored are optimized for the most common tasks, so warehouse workers don’t waste a lot of time.

With your ETL scenario in my analogy, ETL would be “cleaning the garage”: going through your garage, finding all the things you need, dusting them off, making them orderly, and filing them neatly on shelves in your data warehouse.

All well and good, provided you actually know exactly what is in your very messy garage, which you probably don’t.  What’s needed are some tools to help you take inventory of what is in your garage and conduct some basic analysis against it. That way, you’ll know what is actually worth moving into the data warehouse.

Anyway, that’s the way I think about it, with my thinking formed by what some of my clients’ customers want to do. I’d appreciate some feedback on the analogy, btw…

In the past year, Sure Product Consulting has done a lot of work in the emerging “Big Data” sector. Our clients have included newborn startups building their business models on top of NoSQL technologies, as well as traditional business intelligence and database vendors.

In the process, we’ve had to come up to speed quickly on what “Big Data” is exactly, the underlying technologies, the differences, the specific challenges, and so on.

Because Big Data is such a rapidly evolving field – important new developments occur every week — some of the best information sources are blogs.

Here are the blogs that we found most useful in learning about this sector:

1. DBMS2 – Curt Monash’s blog is truly the most informative we’ve found.  You could spend weeks digging through his archives, and in the process you’d learn a ton about the state of the art in database, NoSQL, and BI. He has a no-bullshit style and seems to have a “personality” (if you know what I mean).  Able to speak to the macro business issues as well as the most granular of technical details,  Monash is an independent analyst with a wide readership.  He often posts about what specific vendors are doing: watch the comments on these posts for  extremely informative corrections/clarifications/debate from the aforementioned vendors and their competitors.  His writing style is heavy on bullets and sub-bullets, but very readable and informative.

2. MyNoSQL Blog – by Alex Popescu.  This very active blog keeps up with the latest news and developments in NoSQL databases.  There is frequent discussion about how one type of NoSQL database compares to another, and when and why NoSQL might be a better choice than a traditional relational architecture. It’s not the best written, but makes up for it with currency and completeness.

3. The High Scalability Blog. This blog is about building highly scalable websites in general, and not specifically about data management.  Topics also include “The Cloud” (2011′s most abused technology term) and overviews of the architecture of some of the biggest websites out there.  But Hadoop/MapReduce and other NoSQL technologies get about half the space on this blog.

4.  GigaOm’s Cloud Blog – Dedicated primarily to “the Cloud”, about 25% of the time, this blog covers Big Data topics. These articles are always well written.  But, darn it, some of the articles you really want to read are only available to paid subscribers of the GigaOm Pro service.

I’ve had a lot of conversations lately with product managers who are wrestling with the appliance question:  Should they create a software-only product, or should they deliver an integrated hardware-plus-software appliance that contains all the underlying hardware and software needed to run the product?

It’s an important question because:

1. It has broad ramifications for the company: Are you fundamentally a hardware + software company (with hardware engineers, warehouses of physical product, etc.) or the more streamlined software-only company?
2. It is very difficult (and probably ill-advised for many companies) to do both.

There’s a lot to say about this topic, and there are no pat answers.   Thus far, customers as a whole have not shown an overwhelming preference for one type of product over another, even though specific customers have strong opinions.  So, no matter whether you sell appliances or software-only, you’ll find you can’t participate in many deals for whom your product would otherwise be a great fit.

Because of this, some hybrid approaches are emerging under the monikers of “software appliances,” “cloud appliances,” etc.  I’ll go into these hybrids in a future post, but for now I’m going to focus on the decision between traditional on-premise installed software and a hardware+software appliance.

Anyway, here are some things for product managers to consider as they go through their decision calculus (in no particular order):

The Pros of Appliances

1. Appliances are sometime easier for business users to buy. If your target buyer is on the business side (not in IT), it might be easier for him/her to just purchase and run an appliance rather than getting IT involved.  Corporate IT can be notorious for squelching purchases for new products, especially if they’ve already aligned themselves with a big vendor like IBM or HP.  I heard of one business-side buyer buying an appliance and hiding it under his desk, all so the IT director next door wouldn’t find out, since the appliance was not on IT’s “Approved” list.
2. Customers perceive simplicity. One purchase to make. One price to negotiate. One vendor to deal with. One “throat to choke” if things go wrong. This perceived simplicity might give you the edge versus competitors.
3. Quicker and less risky deployments, leading to higher success and happier customers. At least that’s the theory.  The customer does not have to integrate, configure, tune, and test a complicated combination of storage devices, processors, device drivers, operating systems, application servers, databases, database drivers, security software, etc.
4. Goes with Geoffrey Moore’s recommendation of offering absolute full solutions to your target market. The appliance does it all. For other markets that are opportunistic and not truly strategic, let them (or partners) piece together their own full solutions.
5. Say goodbye to the enormous Supported Product Matrix, and the huge amount of QA effort needed to test your software on all the possible combinations. As a company, you won’t have to purchase and maintain these pieces of hardware and software either. At one company I know that moved their business model to appliances, they were able to reduce their build-and-test time from 36 hours to 6.  And they were able to reduce their inventory of test platforms from 100+ boxes to 10 or so.  (Of course, reducing the number of computers might also be accomplished via cloud Platform-as-a-Service, but that’s a different article).

The Cons of Appliances

If you are offer an appliance, you’ll have to overcome the following:

1. If you, the vendor, are small, and your customers are big companies, you probably can’t buy the hardware as cheaply as your customers. If you’re an appliance, you will eat the difference between your hardware cost and what your customers could obtain it for.
2. You’re at the whims of hardware suppliers. This can be particularly bad if some of your suppliers might decide to become your competitors.
3. Customers can’t re-use boxes they happen to have lying around for your product, which might have lowered their project costs.
4. Hardware and software buying cycles are often different. Hardware is often bought when the previous boxes “become obsolete” according to the customer’s IT department’s obsolescence plan.  Software can more frequently be bought whenever the business case is compelling, regardless of whether the old hardware in the data center is due for a refresh or not.
5. Customers seem to have a tighter range of price expectations for hardware than software. You might find your appliance being compared to a hardware server, or to other appliances that are not even in the same problem domain. Software, in contrast, has a huge range of prices – ranging from free to several million dollars – that seems to preclude customers from taking a “cost plus” mindset or comparing one piece of software to another based on anything but the benefits that the software will bring.
6. Recognizing revenue takes longer. To recognize revenue, you need to get the entire product in their hands.  For software, that means that at 11:59pm on December 31, the customer can download the software from your website and you can count that revenue in the year-end financial results.  But for an appliance, you have to get the product assembled and on a truck first.  And in some cases, you might not be able to recognize revenue until the appliance is actually delivered to the customer’s site.
7. You’ll need to carry inventory, possibly on several continents. You’ll need to develop relationships with warehouses, drop shippers, and hardware assemblers. Alas, more parties between you and your customers means you collect less revenue and that your relationship with the customer is more distant and less under your control.
8. You need to collect sales/use tax and handle the special accounting and financial rules for physical goods.  In fact, you’ll have to learn how to do this in all the countries/states/providences/counties/cities where your customers live.  In other words, you’ll need to outsource order fulfillment.  And that means you’ll need to carry inventory.
9. You’re on the bad end of “one-throat-to-choke.” YOU are now responsible for, say, the bugs in the IBM DB2 instance.  Or the propensity of these particular EMC disk drives to error out, even though you have no ability to fix these problems, other than call into IBM or EMC support.
10. Your appliance might not fit in with your customer’s data center standards. Although you would think that customers should not concern themselves with the technology inside your appliance’s “black box,”  many of them WILL in fact be concerned and will demand extensive technical details.  And if they don’t like what they hear , expect to be eliminated from consideration. This scenario is more common with larger companies and those that put a high priority on security. They’ll want to make sure that your appliance’s O/S kernel is hardened with the latest patches, that you aren’t running software with known vulnerabilities, etc.  These organizations will often have a “do not deploy” list and if your appliance embeds one of those black-listed components, well, you’re S.O.L.
11. Appliances are counter to data center trends of virtualization and cloud deployments. It’s the pendulum swinging back, to the furthest extreme, in fact.  Virtualization and the cloud are all about separating hardware from software so you can add/remove capacity on the fly and have multiple applications more efficiently share hardware.  In contrast, appliances completely meld the hardware and software together.  If you need more capacity, you need to wait until you can buy another appliance.

There are many other pros and cons to consider in the product decision of offering software-only versus integrated appliances. These are just the ones I thought of off the top of my head, so please feel free to add more in the comments.

Big Data is getting really hot these days, and along with it, some of its pet terms.  One of those terms is “In Memory Analytics.”

All of a sudden EVERYBODY — be they providers of data warehouse appliances, OLAP tools, data visualization tools, or other business intelligence derivatives — are claiming to have “In Memory Analytics.”  Even if what they offer is not truly “in memory,” or (for that matter) not really “analytics.”

While I dislike marketing that is deliberately misleading,  I see how this situation developed.  A lot of customers have hopped on the bandwagon and are specifically asking for “In-Memory Analytics” by name.  But the thing is, no matter how they phrase it, the actual customer problem is not “I need in-memory analytics.”  Instead, the problem is “I want to analyze a my data with near-instantaneous response.”  Thus, vendors who have covered at least some part of this problem are inclined to just slap the “In-Memory  Analytics” label on their wares, so that they can at least be considered, even if the term is not technically accurate.

The actual definition of “In-Memory Analytics” is a specific technical approach to the instantaneous analysis problem — an approach where the entire dataset is pulled into the server’s RAM in order to greatly speed up analytic queries.

It works great, but In-Memory Analytics is not the end-all-be-all approach by any means.  It doesn’t deserve to be synonymous with “instantaneous analysis,”  because:

1. Other technical approaches may also provide solutions to this problem.
2. “In-memory” cannot solve the problem at all for larger data sets.

More color on the first point:  In the next few years, we’re going to see lots of new approaches and technologies that provide the user with instantaneous insight into large data sets.  For example, MPP data warehouses, Hadoop, MapReduce, and other technologies will combine and be refined to give the user (the analyst) instant understanding of their data – regardless of whether all the necessary data is “in-memory” or not.

And on the second point: no matter what, we will always want to analyze data sets that are larger than what can be stored in memory alone.  The amount of data in the world is growing faster than Moore’s law and memory capacity improvements.  (Don’t believe me?  Take your favorite “in-memory” analysis problem and add a requirement for LOTS of historical data.)  Further, for the foreseeable future, disk storage will continue to cost a tiny fraction of memory storage.  Right now, Amazon Web Services charges about $0.10 per GB on disk, while memory costs 240 times as much, about $24.00 per GB.  So, expect lots of situations where the entire data set is huge — so huge that most of it is on cheap disk and only the most frequently accessed data is in-memory.  In-Memory Analytics alone won’t cut it.

Forbes.com recently interviewed me about the software and online services I use to run Sure Product Consulting.

They were interested in my perspective as a self-interested small business owner and as an expert in evaluating, defining and launching software products with real business value.

Check it out: “A Software Maven Picks Her Tools,” by David F. Carr, Forbes.com, June 18, 2010.

I especially like the part where they refer to me as a “software maven.” 

A colleague asked me this the other day: If a company is pursuing a “freemium” business model, how should they determine the optimal mix of features to offer in free vs. paid software?

It’s a good question. Freemium – where a product is made available in both a free and paid-for version – is a scenario I’ve encountered quite a few times and one that’s becoming increasingly common.

Today, customers pretty much expect free versions from any kind of online service. Free versions are also becoming common in installed software, even in the case of very high-priced, server-based enterprise software.

The free vs. paid question is worthy of an entire project, but here are some high-level thoughts that may be helpful to your decision calculus.

First, there are three basic kinds of “free” software, though some companies offer a hybrid:

1. Free product is functionality-limited.
2. Free product is simply a limited-time trial of the paid product.
3. Free product is identical to the paid product but includes neither customer support nor indemnification.

I strongly recommend staying away from offering only Option #3. In my experience, most companies that start this way eventually move to option #1 (functionality limited). Option #2 (time limited) is easiest to implement but may not garner the same widespread installed base as Option #1.

If you choose Option #1 (functionality limited), expect that all your customers will try the free version first. If they like it, fewer than 10%—often fewer than 1%—will go for the paid version. (Rarely will anyone to go for the paid version out of the gate.) After anywhere from a week to a few months to evaluate its benefits, the afore-mentioned small percentage of customers will upgrade to the paid product.

With this “typical purchase pattern” in mind, you’ll want to consider the following:

1. Be sure you understand your product’s value proposition and benefits. Extra research to nail that down is worth it at this stage.
2. Make sure the free version is a pleasure to use and delivers real value (see the previous bullet). If the user experience is not very good and if benefits take longer to percolate to the top than expected,then  don’t offer a free version to begin with. You’ll do more harm than good.
3. The free version must lack key functionality that is in the paid version. The paid version should provide a “pain killer” that the free version does not. The paid version shouldn’t merely offer a “vitamin,” it should deliver real pain relief. Otherwise, users will never upgrade. 
4. If possible, encourage customers to integrate the free product into their business processes and IT systems, so that it is hard to remove. Then, when they hit the above-described pain point, they’ll buy the paid version instead of starting a selection process with multiple vendors. For example, you might want to make a free online product easy to integrate with the company’s single sign-on systems and other sercurity protocols, or maybe make it easy to integrate with in-house databases and services.
5. The differences between free and paid versions must apply to a customer who has been using the free product for two to three months. This sounds obvious, but it is amazing how many companies botch this. For example, if you offer any of the following features, tools, and support, then they should be available in the free version and not limited to just the paid version. Remember, the whole point of Freemium is to get as many people as possible using your product. Without the following features, you’ll needlessly limit your audience:
   • Installation and configuration wizards.
   • Access to APIs and tools for developers. A goal is to have your customers’ programmers embed the free product into their business processes if possible (see previous bullet).
   • Documentation on setting up the product and integrate it with business processes / IT systems.
   • Assistance with getting started. For example, don’t limit access to “getting started” user forums to paying customers.
6. Be sure that it is exceedingly easy to upgrade to the paid version, even if the free version has been programmatically embedded into business processes and IT infrastructure. If customers must rewrite code, install new software, or implement a migration process in order to use the paid version, they will likely continue to use the free version—or contact your competitors.

Finally—and this is important—realize that if you make too much available for free, it is very hard to put the genie back in the bottle—unless users understand that you are offering a time-limited free trial from the get-go. People are furious at Ning, for example, for charging formerly free communities. I am not sure the brand or company will survive. In fact, it appears that competitors are already hovering to snap up disgruntled customers.

Clearly, I could write a dissertation on this. Maybe I already have. Let me know if these high-level thoughts are useful to you.

I’ve said it before, and I’ll say it again.  All product managers should develop well-researched and well-supported product strategies, even if your boss is not requiring it.  It’s difficult to make the time, especially with the dozens of activities that product managers are usually simultaneously juggling, but it must be done.

Why?

1. Otherwise, you will only arrive at your desired destination by happenstance.
2. Otherwise, all your time will slip away fighting fires – many of which would never have started to burn had a clear product strategy been in place.

80% of the value of creating a product strategy is for the product manager. It clarifies your thinking, challenges you to come up with new ways of addressing problems, helps you better express WHY this is the best plan to all concerned, and gives you confidence that your plan is good, which in turn gives you the motivation to doggedly pursue your vision for the product.  You will need this motivation.  In fact, the best product managers are beyond “motivated” – they are “passionate”.

The other 20% of the value is in communicating your vision and your plan to others – the developers, the sales people, your customers, the market in general.   Having done your homework, you’ll be confident with yourself, credible with others, and convincing even to the skeptics and folks who like to play “stump the chump”.