Realtime Analytics for the rest of us

My two cents added to the discussions here and here.

John's points about the DW purchase decision being pushed to the SaaS vendor and being less relevant to the enterprise (analytic application mid-market customer) is the key to the big switch. A similar analogy would be that when an Enterprise gets its electricity from the electrical company (the GRID ). All that it cares about is the SLA – Can the supplier meet my XXX Megawatt per hour demands at peak loads of YY, 24/7 and after that the pricing is the decision point. What kind of generators the electrical company uses (Hydro , coal , windmills) is an important decision for the electrical company but certainly not relevant to the Enterprise using the electricity.

So this analogy leads to a bit of soul-searching for DW appliance vendors like Terradata , Greenplum etc. “Who is their customer?” are they taking up the role of GE (who manufactures turbines , windmills etc) to serve the SaaS vendors OR do they want to be offering solutions at a level higher to the end consumer that ultimately end up using their appliances. I think a little bit of both and mixture of a lot more partnerships is probably what is going to happen.

It is also interesting to note two other trends that will shape the BI world.

1. Fragmenting of DB market to specialized Database
   
2. Availability of webscale level specialized Databases like BigTable , Hadoop/Hbase at very low entry points.

These trends will lead to a development of a longtail-type market for real-time analytics in an SaaS model (example – Recommendation service based on collaborative filtering, using Predictive modeling results during the underwriting workflow for approving a quote).The reason this will happen is because

a) These kinds of applications are more focused and can be performed in silos’. The whole concept widgets moving at the next layer of functionality and reuse.

b) They are also better served by vendors whose livelyhood depends on bettering the algorithms that power the analytics engines.

c) With things like Bigtable , Hadoop/Hbase exposed to the world at a very low entry point , all it takes is one guy to improve an algorithm and expose to the world as a service.

Update: May 22 ,2008 : I ran into one more Cloud computing Data analytics solutions which is another Column oriented database check it out at Vertica

Cloud computing and Datawarehousing

Following my last blog entry , my brain continued elaborating on the thought of Cloud Computing Adoption in the Enterprise.

Having worked in the Enterprise Space for So long , I am hard pressed to come to terms with the notion that Enterprises will be willing to completely outsource their Information Management and IT infrastructure and more so in a constrained environment like that of Google AppEngine.

Amazon AWS with its ala carte is still a better option to Enterprises as compared to Google App Engine. As you can pick an choose what you want. I think the key is that any Cloud computing vendor needs to "FIT IN" into the Enterprise's Architecture . This basically imply's that more entry points you have to the cloud infrastructure the more use-cases you will have for Enterprise Adoption. So , it seems like Amazon has a better strategy for Enterprise Adoption. Another use-case for Enterprise Adoption is via an SaaS vendor case in point - Vertica . Vertica is a user of cloud infrastructure from Amazon. But the way the cloud is coming into the Enterprise Architecture is via an SaaS vendor.


Another use-case that came to my mind is the impact to the EDW world. With things like BigTable and simpleDB exposed , why would an Enterprise invest in highend Databases like TerrrData - why not use a proven scalable platform like BigTable to run your analytics. In any case you need to do your EDW work in house on separate machines from your core systems - So using an on-demand infrastructure for such needs makes sense.

Database 2.0

For a long time (decades I believe) it seemed like the General purpose Database was the silver bullet. Object Oriented Databases tried to take the shine away from RDBMS , but only to strengthen the notion that RDBM’S do best at what they are supposed to do – Store , Query and Update Data.

The application development world adjusted to the notion that the Object Relational mismatch is a reality and instead of fighting it let us work with the RDBMS systems and make world a better place. Tools like Hibernate had great success because they accepted the strengths of RDBMS as compared to competing technologies at that time like EJB's . The Enterprise was also caught up in the ERP wave where everything in the Enterprise needed to have a predefined structure, vocabulary established etc.

2007 brought some interesting changes in the DB world. Web 2.0 brought the concepts of self organization , realization that all information does not reside in-house , that everything cannot be structured , that Information Management is not just EDW / BI - there is a whole world out there with unstructured information and Semantic Web . Consequently SQL and RDBMS is not the silver bullet.

2007 also brought some interesting DB’s to the forefront including SimpleDB , BigTable , HP Neoview , CouchDB each of these serving very different purposes finetuned for particular needs.

So the definition of Database 2.o in my opinion is that it is the realization that all information in the Enterprise cannot be out into RDBMS , cannot be structured and that there can and will be multiple datasources to information within and Enterprise.

So what is my point ?

I think the Database market is splitting into two layers
1. The general purpose database market - now turning out to be a commodity market
2. The specialized DB market

So finally I would like to close on the trigger that got me to think the above - The mysql acquisition by sun – Why does sun need a database product NOW ? it has severe wall street issues and convincing the market of buyout of a company that gives away free products is going to amazingly difficult.

Here is the answer I came up with .
With the technology and engineering capability needed to build a general purpose database generally available – Building a high performance RDBMS system that competes with the likes of Oracle and DB2 does not seem like a very big challenge. So if competing on price and brand are your differentiator's in the general purpose DB market then what better way to compete by giving the product free.

Now monetization on mysql and revenue potential is a separate discussion. My discussion above is keeping the number of deployments in mind.

more reasons for the Sun / mqsql acquisition

Collective Business Intelligence and Enterprise 2.0

Web 2.0 as defined by Tim O Reilly is "the design of systems that harness network effects to get better the more people use them" . In his blog piece he describes very eloquently how Google's Page Rank mechanism is good case of harnessing user generated content.

Now take this to the Enterprise context. Up till now most of the discussion of Enterprise 2.0 I have seen revolves around two things
1. Harnessing Collective Intelligence of workforce to collaborate and share via tools like Wiki's , blogs . Product sets include Lotus Connections , Suite Two
2. Conversion of traditional office tools to Office 2.0 style platforms. Google Docs , HP Adaptive Enterprise Solution are excellent examples that basically focus on the ability to conduct normal business in a collaborative environment without being co-located.

A third part perspective of Enterprise 2.0 that should probably be included in the mix is the Business Intelligence gathered as a result of the network effect taking place with the continuous accumulation of Employee and Customer generated data and having an even broader data set as a result of SaaS applications serving more than one Customer.

So consider Salesforce.com - with several organizations using salesforce.com , they pretty much know the business benchmarks of processes in an industry (example :If they have 15 customers in the consulting space who use salesforce from sale to staffing cycle , salesforce will have the data to understand what it takes to improve the cycle by comparing various customers).

Frankly , If I was an SaaS player , I would pay to get users to use my software so that I can get their data and further master the business processes based on Data Analysis. As Tim pointed out , that in Web2.0 world Data is King and for Enterprises in 2.0 world , knowing what data is your core asset (and hence not shareable) and what data is something that you can share is going to be key.

So now you know - why did SAP buy Business Objects ? To enable Collective Business Intelligence on their Business By Design platform stupid.

Why did SAP buy Business Objects?

Recently I ran into a client who had made a significant investment in Business Intelligence infrastructure and a decision support system. This Insurance Carrier had 8000 reports in their portfolio. Yes once again the number is 8000 ?? Needless to say that one of reason I was engaged was to rationalize the reports(besides fixing other problems they had like Data quality)


I work in the Information Management space and have come to realize that 80% of our work is in getting the infrastructure for decision support right (i.e focussing on ETL , Data Quality , one version of truth etc). A successful project's end point is generally the ability for the Business users to get the information they are looking for accurately.This measure of success traditionally has focussed most of the BI players and consultants to be technology centric and tools heavy on the plumbing aspect for data.


So where does SAP(NASDAQ:SAP) and Business Object's (NASDAQ:BOBJ) fit in? With the aquisition SAP now gets the capability to be more than a core business transaction processor. i.e with the intersection of its industry knowledge and the BI knowledge from Business Objects , it can now enter into the relatively new market of Business Health Monitering and Business Benchmarking (not sure if this is a standard term-just coined it). Its future software will probably be able to analyze the business outcomes against indstry norms and suggest remediation (example - Your billing cycle is 10days while in your industry it is 4 days , you can fix it by doing X Y and Z). OR it might be able to proactively moniter the health of your business taking the Enterprise view in context (Example - For the insurance carrier reserve level for the Auto LOB has gone below the norm of X , but the Claims ratio in Commercial LOB is looking good so overall reserve level should not change for the enterprise).

SAP gets a chance to deepen the relationships it has with its customers by adding the business outcomes offering and broaden by penetrating into the Business Objects Legacy contacts.

I expect the Business Objects aquisition to have the effect of 1+1 = 4. The deal is not accretive and on strictly financial terms dilutes EPS and growth rate and hence the resentment from Wall Street on the deal. But then Wall Street was never expected to be a thought leader in how the markets are going to evolve.

I also think that Business Objects and SAP are a good cultural fit. Business Objects and SAP are both leading edge Web2.0 players in their areas (Refer here and here) and hence understand the overall direction of the consumer.