Realtime Analytics for the rest of us

My two cents added to the discussions here and here.

John's points about the DW purchase decision being pushed to the SaaS vendor and being less relevant to the enterprise (analytic application mid-market customer) is the key to the big switch. A similar analogy would be that when an Enterprise gets its electricity from the electrical company (the GRID ). All that it cares about is the SLA – Can the supplier meet my XXX Megawatt per hour demands at peak loads of YY, 24/7 and after that the pricing is the decision point. What kind of generators the electrical company uses (Hydro , coal , windmills) is an important decision for the electrical company but certainly not relevant to the Enterprise using the electricity.

So this analogy leads to a bit of soul-searching for DW appliance vendors like Terradata , Greenplum etc. “Who is their customer?” are they taking up the role of GE (who manufactures turbines , windmills etc) to serve the SaaS vendors OR do they want to be offering solutions at a level higher to the end consumer that ultimately end up using their appliances. I think a little bit of both and mixture of a lot more partnerships is probably what is going to happen.

It is also interesting to note two other trends that will shape the BI world.

1. Fragmenting of DB market to specialized Database
   
2. Availability of webscale level specialized Databases like BigTable , Hadoop/Hbase at very low entry points.

These trends will lead to a development of a longtail-type market for real-time analytics in an SaaS model (example – Recommendation service based on collaborative filtering, using Predictive modeling results during the underwriting workflow for approving a quote).The reason this will happen is because

a) These kinds of applications are more focused and can be performed in silos’. The whole concept widgets moving at the next layer of functionality and reuse.

b) They are also better served by vendors whose livelyhood depends on bettering the algorithms that power the analytics engines.

c) With things like Bigtable , Hadoop/Hbase exposed to the world at a very low entry point , all it takes is one guy to improve an algorithm and expose to the world as a service.

Cloud computing and Datawarehousing

Following my last blog entry , my brain continued elaborating on the thought of Cloud Computing Adoption in the Enterprise.

Having worked in the Enterprise Space for So long , I am hard pressed to come to terms with the notion that Enterprises will be willing to completely outsource their Information Management and IT infrastructure and more so in a constrained environment like that of Google AppEngine.

Amazon AWS with its ala carte is still a better option to Enterprises as compared to Google App Engine. As you can pick an choose what you want. I think the key is that any Cloud computing vendor needs to "FIT IN" into the Enterprise's Architecture . This basically imply's that more entry points you have to the cloud infrastructure the more use-cases you will have for Enterprise Adoption. So , it seems like Amazon has a better strategy for Enterprise Adoption.

Another use-case that came to my mind is the impact to the EDW world. With things like BigTable and simpleDB exposed , why would an Enterprise invest in highend Databases like TerrrData - why not use a proven scalable platform like BigTable to run your analytics. In any case you need to do your EDW work in house on separate machines from your core systems - So using an on-demand infrastructure for such needs makes sense.

Shame on you if you cannot start a .com now !

Life could not have been better for developers. I thought that what Amazon had done with AWS was the best developers could have had , But now we have one more entry into the space-- Google App Engine

It is interesting to see how the SaaS/HaaS space is shaping up. Googles App Engine seems like a layer above AWS. i.e a hosted development platform - the toolset for developing an application on App Engine will be focussed and consequently also limiting. App engine provides excellent integration to google's services(like single signon etc) but it is also limiting , i.e I cannot use Java (at least right now).

AWS on the other hand gives you more power (build your own machine the way you want it) but it has a higher learning curve and if you are running a shop on EC2 you will require a System Administrator to manage your website.

While at this nascent stage of Cloud Computing it seems obvious to compare AWS and Google's App Engine but I think that as the market evolves both the offerings will really address separate markets. Google will get its pie from transitioning the traditional Rapid Application development tool (RAD) customers and Amazon from the traditional IT Infrastructure shops. They certainly converge at some level but it will be years before that happens.

It is also interesting to see that traditional IT Majors like IBM , SUN and HP are missing in action from this revolution. One reason could be that at this point AWS and AppEngine seem like a mom-pop shop offerings(Small business and non mission critical) , how do these offerings translate to Enterprise customers and what are the SLA's that will mature these offerings to Enterprises is something I look forward to understanding in 2008.

Maybe Google and AWS needs someone like Capgemini or Accenture to do this.

If you have the time check out the video on google app engine

The Long Tail - Taken to the extreme

OK , I have read the long tail and I agree it is a great book. It is very interesting to see how Amazon is taking the concepts of Long tail to the extreme.

Refer : AWS fulfillment service : is a service where you can use the Amazon warehouse to store your products and they will then manage the shipment , order execution and payments for the item.

Earlier , when AWS-EC2, S3 got introduced I really thought that it was vertical integration. But now I am starting to believe that what Amazon is doing is just scaling up every core competency they have. This is not really vertical integration in the sense that it is not coming from strategic business decision of getting into a market, but really coming from the fact of making the overall cost model of operations better. Volume drives costs down and if Amazon can manage to get more shipments there cost per shipment will go down. I don’t thing that they are getting into the warehousing business big time (i.e taking shipments from containers from china etc) , But seems like a experiment that might / might not work.

The key thing in Long Tail economics really boils down to - what is the incremental cost of exposing an Enterprise's inner business functions to the world as a competency. Frankly if you need to re-architect your processes to support this you need to rethink on adopting Long Tail concepts into your business.

Database 2.0

For a long time (decades I believe) it seemed like the General purpose Database was the silver bullet. Object Oriented Databases tried to take the shine away from RDBMS , but only to strengthen the notion that RDBM’S do best at what they are supposed to do – Store , Query and Update Data.

The application development world adjusted to the notion that the Object Relational mismatch is a reality and instead of fighting it let us work with the RDBMS systems and make world a better place. Tools like Hibernate had great success because they accepted the strengths of RDBMS as compared to competing technologies at that time like EJB's . The Enterprise was also caught up in the ERP wave where everything in the Enterprise needed to have a predefined structure, vocabulary established etc.

2007 brought some interesting changes in the DB world. Web 2.0 brought the concepts of self organization , realization that all information does not reside in-house , that everything cannot be structured , that Information Management is not just EDW / BI - there is a whole world out there with unstructured information and Semantic Web . Consequently SQL and RDBMS is not the silver bullet.

2007 also brought some interesting DB’s to the forefront including SimpleDB , BigTable , HP Neoview , CouchDB each of these serving very different purposes finetuned for particular needs.

So what is my point ?

I think the Database market is splitting into two layers
1. The general purpose database market - now turning out to be a commodity market
2. The specialized DB market

So finally I would like to close on the trigger that got me to think the above - The mysql acquisition by sun – Why does sun need a database product NOW ? it has severe wall street issues and convincing the market of buyout of a company that gives away free products is going to amazingly difficult.

Here is the answer I came up with .
With the technology and engineering capability needed to build a general purpose database generally available – Building a high performance RDBMS system that competes with the likes of Oracle and DB2 does not seem like a very big challenge. So if competing on price and brand are your differentiator's in the general purpose DB market then what better way to compete by giving the product free.

Now monetization on mysql and revenue potential is a separate discussion. My discussion above is keeping the number of deployments in mind.

more reasons for the Sun / mqsql acquisition

Struts , JSF , spring mvc , GWT - what is it going to be

I was reading the blog at java based web frameworks for 2008 and ended up doing a query on indeed.com.

So I compared the following
1. Struts
2. JSF (Java server faces)
3. GWT (Googles web application toolkit)
4. Spring MVC

Here is what I found

And Now comparing the Relative growth

As I had expected Struts is the clear leader and my guess just like Mainframe's still are a very important part of the Enterprise , Struts will always be there in the web 1.0 world.

JSF - to me is really a web1.0 technology -just an upgrade to struts.Sun has a knack of making things complicated and I dont think it has handled JSF any different than EJB's 1.0.

Spring MVC - has an interesting story, Spring Core(the middleware piece) has a very strong acceptance and spring core has a bigger audience than spring mvc. It's simplicity of programming model , well thought feature's make it a good candidate for use.

GWT in 2008 is IMO going to be the real winner. About an year ago I had posted that 2007 would be the year for GWT . Well looking at the numbers I think it was. It has not overtaken struts but the relative growth has been the highest - 30,000 % WOW!.

It has three things going for it
1. Java based (so the corporate java developers have an upgrade path)
2. It exposes a whole new capability of ground up AJAX applications not available in Struts , JSF or spring MVC
3. Simple (in every way - learning , developing , building , deploying)

I ran into a client recently who was using GWT very aggressively , Spring core and Spring MVC.When I asked him about why he was using Spring MVC and GWT- he commented the following

- Everything on website cannot be AJAX based application style. I will have some pages that really are suited for html (jsp) type content . I am using Spring MVC for that(example login page , help pages )
-Also Spring MVC is also a good place to put everything together. It works well with J2EE file structures etc and a great way to launch everything and coordinate the website.

The above view is what I think will shape 2008.

Google Energy and Vertical Integration

Google’s(NASDAQ:GOOG) announcement to enter the Energy space kinda surprised me but when you look under the hood there seems to be very good business justification for this.

Google is doing what Henry Ford did in the auto industry and Rockefeller did in the Oil industry – Vertical Integration.

Vertical Integration is defined as - The ownership by the same company of different functions in a supply chain relating to the provision of a particular good or service. Vertical Integration is generally done to lower the transaction costs in the supply chain and synchronize the supply and demand across the chain. Vertical Integration was pioneered by business leaders like Rockefeller and Henry Ford , where after reaching a certain size in their business they expanded into owning the supply chain so as to minimize the risk , control quality and lower transaction costs. Rockefeller after owning oil refinery branched out into oil distribution, oil retail and oil production.

Electricity costs for any SaaS vendor ( Google included) Electricity costs are significant and it is only going to get worse with Oil at $100 and the Global warming problems with coal. So it makes sense to manage the risk on the supply side and try to solve the problem. Google with its core competency of managing and monetizing innovation is also well placed to take up the challenge.Besides in my opinion Energy is going to be next frontier of innovation and to keep on sustaining at PE multiples of 50's long term GOOG needs to be looking at the next growth industries.

What got me thinking on reading about the Google Energy post is the approach Enteprise2.0 companies(like Amazon and Google) are taking towards Vertical integration. Thought leaders like Don Tapscott have argued that in the 2.0 world the transaction costs (the costs for collaboration) that justified vertical integration have evaporated to almost nothing now and vertical integration potentially does not make sense now as the integration costs internally in the Enterprise could be more than taking the route of ideagoras or Crowdsourcing

Google’s Energy entry and Amazon’s Webservices business in my opinion are Aspirational Core Competency based Vertical Integration transactions.

For Google:

Aspirational –Energy is the next thing
Core Competency - Managing and monetizing innovation
Vertical Integration – improve downstream supply chain.

For Amazon:

Aspirational - Become an HaaS Vendor
Core Competency - Software Engg and Systems Management
Vertical Integration - improve downstream supply chain.

Update : dec 29th: Here is another example for Googles vertical integration strategy. Also interesting to note that Google has been integrating downstream and not a lot upstream ( Android) , it did not not end up launching a GPhone as people had predicted.