Thursday, February 28, 2013

Continue With My API Service As A Guest

Making onboarding with an API as frictionless as possible is one thing I like to study and educate my readers about. While I was studying what I call the API reciprocity space currently, which is an evolution of what is known in the enterprise as ETL, I came across a company called FoxWeave.

FoxWeave is a service that can migrate and synchronize data across all your cloud and on-premise apps and databases. When I went to “try it now” at FoxWeave, I was told “no signup needed”. But I didnt’ think much about it until I saw the temporary guest account popup.

What a powerful concept. Evolving the freemium concept into the world of anonymity, allowing me to play with and understand your service before having to commit in any way. One less piece of friction when I’m trying to understad a new tool or service.

Nice work FoxWeave. Way to make interoperability and automation using APIs as frictionless as possible.

from API Evangelist

From ETL to API Reciprocity, Looking at 20 Service Providers

I spent time this week looking at 20, of what I’m calling API reciprocity providers, who are providing a new generation of what is historically known as ETL in the enterprise, to connect, transfer, transform and push data and content between the cloud services we are increasingly growing dependent on.

With more and more of our lives existing in the cloud and via mobile devices, the need to migrate data and content between services will only grow more urgent. While ETL has all the necessary tools to accomplish the job, the cloud democratized IT resources, and the same will occur to ETL, making these tools accessible by the masses.

There are quite a few ETL solutions, but I feel there are 3 solutions that are starting to make a migration towards an easier to understand and implement vision of ETL:


These providers are more robust, and provide much of the classic ETL tools the enterprise is used to, but also have the new emphasis on API driven services. But there are 10 new service providers I’m calling reciprocity platforms, that demonstrate the potential with offering very simple tasks, triggers and actions that can provide interaction between two or more API services:

I consider reciprocity an evolution of ETL, because of three significant approaches:

  • Simplicity - Simple, meaningful connections with transfer and tranformations that are meaningful to end users, not just a wide array of ETL building blocks an IT architect has to implement
  • API - Reciprocity platforms expose meaningful connections users have the cloud services they depend on. While you can still migrate from databases or file locations as with classic ETL, reciprocity platforms focus on APIs, while maintaining the value for end-users as well as the originating or target platforms
  • Value - Reciprocity focus on not just transmitting data and content, but identifying the value of the payload itself and the relationships, and emotions in play between users and the platforms they depend on

This new generation of ETL providers began the migration online with Yahoo Pipes. Which resonated with the alpha developers looking to harvest, migrate, merge, mashup and push data from RSS, XML, JSON and other popular API sources--except Yahoo lacked the simplicity necessary for wider audience appeal.

While I feel the 10 reciprocity providers isted above represent this new wave, there are six others incumbents trying to solve the same problem:

While studying the approach of these 20 reciprocity providers, it can be tough to identify a set of common identifiers to refer to the value created.  Each provider has their own approach and potentially identifying terminology. For my understanding, I wanted to try and establish a common way to describe how reciprocity providers are redefining ETL.  While imperfect, it will give me a common language to use, while also being a constant work in progress.

For most reciprocity providers, it starts with some ecompassing wrapper in the form of an assembly which describes the overall recipe, formula or wrapper that contains all the moving ETL parts.

Within this assembly, you can execute on workflows, usually in a single flow, but with some of the providers you can daisy chain together multiple (or endless) workflows to create a complex series of processes.

Each workflow has a defining trigger which determines the criteria that will start the workflow such as new RSS post or new tweet, and with each trigger comes a resulting action which is the target of the workflow, publishing the RSS post to a syndicated blog or adds the tweet to a Google Spreadsheet or Evernote, or any other combination of trigger and action a user desires.

Triggers and actions represent the emotional connections that are the underpinnings of ETL’s evolution into a more meaningful, reciprocation of value that is emerging in the clouds. These new providers are connecting to the classic lineup of ETL interfaces to get things done:

  • Databases
  • Files
  • Messaging
  • Web Service

While also providing the opportunity for development of open connectors to connect to any custom database, file, messages and web services. But these connectors are not described in boring IT terms, they are wrapped in the emotion and meaning derived from the cloud service--which could have different meanings for different users. This is where one part of the promise of reciprocity comes into play, by empowering average problem owners and every day users to define and execute against these types of API driven agreements.

All of these actions, tasks, formulas, jobs or other types of process require the ability to plan, execute and audit the processes, with providers offering:

  • Scheduling
  • History / Logging
  • Monitoring

With data being the lifeblood of much of these efforts, of course we will see “big data” specific tools as well:

  • Synchronization
  • Data Quality
  • Big Data
  • Analytics

While many reciprocity providers are offering interoperability between two specific services, moving data and resource from point a to b, others are bringing in classic ETL transformations:

  • Reformat
  • Aggregate
  • Sort
  • Dedupe
  • Filter
  • Partition
  • Merge
  • Join
  • Split
  • Convert

After the trigger and before the action, there is also an opportunity for other things to happen, with providers offering:

  • Push
  • Events

During trigger, action or transformation there are plenty of opportunities for custom scripting and transofrmations, with several approaches to custom programming:

  • Custom Scripts
  • JavaScript
  • Command Line
  • API

In some cases the reciprocity provider also provides a key value store allowing the storage of user specified data extracted from trigger or action connections or during the transformation process. Introducing a kind of memory store during the reciprocal cycle.

With the migration of critical resources, many of the leading providers are offering tools for testing the process before live execution:

  • Test
  • Debugger
  • Sandbox
  • Production

With any number of tasks or jobs in motion, users will need to understand whether the whole apparatus is working, with platforms offering tools for:

  • Performance
  • Monitoring
  • Optimization

While there are a couple providers offering completely open source solutions, there are also several providing OEM or white label solutions, which allow you to deploy a reciprocity platform for your partners, clients or other situations that would require it to branded in a custom way.

One area that will continue to push ETL into this new category of reciprocity providers is security. Connectors will often use OAuth, respecting a users already established relationship with platform either on the trigger or action sides, ensureing their existing relationship is upheld. Beyond this providers are offering SSL to provide secure transmissions, but in the near future we will see other layers emerge to keep agreements in tact, private and maintain the value of not just the payload but the relationships between platforms, users and reciprocity providers.

Even though reciprocity providers focus on the migration of resources in this new API driven, cloud-based world, several of them still offer dual solutions for deploying solutions in both environments:

  • Cloud
  • On-Premise

There is not one approach either in the cloud, or on premise that will work for everyone and all their needs. Some data will be perfectly find moving around the cloud, while others will require a more sensitive on-premise approach. It will be up to problem owners to decide.

Many of this new breed of providers are in beta and pricing  isn’t available. A handful have begun to apply cloud based pricing models, but most are still trying to understand the value of this new service and what market will bear. So far I’m seeing pricing based upon:

  • Seat
  • Assembly
  • Tasks
  • Connections
  • Extension
  • Sync
  • Support
  • Training

Much like IaaS, PaaS SaaS and now BaaS, reciprocity providers will have a lot of education and communication with end users before they’ll fully understand what they can charge for their services--forcing them to continue to define and differentiate themselves in 2013.

One of the most important evolutionary areas, I’m only seeing with one or two providers, is a marketplace where reciprocity platform users can browse and search for assemblies, connectors and tasks that are created by 3rd party providers for specific reciprocity approaches. A marketplace will prove to be how reciprocity platforms serve the long tail and niches that will exist within the next generation of ETL. Marketplaces will provide a way for developers to build solutions that meet specific needs, allowing them to monetize their skills and domain expertise, while also bringing in revenue to platform owners.

I understand this is all a lot of information. If you are still ready this, you most likely either already understand this space, or like me, feel it is an important area to understand and help educate people about. Just like with API service providers and BaaS, I will continue to write about my research here, while providing more refined materials as Github repos for each research area.

Let me know anything I'm missing or your opinions on the concept of API reciprocity.

from API Evangelist

Wednesday, February 27, 2013

Merging API Automation and Interoperability Into API Reciprocity

While I’m wading through dictionaries and thesauruses in an effort to find a more appropriate term “governance”, when looking at SOA governance through the API lense--I figured I’d flush out another area I’m working to define a term that appropriately describes automation and interoperability using APIs.

Yesterday I took a look at 31 backend as a service (BaaS) providers, in hopes of understanding more about what value they provide. Today I'm diving into the automation section of my new API trends area. While reviewing, I noticed the exact same companies that were under automation were also in interoperability. So I set out to find a new word to apply to this next generation of ETL providers that are building bridges between cloud platforms using APIs, as well as legacy data connections.

I have settled on the word reciprocity. The dictionary defines reciprocity as:

  • the quality or state of being reciprocal : mutual dependence, action, or influence
  • a mutual exchange of privileges; specifically : a recognition by one of two countries or institutions of the validity of licenses or privileges granted by the other

When you look in the thesaurus, reciprocity has a definition of "interchange" with synonyms of cooperation, exchange, mutuality and reciprocation. Reciprocity is also a synonym of connection with a definition of “person who aids another in achieving goal”. With synonyms being acquaintance, agent, ally, associate, association, contact, friend, go-between, intermediary, kin, kindred, kinship, mentor, messenger, network, reciprocity, relation, relative and sponsor.  (i love that "kin" is a synonym too)

All of these terms apply to what I’m seeing unfold with this new generation of ETL providers. ETL is moving into the clouds, and out from behind the firewall, using the open web, cloud platforms and APIs, and now we have to rethink ETL, and make it accessible to the masses. Put it within reach of the everday problem owners.

What I really like about reciprocity is it describes the mutual relationship between users and the platforms where their data and information resides. Reciprocity describes the connection that has to occur between these valuable API platforms, and their users, in ways the ETL misses. ETL stands for extract, transform and load, very technical and programmatic responses to moving my valuable resources, assets and personal information between the cloud platform I depend on daily.

In the physical world we have reciprocity agreements between countries to ensure free trade and a healthy balance in world markets. This next generation of reciprocity platform providers will not just be extracting, transforming and loading data--reciprocity platforms will be the nervous system of the global API economy, moving valuable resources around the world while respecting relationships between users, developers and the platforms in a way that preserves maximum value for everyone involved.

Now under the trends section you will just see one section for reciprocity, that will house all 19 of this new breed of API service providers. We’ll see how things change, but for now I think reciprocity describes the value created by this new space, in a way that helps me communicate it to others.

from API Evangelist

What Is Better Word For Governance When It Comes To APIs?

There is a great post by Lorinda Brandon (@lindybrandon) of SmartBear on ProgrammableWeb today called Governance vs Innovation: Do They Have to be Enemies?.  She continues a conversation, from API Strategy & Practice around the word "governance", and how in the API space, this is often considered a bad word.  

I was moderating the questions for Alistair Farquharson (@afarqu) CTO of SOA Software, when Irakli Nadareishvili (@inadarei) of NPR actually agreed that it is a bad word and asked if we need to evolve beyond it and not embrace it.  I totally agree with Lorinda's approach to walking the line between governance and innovation, and look forward to more discussion around how to analyze API vs SOA principles within the enterprise--striking a balance that works for each individual approach.

To help stimulate this conversation I wanted to break down the word, governance.  In the context of IT, governance is:

a concept used for activities related to exercising control over services in a service-oriented architecture (SOA). The definitions of SOA governance agree in its purpose of exercising control, but differ in the responsibilities it should have. Some narrow definitions focus on imposing policies and monitoring services, while other definitions use a broader business-oriented perspective

With general definitions being:

  1. the persons who make up a body for the purpose of administering something
  2. the act of governing; exercising authority

I think the illness around the word, when it is applied to APIs becomes clear when you start looking at the synonyms:

  • Governance - administration, authority, bureaucracy, command, control, direction, domination, dominion, empire, execution, executive, guidance, influence, jurisdiction, law, ministry, patronage, political practice, politics, polity, power, powers-that-be, predominanc
  • Management - administration, care, charge, command, conduct, control, direction, governance , government, guidance, handling, intendance, manipulation, operation, oversight, rule, superintendence, superintendency, supervision
  • Regulation - adjustment, administration, arrangement, classification, codification, control, coordination, direction, governance , governing, government, guidance, handling, management, moderation, modulation, reconciliation, regimentation, reorganization, settlement, standardization, superintendence, supervision, systematization, tuning

Let's separate the potentially bad synonyms:

  • administration
  • authority
  • bureaucracy
  • command
  • control
  • domination
  • dominion
  • empire
  • power
  • politics
  • control
  • government
  • manipulation
  • oversight
  • standardization

And the good synonyms:

  • direction
  • execution
  • guidance
  • influence
  • care
  • operation
  • arrangement
  • coordination
  • standardization
  • tuning
  • modulation
  • power

I feel both power and standardization could be both good or evil, so I put in both categories.  But overall you can begin to understand how governance is viewed in negative light in the API approach.  People see it more about power and control, rather than the healthy aspects introduced API innovation.  

In my opinion, this word reflects on why API works in more scenarios than strict SOA.  Too much control can stifle innovation, and if you are controlling for all the wrong reasons, it can be even worse.  APIs require a certain amount of oxygen to grow and evolve before it can achieve a level of innovation, which is an aspect some business leaders have trouble grasping. 

Irakli and I both agreed that we need a new word, something that reflects the bottom up of API, not the top down control of SOA.  Any ideas for a word that can keep be applied in a healthy way, but can institute many of the positive aspects of governance, without the potential innovation killing power and control?

from API Evangelist

Tuesday, February 26, 2013

75 Features From Across 31 BaaS Providers

I’m currently tracking on 31 backend as a service providers, in an effort to better understand how this new breed of platforms are helping developers build web and mobile apps. After looking at all the BaaS providers, there are 13 clear leaders:

Then there are another 18 other players, trying to play catch up in a space that is working hard to define itself in 2013:

My goal is to better understand what features are offered across these 31 BaaS providers. To accomplish this, I spent no more than an hour per provider looking through their sites and playing with their products to get at least a basic understanding of their offerings.

When looking for features I tried to standardize the best I could, but it is difficult when there are different approaches to the deployment of resources on each platform. I found about 75 distinct features being offered across the 31 BaaS providers. I’m sure there are other features, and vital details missing, but I wanted to start somewhere. Here is what I found, organized as best I could:

User Management

  • User
  • User Roles
  • LDAP

Content Management System (CMS)


  • Table
  • Relational
  • Key Value
  • Browser
  • MySQL Connector
  • PostGres Connector
  • Oracle Connector
  • Caching
  • XML
  • CSV

File Management

  • Storage
  • Sync

Image & Photo Management

  • Storage
  • Gallery & Collections
  • Processing

Custom Code / Objects

Programmatic Interfaces

  • Web Service Connectors
  • Custom REST API
  • Query


  • Product Catalog
  • Shopping Cart

Virtual Commerce

  • In-App Purchases
  • Custom Virtual Store 
  • Virtual Goods Management 
  • Currency Maintenance 
  • Virtual Economy Regulation

Other Monetization

  • Promotions
  • Subscriptions
  • Billing
  • Passbook


  • Recomendations
  • Reviews
  • Ratings
  • Likes



  • SMS
  • Email
  • Email Templates
  • Push Notification
  • Interactive Voice Response (IVR)
  • Messaging System

Calendar Events



Shared Links


  • Spatial
  • Location
  • Check-In
  • Places


  • Players
  • Ranking
  • Scores
  • Boards
  • State

3rd Party Integration

  • Twitter
  • Facebook
  • Dropbox
  • Fitbit
  • Foursquare
  • Github
  • Instagram
  • LinkedIn
  • Meetup
  • Tumblr
  • Withings
  • Wordpress
  • Yammer
  • Twilio
  • Underscore
  • SendGrid
  • Moment
  • Mandrill
  • Mailgun
  • CrowdFlower
  • Google Places
  • Google Apps
  • Salesforce
  • SAP
  • Siebel
  • Wordpress



  • Performance
  • Scaling
  • Load Balance


  • On-Premise
  • Virtual Private Cloud
  • Public Cloud


  • Sandbox
  • Production


  • Logging
  • Backups
  • Clients
  • Jobs


These BaaS providers support a wide variety of mobile devices, platforms, frameworks in multiple languages:

Mobile Devices

  • iOS
  • Android
  • Windows
  • Blackberry

Reader Devices

  • Kindle

Mobile Platforms

  • PhoneGap
  • Titanium

App Frameworks



  • Temboo


  • JavaScript
  • Java
  • C#
  • PHP
  • Python
  • Ruby

There were many different ways the BaaS platforms provided support to its developers:


  • Phone
  • Web
  • Chat
  • Dedicated Account
  • Dedicated Tech

I found 10 different ways that BaaS providers delivered pricing:


  • API Calls
  • Push Notification
  • Bandwidth
  • Storage
  • Active Users
  • Analitics
  • Support
  • App
  • Synchronization
  • Features


You can view all 75 features at the BaaS Github Repository I setup. Let me know any that you feel are missing, and I’ll consider adding.

Next up, I will add the features into my BaaS tracking database and publish a breakdown of providers, with the features they offer. Letting people search and filter, and also open up to each BaaS provider to comment and submit additional features they offer.

from API Evangelist