Doshii CTO, John Twomey, has been working on a blog series covering all aspects of API Integration. The first articles of the series have been released. For more on this series make sure to follow us on Facebook, LinkedIn, and Twitter, or .
API Integration Security…coming soon
Integration Monitoring…coming soon
What is API Integration
Application Programming Interfaces (API’s) provide a documented pathway for accessing the features and functionality of third-party systems through a (typically) stable and controlled interface. By leveraging API’s effectively, a developer can quickly and easily integrate with or expand upon a preexisting external system.
API Integration Background
Prior to the widespread adoption of web-hosted REST/HTTP API’s, service-oriented architectures, and other supporting aspects, integrating with a third-party solution could be a mammoth undertaking. In the worst-case scenario, it could mean attempting to reverse-engineer a way to communicate with the system using disassemblers, packet sniffers, and similar tools. Often this would mean dealing directly with the solution’s inner workings, with no guarantees that its core business rules would be enforced should a developers’ code accidentally break them. Often the next update to the software being integrated with would break some or all of the integration, requiring hasty and sometimes extensive revisions.
Such is the developer’s world in the absence of a published API. And such is the importance of having, and being able to effectively use an API. What an API brings to the table is an officially supported, documented, (hopefully) stable, and (hopefully) safe roadmap for integrating with an external system. APIs provide access to a service through a consistent interface, regardless of the technology being adopted by the client
Gone are the disassemblers and the packet sniffers, and the need for the developer to worry about any of the system’s business logic, security constraints, or other internal machinations. If the API is well-managed (not every API is), breaking changes should be a rare occurrence that comes only after plenty of warning. In essence, an API frees a developer to focus purely on their own solution and allows them to talk with external systems in a safe, secure, and maintainable way.
The importance of being able to understand and work with third-party API’s had only increased as technology has trended towards a more cloud-based, SaaS-oriented model. Even with small projects, giving users the option to authenticate using their Facebook, Google, or Twitter account is nearly a de rigueur requirement these days, with far deeper integrations hardly uncommon. And with a huge budget and a large team of world-class hackers maybe you could find your own way into the massive and extremely complicated systems operated by these top-tier technology companies. But why would you bother, when every one of these companies provides a public API supporting authentication (amongst a myriad of other things)? The only answer is that you wouldn’t, and it would be crazy to even try.
Instead, you would check the published API’s, and quickly discover that all three companies have authentication API’s standardized around something called OAuth2. And you would integrate using that, confident that your integration will be secure, stable, and maybe even flexible enough to accommodate any other OAuth2 provider out there. And after that, the sky’s the limit. Want to integrate with Slack? There’s an API for that. Or maybe Jira? There’s an API for that as well. Or maybe you just want to sell some stuff from Amazon or stream some cool music. All of that can be accomplished, and in every case, it’s accomplished by using a published webservice API. So, as we move into the modern era of distributed, cloud-based solutions, the answer to almost any question is “there’s an API for that”. Being able to navigate, understand, and leverage those API’s effectively is key.
With Doshii this means that:
- No matter how many businesses you connect with through Doshii, you only need to maintain one integration
- You can be sure of a stable Interface where any changes will be announced in advance and backward compatibility is of paramount importance
- Detailed documentation of an existing standardised connection
- Simplified process for development and integration
- An experienced API Development team available for support, consultation, and implementation
- Allow your development team to focus on product development rather than integration and maintenance
- You have confidence in a trusted, secure connection
- And a Dashboard to monitor all your integrations and help troubleshoot issues
API Integration Basics
APIs can come in all shapes and sizes. However, most successful APIs follow similar patterns in terms of structure, security, data formats, and protocols. Understanding these key aspects can be the difference between a successful API integration and hours of frustration.
Features of an API Integration
While there exists a nearly endless series of options with respect to how to create an API Integration, most webservice APIs will be defined by a handful of key features.
Most typically, a webservice API will be accessible over HTTP. This isn’t a given as technically anything accessible over the Internet may be classed as a “webservice”, however it’s generally a safe assumption. Webservice APIs that use any other strategy are quite rare, outside of the embedded-device space.
Slightly more common is layering a higher-lever protocol on top of HTTP, such as SOAP. While once quite common and still quite relevant in the .Net sphere, this strategy has been gradually losing ground to smaller and lighter APIs that run directly over HTTP
The most common approach to structuring a HTTP-based webservice API is called the RESTful paradigm. A significant topic in and of itself, in essence, this approach means that different HTTP Verbs are used to perform different operations. For instance, you might need to use a ‘POST’ request to create something, a ‘PUT’ request to the same URL/endpoint to update it, and a ‘DELETE’ request (again to the same URL) to delete it.
How closely a wbservice follows this approach varies widely. Some keep it formal and expect the integrator to be fluent with their ‘PUTS’ and ‘POSTS’, ‘GETS’ and ‘DELETES’. Others may take a more middle-ground approach and follow a simplified “use ‘GET’ to read, and ‘POST’ to write” pattern. Others may be more laissez-faire about it and take an “all (or most) operations can be performed using both ‘GET’ and ‘POST'”, leaving the integrator free to do whatever is more convenient on their end.
Generally, any pattern is fine, so long as there is one. A quality API will not stipulate arbitrary REST-related requirements with no discernable rhyme or reason.
Although APIs have used (and will continue using) a variety of data formats, with public webservice APIs XML and JSON are the dominant players by a wide margin. Although XML was established first and adopted into pretty much everything back in its heyday, it has been rapidly losing ground to the lighter-weight and more human-readable JSON.
Many APIs will allow an integrator to choose whether they want to receive their response in XML or JSON. Where only one format or the other is offered, it’s often possible to judge the age of an API by the data format it returns; XML tends to be preferred by older webservice APIs, and JSON is more common in newer APIs. Some high-level protocols, such as SOAP, may essentially dictate that one format or another be used.
In general, however, if you’re comfortable working with XML and JSON you’ll have no problems talking with most published webservice APIs.
For a modern webservice API integration SSL is the de facto minimum standard when it comes to security. Some APIs may layer additional security on top of this, such as request signing and other cryptographic techniques to verify the origin of the request and guarantee that its contents haven’t been tampered with. For instance, Amazon operates all of their APIs over HTTPS and in addition to that requires API consumers to implement a fairly involved request-signing algorithm.
As a general rule, however, SSL is “secure enough” for an API integration and represents the common starting-point for most webservice APIs. Requiring additional security over and above what SSL offers is somewhat rare, and often costs more in frustration than what it gains in added security.
While increasingly uncommon, it’s not unheard of for a webservice API to operate over plain HTTP with no encryption or other forms of security. This may be okay, but only if the API is neither receiving nor transmitting any personal user data or other sensitive information (including API access/secret keys!). If the API contains any sensitive information, you should not use HTTP!
Other Important Bits
While not part of the API itself, documentation is also of critical importance to an integrator. In the absence of documentation, connecting to an API is all but impossible. Without quality, consistent, up-to-date documentation the task can easily become a frustrating exercise in trial-and-error.
High quality documentation that uses consistent, intuitive terminology to explain concepts and important aspects of an API can be a critical factor in a successful integration. In the event you’re attempting an API integration and you discover that important API documentation is incomprehensible, inaccurate, outdated, or just plain missing, it’s a wonderful opportunity to take a step back and see if maybe there’s an alternative API available that can be used to accomplish the same task.
That’s why here at Doshii, we’ve created a detailed API integration documentation hub, which we constantly keep updated with guides for integrating with our platform. Check it out here
Putting it Together
Success as an integrator depends upon having a good understanding of the defining features of a modern webservice API. A solid knowledge of these basics will help when assessing the relative strengths and weaknesses of different APIs, so that the best candidate can be chosen and wasted time and frustration can be avoided. Combined with quality documentation this should help you quickly develop a clear understanding of
the API and identify any gaps between your own system and the API as early as possible so that any potential issues can be planned for and addressed.
To learn more about integrating with Doshii and to speak with our technical team, get in touch through our contact page.
Architecting Multiple integrations via one API
The complexities of integrating with third-party API’s can quickly multiply due to the rapid proliferation of competing API’s and standards. This significantly increases development, testing, and maintenance time/costs, but can be mitigated against by integrating with a ‘middleware’ API that delegates to various third-party services.
The Proliferation Problem
While the rapid growth and adoption of webservice API’s and standards have been a real boon for the software industry, nothing comes without a cost. The cost, in this case, is a rapidly expanding and constantly evolving frontier in which there are often multiple offerings in any given domain, several of which often proving desirable candidates for integration.
As we saw in the previous articles What is API Integration, and
API Integration Basics, the answer to “How do I do X or integrate with Y?” is almost always “There’s an API for that”. However in many cases, the answer actually comes back more along the lines of “There’s 5 or 6 different API’s for that, which one do you want to use?”, and then as an app developer or POS integrator you have to choose. What’s the one API you’re going to integrate with? Or do you invest the time and effort that it takes it integrate with several of them? And the extra time it takes to maintain those integrations?
Say you’re creating a solution that allows a user to browse and purchase digital music from within your application. Amazon offers an API that will let you do exactly that. And so does Apple. And so do Bandcamp, Spotify, Murfie, and others. Each one may offer some catalog items that the others don’t. And wouldn’t it be great if your solution could seamlessly integrate with all of them?
Certainly, it would! But the cost of doing so is very high. Although each API ostensibly performs the same function, each one is independent and structured differently from the others. Where one codepath should logically suffice, you’re going to need five. Each one is going to have its own credentials, its own authentication strategies, its own API quotas and limits, and its own quirks that need to be smoothed over if you want your users to have a seamless experience. And once you’re done you’re going to have to track changes to each API and ensure your app is updated and maintained accordingly.
Thus the proliferation problem should be obvious. Unless you’ve got the development might of a Google, Apple, Amazon, or Microsoft there are more API’s out there than you can realistically integrate with. You’ve either got to settle for an integration with a single API and dealing with frustrated users upset that they can’t find any content from ‘API B’ within your solution, or else dive in and try to support as many parallel API’s as you think you can get away with.
A Better Solution
Imagine if instead of talking to a bunch of related API’s individually your application or POS system could talk to a single service that internally handled communication with each of those related API’s and then just shipped you the result. It might look something like this:
We’ve added a “middleware” layer that sits between your solution and the rest of the world. And now instead of talking to a whole bunch of completely different and differently-structured API’s your solution only needs to integrate with one API; the middleware’s. It’s connecting to the same amount of information, if not more, however, the complexity on the application/POS side has been drastically reduced. This carries with it a corresponding reduction in development, testing, and maintenance effort; or in other words, time and money.
You’ll notice that the middleware has a proper name. It’s called Doshii. Doshii provides API integration, management, and aggregation as a service. This frees up your time to focus on your actual product and saves you from having to worry about whether to integrate with one service or several and from the complexity and overhead associated with the latter option. And of course, it’s all accomplished through an API.
Strangely enough, it seems that the answer to the proliferation problem is yet again “There’s an API for that”. By leveraging a middleware API such as Doshii, your solution can gain the benefits of multiple parallel API integrations without any of the added development costs. There’s no longer a need to implement and maintain multiple codepaths that all accomplish the same logical task. And dealing with changes, updates, and maintenance as all of those API’s evolve? Well that all becomes Doshii’s problem; you don’t have to worry about it!
By placing a ‘middleware’ webservice between your solution and the webservices that it ultimately wants to communicate with, multiple integrations can be replaced with a single implementation and API proliferation becomes much less of an issue. The power and flexibility that this kind of solution provides are difficult to overstate.
API Integration Best Practices Overview
There are a number of challenges that must be overcome when integrating with multiple external APIs. This article discusses some common design practices and patterns for doing so.
The heart of any non-trivial application will be its data model. Ideally, you’re already using an ORM framework like Hibernate, ADO.NET, CoreData, ORMLite or one of a plethora of options for your particular technology stack, though whether you’re doing that or connecting directly to a relational database, or just writing your data out to some JSON or CSV (or proprietary format) datastore somewhere, some basic principles will always apply.
This first is that you should always model the state your application stores in terms of how your own business logic understands it. There are no bonus points for unnecessary complexity or convoluted relationships here. Just think about how your application needs to work with the data, what information it needs, and in what order it needs it, and go with the simplest design that works.
Once you’ve got a baseline data model that covers your core business requirements, it’s time to take a look at the external API’s that you’re integrating with and work out how they best fit into your model. The specifics will vary depending upon the features you’re providing, however at a high level what you want to do is annotate your baseline data model with additional fields provided by your external API’s. And you want to limit this strictly to those fields that are necessary for successfully communicating with your external API’s.
For instance, maybe an external API provides ‘Order’ fields like ‘orderId’, ‘purchaseDate’, and ‘revision’. In most cases ‘orderId’ will be the only field from that list that you actually need to add as a field in your data model. The rest is just noise that you’re better off ignoring.
Your business logic is the bridge between the persistent state within your data model and your application’s UI. It’s the glue that holds the app together and allows your end-users to accomplish useful things.
Having modular, extensible, and maintainable business logic is critically important to the long-term success of an application, particularly in the case of an app that relies upon multiple external API integrations.
When implementing your business logic, try to define logical operations based upon how your solution needs to work with or upon objects from its data model. Ensure that these operations broadly follow the principle of single responsibility; which is to say don’t have a ‘registerAccountAndPlaceOrder’ operation when you could instead have separate ‘registerAccount’ and ‘placeOrder’ operations.
Keep each individual operation as focused and concise as you can get away with, and allow more complex tasks to be defined as a series of operations. It may seem like a small thing, however putting in the effort up front to ensure that your business logic is focused, concise, and modular makes things much easier when the time comes to start wiring in external API’s.
There’s an adage in computer science that goes roughly along the lines of “any problem can be solved by adding another layer of indirection”, and at a fundamental level, this is correct in regards to the problems introduced by external API integrations. We’ve already touched briefly on the broad classes of problems and challenges typically encountered in this space, and now that your application has a fit-for-purpose data model and some well-structured business logic we can look at some common solutions.
Business Logic -> Data Model
The majority of external API’s that you integrate with will act as data sources for your application. You’ll pass them a search query and they’ll send back a list of matches, or you’ll tell them to perform an operation and they’ll send back some JSON or XML describing the result. In most cases, you need to keep track of this information.
The problem is, of course, that even within the same category/domain/problem-space, every external API you integrate with is going to have a completely different data format. Even when they’re all describing logically the same thing!
To solve this, the first layer of abstraction you’ll want to add is one that sits between your business logic and your data model. This layer should be able to accept as input the raw data from any of your supported API’s and produce as output a populated, normalized object ready for insertion into your database.
Which is to say, if your business logic implements a ‘placeOrder’ operation, and this operation delegates to an external API for actually placing the order, then you should have a layer that can accept the raw order data from the external API and return an ‘Order’ object suitable for insertion into your application’s database.
What this setup does is it frees your ‘placeOrder’ operation, your data model, and any of your business logic that later interacts with the ‘placeOrder’ result from having to know anything about the external API that was used to actually place the order.
This means you’re now able to switch to a different API provider and/or connect additional API’s without needing to modify any of these things. The required changes are limited solely to the translation layer sitting between your business logic and your data model, which is hugely powerful and gives your app a great deal of flexibility!
Business Logic -> External API’s
At this point, you’ve solved the problem of getting information from an external API into your local data model quickly and easily and without needing to have tons of API-specific code sprinkled around throughout your application. That’s great!
However many external API’s will require an object to be carried forwards across multiple calls as part of its business lifecycle. For instance, maybe your user wants to cancel that order that they placed using ‘placeOrder’. The external API will likely provide a ‘cancelOrder’ feature, but in order to work it’s going to need to receive the object that it originally sent to your ‘placeOrder’ operation, and of course, it’s going to expect you to send that object according to its internal requirements. So we’re not done just yet.
The next layer of abstraction that you’ll want to add to your app is one that sits in between your business logic and and external API’s it integrates with. This layer should be able to take as input an object from your data model (in your internal format) and an operation (like ‘cancelOrder’), perform that operation by making the necessary API call(s), and then return the API response as output.
With this setup, your business logic remains free to work purely against its own internal representation of its data. All the mapping and translation that’s required to convert an external API response to an internal data object (and vice-versa) is handled by additional layers designed for that one specific purpose.
This keeps your core application code clean and focused, and again makes it very easy for you to swap out or add additional API integrations as your requirements change. This is very powerful, and all comes thanks to the addition of two relatively straightforward layers of abstraction.
Rinse and Repeat
Done correctly, the steps above should allow you to easily connect to additional external API’s that fit in the same category as the one(s) you’re already interfacing with. To connect to a new category of API’s (for instance, to enable a completely new in-app feature), simply repeat the process as needed. This does add work, but if you focus on only implementing what you need and adding features only as required rework can be kept to a minimum and the impact of breaking changes to an external API can be mitigated, if not avoided entirely.
Also, note that there may be various supporting bits and pieces that need to be tacked on in support of the layers described above. In particular, this may include things like configuration, credential management, and request signing for each individual API. Where possible this should be abstracted out to its own layer (yet again), so that these functions are decoupled from your core business logic, allowing either to change independently with minimal impact to the other.
Doshii offers an out of the box solution that can provide your application many of the same benefits discussed above with only a fraction of the coding time, effort, and expertise required. We’re here to help so please don’t hesitate to get in touch for more information.