Synchronous and Asynchronous Service Integration

Integration of services is one of the most critical solution elements of systems based on Service Oriented Architecture (SOA). There are few influencing factors which can significantly impact system capabilities depending on the type of the integration used. The most critical factors include and they are equally relevant for both regular services and microservices:

Service changes

 

From time to time, we will make service changes that may require service consumers to change as well. We have to ensure that the selected  service integration technology requires this to happen as rarely as possible.
More service consumers down the road

 

If more service consumers will be using a service in the future, the service integration capabilities should offer an efficient integration solution that will minimize service and service consumer changes and keep performance in a range of business SLAs.
Implementation locality

 

When implementing integration interfaces for a service, its internal implementation details should be hidden internally and they should not be exposed to consumers of the service. If it is not the case, it means that if something is changed inside the service, we can break the service consumers by requiring them to change as well. That increases the cost of change and  we want to avoid that. It also increases the risk associated with the changes.
Communication

 

Should communication be synchronous or asynchronous? This choice will shape the overall service integration implementation. Synchronous communication blocks until the operation completes. With asynchronous communication, a service consumer does not wait for the operation to complete before returning.
Stateful vs. Stateless

 

While the synchronous integration mostly supports stateful services, the stateless services are mostly supported by the asynchronous integration. If the statefulness has to be supported by the asynchronous integration it may increase the complexity of the solution and data replication
Complexity of service interactions

 

The complexity of the services’ interactions can further complicate overall integration especially in a case of the asynchronous integration.
Performance

 

Performance will be especially critical for the real-time processes with direct interactions with clients. The integration types supported by the services in the real-time processes can significantly impact performance during the peak hours of operations.
Technology agnostic APIs

 

We want to ensure that the service APIs used for communication with consumer services  are technology-agnostic. This means that the integration technology that dictates what technology stacks we can use should be avoided.
System breakdowns There is a crucial question about the system availability if some of its components go down. It is acceptable for some types of applications to continue functioning in a partially available state if some of its services fail.

 

Some Facts About Synchronous and Asynchronous Communications

Since it is straight forward with the synchronous communication to find out whether things completed successfully or not, the synchronous communication can be easier to reason about. However, since the synchronous communication blocks until the operation completes it can increase computing resources consumption and slow down performance during the peak periods.

Asynchronous communication is useful with the following types of executions:

  • It works very well when we need low latency since it does not block a call while waiting for the result.
  • It also works well with long-running jobs that would otherwise keep a connection between the consumer and a service open for a long period of time what would cause inefficient utilization of resources.

These two different modes of communication enable two different types of collaboration between the system components:

  • request/response or
  • event-based.

With the request/response collaboration, a client sends a request and waits for the response. This type of collaboration aligns well to synchronous communication. However it can also work for asynchronous communication. For example, a consumer can send a request to a service and register a callback, asking the service to send a corresponding response when the operation has completed.

With the event-based collaboration pattern, system components raise events when things change. Other components can listen to events and react to them. The event-based collaboration supports a very loose coupling between services and its consumers. The addition of the new consumers of a service and under assumption that the added consumers can listen to the events, would be pretty straight forward. This type of the collaboration keeps each service simple in a context of its integration. Generally event-based collaborations have advantages such as loose coupling, simplification of the system components, better performance, and it is more resilient to breakdowns. However at the same time it can increase the complexity of the interactions and data replication in a case of stateful services since this type of collaboration is stateless.

 

Service Integration Matrix

Critical Factor

Preferable Integration Type

Service changes Asynchronous

Service decoupling is the property that is supported by the asynchronous integration. Decoupling minimizes impacts caused by service changes.

Additional service consumers Asynchronous

Decoupling property of the integration becomes important when more service consumers start to use the service. The less coupling is involved the simpler and more cost effective integration solution will be implemented.

Implementation locality Asynchronous

Since the asynchronous integration generally hides internal details of the service implementation it will better support the implementation locality.

Stateful Synchronous

The synchronous integration is the best fit for the stateful services. If the statefulness has to be supported by the asynchronous integration it would make implementation more complex and also increase data replication.

Stateless Asynchronous

The stateless services are mostly supported by the asynchronous integration.

Complexity of service interactions Synchronous

More complex service interactions will make asynchronous integration less affordable option. Synchronous integrations simplify overall solution when more complex service interactions are required. However the performance aspect of the solution has to be considered and we should analyze the weight of the solution complexity factor vs. performance since the synchronous integration with complex interactions can significantly impact performance.

Performance Asynchronous

Asynchronous integration generally  provides better performance than synchronous integration.

Technology agnostic APIs Asynchronous

The technology agnostic APIs are better supported by the asynchronous integration since it minimizes exposure of the service internal details and enables either loosely or fully de-coupled services.

System breakdowns Asynchronous

A system that uses asynchronous integration is more resilient to breakdowns because its components function in a more independent mode. In some cases that can be acceptable but we have to be carefully about how much the currency of data (up-to-date information)  is required in a process using the service.

Advertisements

MEAN and Full-Stack Development

JavaScript followed by Node.js enables a single language use across all application layers. Before this change emerged few years ago, we had fragmented technologies and separated teams of designers and developers working in these fragmented technology fields in order to build applications. The “JavaScript everywhere” enabled appearance of the full-stack frameworks that bring common modules from different technology layers together in order to build software in a fast and more agile way making it more efficient solution for frequently changing and highly scalable systems.

Full-stack development is about developing all parts of the application by using a single framework. It includes back-end which belongs to the database, middleware where the application logic and control reside and the last but not the least part is the user interface.

MEAN is a JavaScript and Node.js full-stack framework comprised of four main technologies:

You will need some time to learn all technologies involved in MEAN but it will be rewarding and professionally exiting. At the same time, a single language, JavaScript, is used through the framework and all parts of the application can user and/or enforce Model-View-Controller (MVC) pattern. MVC is fully data oriented. Model holds data, controller processes data and view renders data. Data marshaling is done using JSON so that the serialization and deserialization of data strictures are not needed.

The big advantage of the full-stack framework is that it has a holistic approach that looks at the system as a whole with all its components that exist on their own. In order to function together as the whole the system components have some interdependencies that need to be considered as well. These  interdependences should be minimized in order to properly support decoupling between the system components which is one of the most important aspects of the overall architecture of the system.

The framework is modular what means if tomorrow some of the components become obsolete it can be replaced with the new component. It would require some changes with some of the dependable components but they should be minimal.

The full-stack approach gives you better overall control since it helps the different parts work seamlessly together since they are built by a single developer or a small team of developers. This also supports microservices way of service design and implementation especially for systems that change frequently and/or have to be web scalable. The disposable services are the way to go in these kind of environments.

This post contains overview of MEAN applications, MEAN technologies, and MEAN architectural patterns.

If you are interested in additional details about the MEAN architectural patterns, Getting MEAN with Mongo, Express, Angular and Node book authored by Simon Holmes is a good source of information.

MEAN Applications

There are two types of MEAN applications:

  • Server Applications
  • Single Page Applications (SPA)

With a server application, each user request is routed through Express. Express finds out from its routes which controller will handle the request. This is the same process for each user request. This application type supports one-way data binding.  Node.js gets the data from MongoDB, and Express then compiles this data into HTML via provided templates and finally the HTML is delivered to the server. This implies that most of the processing is done on the server and browser just renders HTML and runs JavaScript if it is provided for interactivity.

MEAN-Architecture-NodeExpress_NEA

With SPA, the application logic is moved to the front-end away from the server and that is why it is called Single Page Application (SPA). The mostly used JavaScript frameworks for SPAs are AngularJS, Backbone and Ember. MEAN uses AngularJS. While this approach has its pros and cons, it is obvious that moving the application processing from the host (server) to the users’ browsers will lower the load on the server and network and bring the cost down. In some cases it will also improve the performance of the application. A browser sends an initial user’s request to the server and server returns AngularJS application with requested data. The subsequent user’s requests are processed most of the time by the AngularJS application running in the browser while data goes back and forth between the browser and server. SPA also supports two-way data binding where the template and data are sent independently to the browser. The browser compiles the template into a view and the data into a model. The view is “live” since it is bound to the model. If the model changes the view changes as well and if the view changes then the model also changes.

MEAN-Architecture-Angular_SPA

MEAN.IO and MEAN.JS are full-stack frameworks for developing MEAN-based applications.

 

MEAN Technologies

MEAN includes five main technologies:

  • MongoDB database and Mongoose object data modeling (ODM) tool
  • Express middleware
  • AngularJS front-end
  • Node.js server platform

MongoDB is a NoSQL document-based database management system which data model includes:

  • Collections
  • Documents
  • Fields
  • References

M2_Model

A Collection is a top model element. Each model can have one or more collections. Collections are analogous to tables in a relational database. Each collection contains documents that are analogous to records in the relational database. Collections model one or more concepts (e.g., account, user, order, publisher, book, etc.) the data is based on.

Documents are JSON-like data structures containing fields that have values of different types (e.g., String, Date,  Number, Boolean, etc.). A value can also belong to another document or an array of documents embedded in a document. Documents can have different structures in a collection. However, in most cases in practice, collections are highly homogeneous.

Fields are analogous to columns in the relational database. The field/value pairs (better known as key/value pairs) construct document’s structure.

MongoDB resolves relationships between documents by either embedding related documents or referencing related documents.

Mongoose is a MongoDB object data modeling (ODM) tool designed to work in an asynchronous environment. Besides the data modeling in Node.js, Mongoose also provides a layer of CRUD features on top of MongoDB. It also makes it easier to manage connections to MongoDB databases and perform data validations.

Express is a middleware framework for Node.js that abstracts away some common web server functionalities. Some of these functionalities include session management, routing, templating, and others.

Node.js is a foundation of the MEAN stack. Node.js is not a language. It is a software platform based on JavaScript. You will use it to build your own web server and applications that will run on top of it. Node.js applications when codded correctly are fast and they efficiently use system resources. This is supported by the core Node.js feature that it is single-threaded and executes a non-blocking event loop.

The web server running on Node.js is different from traditional multi-threaded web servers (e.g., Apache, IIS, etc.). The multi-threaded servers create new thread for each new user session and allocates memory and other computing resources for it. During the peak periods when many users access the server concurrently its resources can get exhausted in which case the system could halt its operations until the load decreases and/or more machines and resources are added. The precocious approach that many systems take is to often overpower the servers even if they do not need so much resources most of the time.  This definitely increases the cost of system operations. When Node.js is used, rather than giving each user a separate thread and pool of resources, each user joins the same thread and the interaction between the user and thread exist only when it is needed. In order to ensure that this approach works Node.js supports non-blocking by making blocking operations run asynchronously.

While you can use Node.js, Express and MongoDB to build data-driven applications, the use of AngularJS will bring more sophisticated features to the interactivity element of the MVC architectural pattern supported by MEAN. AngularJS puts HTML together based on provided data. It also supports two-way data binding by immediately updating the HTML based on changed data and also by updating the data if HTML changes.

 

MEAN Architectural Patterns

When you create MEAN-based applications, you can choose any of the architectural patterns or a combination of the architectural patterns (hybrid architectural patterns) listed here.

MEAN architectural patterns are based on the Model-View-Controller (MVC) pattern.

The MVC pattern is data oriented. Model holds data, Controller processes data and view renders data. There is also a route component between the controller and users’ browsers (Web). The route component coordinates interactions with the controller.

MVC.png

A common way to architect MEAN stack is to have a REST interface feeding a single page application (SPA).  REST interface is implemented via REST API that is built with MongoDB, Node.js and Express and SPA is built with AngularJS that runs in browser.

REST API creates a stateless interface to your database. It enables other applications to work with your data. There is also one more important technology component, Mongoose, that is a liaison between the controller and MongoDB.

Mongo-Mongoose-Express-Angular-Communications.png

MongoDB communicates with Mongoose only. Mongoose communicates with Node.js and Express and AngularJS communicates with Express only.

The REST API is a common architectural element used in all MEAN architectural patterns.

The following architectural patterns are enabled by the MEAN framework:

  • Node.js and Express Application (NEA)
  • Node.js and Express application with AngularJS addition for better interactivity (NEA2)
  • AngularJS Single Page Application (SPA)
  • Hybrid Patterns:
    • NEA and SPA
    • NEA2 and SPA

Node.js and Express application (NEA)

HTML and content are directly delivered from the server. The HTML content requires data that is delivered via REST API. REST API is developed with Node.js, Express, Mongoose and MongoDB.

MEAN-Architecture-NodeExpress_NEA

Node.js and Express Application with AngularJS Addition for Better Interactivity (NEA2)

If you need a richer interactive experience for your users, you can add AngularJS to your pages.

MEAN-Architecture-NodeExpressAngular_NEA2

AngularJS Single Page Application (SPA)

In order to implement Single Page Applications, AngularJS is needed.

MEAN-Architecture-Angular_SPA

Hybrid Patterns

The three above listed architectural patterns can also be combined into hybrid architectural patterns. The two most common combinations are:

  • NEA and SPA
  • NEA2 and SPA

NEA and SPA

This pattern is for the applications that require combination of application constraints that are best supported by both NEA and SPA. For example, NEA best supported application constraints include: short duration of user interactions, low interactions, content rich, etc. SPA best supported application constraints include: feature-rich, highly interactive, long duration of user interactions, private, fast response, etc..

MEAN-Architecture-NodeExpress_NEA-SPA.png

NEA2 and SPA

Finally, NEA2 and SPA is like NEA and SPA with a bit richer interactivity on the server side (NEA2) via AngularJS addition.

MEAN-Architecture-NodeExpress_NEA2-SPA.png