How to Architect E-commerce Solution for the Cloud using AWS or Microsoft Azure
Congratulations if have decided that all your new applications would now be on cloud, the easiest and laziest way to embrace cloud for your application based on cloud would be to take your application and use compute instance for your server needs (compute = processor or server), your data and file storage to object store (S3 or equivalent) and declare victory. Using this architecture does not mean that you are not on the cloud. You are on the cloud but you are not using the cloud in the most optimal way possible, optimal in terms of costs and resource usage. This is like using the proverbial cannon to kill only a mosquito. The good thing is with a little architectural skills you can use the cloud as a cannon to destroy most of if not all your IT operations problems.
With cloud you can easily get rid of your network, server and storage maintenance issues by simply hosting your applications on the cloud. Amazon Web Services aka AWS, Microsoft Azure and Google Cloud Platform aka GCP are the front runners for providing these services with a very long tail of other providers. However if you architect your solution to fully use the services offered by Cloud you can get rid of your database and midlleware maintenance and scaling issues also, and you will be easily able to upgrade and scale your application on demand. In fact you can also easily and cheaply build your application to mimic the applications of large and mature organizations by embracing more and more of cloud in your applications.
Lets take a look at an application scenario where we can fully leverage the power of cloud — an e-commerce application. Lets assume that we want to build an e-commerce application that can be accessed by an Android Phone, an iPhone, a mobile website, a desktop website and a tablet website. The site would need to integrate with various partners internal and external to take and fulfill the orders and also manage returns.
To build such an app you would need to develop various clients:
a. Android app
b. iOS app
c. Mobile app
d. A responsive or progressive web app
These clients should be able to work with the backend:
a. To capture and serve product related information
b. To upload and serve product images & videos
c. To capture and serve customer related data
d. To take orders, provide order status and process refunds
The backend then should be able:
a. Complete transaction with a payment gateway or gateways
b. Update the financial accounting system
c. Put into and receive information from an inventory system or systems
d. Put into and receive information from a fulfillment system or systems
e. Put customer related information into a CRM
f. Send notifications to customers & partners on order & account matters
g. Show recommendations to customers
h. Collect and process data for analysis
Apart from this your systems should have strong application security and when you have a lot of customers you should be able to serve them with information quickly. Must to have thing for the management would be to have a nice analytics dashboard and a tremendously nice to have things would be for the system to provide real time analytics as well so that the systems become intelligent and can do real-time marketing and fraud related activities without human interference.
Simplistic e-commerce application architecture
Based on our business requirements our simple e-commerce application architecture should look like this:
We can implement this architecture by building an Android App, an iOS App, a React based Javascript front end for Desktop and Mobile Web and have this connected through a node.js application that fetches you the product catalog, takes the order and gets your customers all the information about orders and account as and when required. You can store the product catalog and user and orders related information in a MySQL Database. If you are expecting your company to be wildly popular you may want the ability to deliver the product catalog to your customers super quick and have the ability to modify product information as and when you want. You can split your database into a product catalog database which you can store in a document database such as MongoDB or Couch DB and store your transaction related information like orders and customers information in MySQL database.
Signup and authentication
To signup and authenticate users into your e-commerce system you can store the user credentials and information in a MySQL database and use your node.js javascript code to manage the signup and authentication of users.
Search
Selecting a search algorithm for your product search on the site is dependent on the importance of search as a product discovery method on your e-commerce application. If you have few products and are not adding products every day or continuously, having a sophisticated search functionality may actually be a waste of time and resources, on the other hand if you have a large selection of products and you want customers to be able to accurately search for products you should have a sophisticated search engine that can understand customers intent better. Based on your need you can use full text search functionality of your database or use Elasticsearch or Solr.
For example a customer searching for a phone should find the phone most relevant to his search. If a customers searching for
- Android phone
- Oppo Phone
- OnePlus 64GB or
- OPPO A92
should get different search results and search results should be closest to what the customer has been searching for.
Integration with other systems
No application is an island so you would need to integrate your e-commerce applications with other systems such as payment gateway to take payments from customers, inventory management system to show customers available products, logistics and shipping to ship the order to customers and manage returns, CRM to manage the customer history and finance to manage all the financial information about a transaction conducted on your website. You can do all this by writing custom code for each application. The challenge you will face is that if your code fails then your order details may be lost leading to a customer satisfaction issue. It happens to Amazon so be assured that it can happen to you also. To overcome this you want a system where the disparate systems do not lose messages from other systems and all the interactions and interfacing occurs smoothly. You can build this using a Microservice architecture using a Messaging based system that uses messages to integrate disparate systems. While choosing the messaging system you have to ensure that the system runs all the time, does not lose any messages and if there are errors in messages the same should be alerted to the administrator. You do not want a system where the customer successfully pays for an order and the shipping system has no record of that order. A good choice would be between Rabbit MQ and Apache Kafka, both provide excellent message queues that allow you to integrate your e-commerce application to different systems. To get a better understanding and help you choose you can read this article Apache vs Rabbit MQ.
Notifications system
To keep your customers informed about everything that they are doing on your application like what are you doing about the orders, when is it shipped when is it arriving and so on, you need to send a large number of messages to the customer via WhatsApp, SMS/Text, e-mail and in-App, last I counted Amazon sent me 12 messages on different channels for a single order, this is when the order was shipped seamlessly. In case of delays and returns the number of messages increase exponentially. Add to that the need to authenticate the email and phone numbers of the customers again requiring another set of notifications. To cross-sell and up-sell to the customers you need to send a bunch of notifications both in-App and through e-mails, WhatsApp Messages and SMS/Text. This means you need a notification system that is able to send e-mails, WhatsApp Messages, SMS/Text and in-App notifications.
To enable all these notifications you would need to develop a robust and reliable notification system that sends e-mail, SMS/Text, Push Notification and WhatsApp messages, sometimes to the scale of millions of messages in a day (Assuming you become hugely successful).
Recommendation engine
A nice to have system on your website would be to show customers recommendations based on what they are looking at on the site, like Amazon’s the “customers who bought this also bought this section”, or based “order again” and here is your your browsing history.
You can build this recommendations engine using a MySQL database, by running queries on previous orders and showing the result of the query on the results page of the customer. This will work efficiently if your customers are few and they don't mind the slow loading times of the search query. A more efficient way would be to store all your order data in a Redis database. You can use Redis to duplicate all your orders data from transactional database to the Redis database and then use Redis Sets to find items most associated with this kind of cart and sort the items in the different orders in order to find the highest scores to make a recommendation. Once we have the items we can pull their details from the product catalog in the MySQL database or Document store to show to the customer. Using Redis instead of making recommendations through a transactional database would be faster and easier and execute.
Amazon also shows you your browsing history, and recommendations based on your browsing history, and they do this across multiple devices. So if you have looked at a book in your app, it will show you the same book in your browsing history on the browser and vice-versa. And it is able to do so in under 5 seconds. Based on your browsing history it also sends you recommendations on email and sends you recommendations on the app and website. This feature is slightly complicated to implement as it uses things like streaming data analysis, or analyzing and using data while the data is still in motion. In simple terms it means that we are analyzing the data while it is still being generated instead of storing it in a data warehouse or a data lake for analyzing it in the future and act on a group of data. This is like personalisation on steroids. Based on this streaming data we can recommend products, conduct fraud analytics and even train our AI models. The architecture for doing this is broadly -
We can use a combination of Kafka and Spark to achieve our goal of analyzing live or streaming data and making it actionable, where Kafka works as the message broker for events based and live data. The data received in Kafka can then be analyzed using Spark and converted into actionable information, such as data being passed to APIs, being visualized in a Dashboard or be used to send an Alert or notification and then eventually be stored in a long term data store for future analysis and as input for Artificial Intelligence and Machine Learning models.
Data Warehousing and analytics
Last but not the least for better management you need a system where you can analyze all the data coming in through various systems. You should be able to get a view of all the visitors coming in, time they are spending on the site, things they are buying or abandoning, products that are performing well, the marketing campaigns working and not working, the ads that are performing or not performing etc, analysis of the demographics of the customers and their performance and preferences by demographics. Add to that an analysis of the shipping and handling, and inventory, to understand products in stock, not in stock and the time taken to fulfill an order, so on and so forth.
With all the data collected you should be able to also created Machine Learning models and Artificial Intelligence models to better manage your business and merge this with human experience and expertise to create better advertising, marketing, shipping models.
We can create a warehousing and analytics sub-system using Kafka, Spark, HDFS and if the scale is pretty low using the normal RDBMS like MySQL. For Dashboard and Visualization we can use open source software like Superset or buy something like Qlikview or Tableau.
Designing the system for cloud
Now that we know the different components that we need to build and deploy we can go about developing the operational architecture of the e-commerce system. We know that we would need a Node.js server, a MySQL Server, Mongo DB, LDAP, Elastisearch, Rabbit MQ, Spark, Kafka, Tableau, LDAP etc. we can start by hosting instances of these application middlwares on Cloud providers like Amazon AWS and Microsoft Azure.
As you can see to achieve high availability and resilience we would need to host these applications on multiple virtual machines or EC2 instances to achieve high availability. That would be a lot of machines and for those machines we would need a large number of human resources to man those machines for availability, scaling and performance. Needless to say some of these machines would be running at very low capacities initially. Overall this becomes a very expensive proposition.
Now lets step back and think how we can leverage the cloud to be more than just network, storage and computing resource provider. How do we leverage it so that our operational overheads, costs and development times are greatly reduced while our solution and software capabilities increase substantially. As you can see from the graphics above these cloud providers do not just provide the compute resources, they also provide these platforms and softwares as a service. So instead of hosting our own MySQL server and MongoDB and managing its availability and scaling, we let Amazon or Microsoft do it and we just use these database services (like Amazon RDS MySQL, Amazon Dynamo DB, Azure Database for MySQL, Azure Cosmos DB) in our application. Similarly instead of building our own authentication subsystem we use Amazon Cognito or Azure Active Directory B2C to signup users, and manage their sessions. For search we don’t build our own instead using Amazon CloudSearch or Azure Cognitive Search. For integration instead of building our own highly available Rabbit MQ / Kafka based integration we can build using Amazon SQS or Azure Services Bus, which does everything we need at scale. For notifications we can use the best in class service available from Amazon — Amazon SNS which can take care of SMS/Text, e-mail and In-App notifications, building this on your own would mean integrating different vendors and managing a large number of moving parts to achieve the same functionality and reliability. Microsoft Azure does provide equivalent service through Azure Notifications Hub and Azure Services Hub but you would have to integrate third party providers such as Sendgrid and Twilio to send notifications through e-mail and SMS/Text. Lastly for building e-commerce recommendations engine and doing data warehouse, analytics and analytics based marketing you can use Amazon Pinpoint (For events and analytics based marketing, Azure does not have an equivalent product), Amazon Redshift or Azure Data Lakes Analytics (for data lakes, data warehouse), Amazon Kinesis & Amazon Athena (for recommendations engine and analytics) and Amazon Quicksight (for Dashboard and KPIs management).
As you can infer, for all your simple and complex technical requirements you can use the cloud platform components provided by AWS and Azure to run your e-commerce application, and let them manage the operations and running of the infrastructure while you focus on development and strategy. In fact if you repurpose your code to run on AWS Lambda or Azure Function you can host your entire application serverlessly.
