Best Practices of Continuous Integration

• Code Repository : maintain a main branch in the Code Repository which stores all of the Production ready code. From this main branch we can deploy code to Production at any time.

• Build Automation : create build environment in such way that even with one command build can be triggered.

• Self Testing Build : Every build should be self tested, which means that with every build there is a set of tests that runs to ensure high quality

• Daily Commit to Baseline : developers should commit all of their changes to baseline on a daily basis. This is to ensure that large amount of code is not waiting for integration with the main repository for a long time. 

• Do Not Check In Broken Build : developers tend to check in broken build. And when the build breaks, the same developers who broke the code has to spend time to build it and get it working quickly.

• Always Commit Tests Locally : Running a commit tests locally is a sanity check before actually committing to the action. This will ensure that what we believe to work actually works. 
  
• Production Like Environment Testing :  maintain a pre - prod environment which is very close to or nearly like a production environment. Perform testing on this pre-prod environment to check for any integration issues.

• Publish Build Results : publish the build results on a common platform or site so that everyone can access and see these results and take corrective actions immediately.
• Deployment Automation :  automate the deployment process to the extent that in a build process we can add the steps of deploying the code to a test environment. On this test environment the latest delivery can be tested by all stakeholders

Artificial Intelligence impact on Retail Industry

Do we know how technology is making an impact on the retail industry as of now? The retailers have massive amounts of data about their customers, their shopping patterns and experiences and much more. These vast collections of big data are often too difficult for some of the average companies to drill down into and analyse properly. Therefore, large amounts of this valuable information are left to rot, resulting in very low conversion rates overall.

Nevertheless, AI technology may change that for these retailers by using this data to create web shops that take customer information and turn it into targeted shopping experiences, online chatbots that will easily answer questions and assist customers, and in-store intelligence to make the experience even more interactive.

Some of the Retailers are in the process of transforming their contact center to include ‘chatbot agent’ with an AI program determining which calls to route to chatbot and which ones to the actual agent.

Web shops that customise their marketing and items based on customers' previous purchases, search patterns & habits, number of clicks, age, gender, and other variables by using self-learning algorithms could take what once was a lost customer and make their experience far more intimate, targeted and effective. This allows retailers to make more sales online and also provide more items on their online stores. The algorithm could easily bring up items it feels the user would want, as a start.

Some of the Retailers are looking to use Machine Learning algorithm to determine how promotional campaigns will work to predict the success of the next customer touch point.

In addition to the automated personal assistant chatbot, a customer could receive targeted marketing campaigns when they shop. Current chatbots rely on particular keywords and phrases to generate responses. However, through advancements in machine learning, a whole new form of chatbots, which will adapt to your questions to answer more complex things and think for themselves, will be deployed on retail sites in the future.

The way forward is clear, we must find the means for Humans and machines to work together.

Some of the retailers has also begun to replace staff with automated machines to make consistent products and reduce the time to serve customers.

The retail sector is in an immense state of change and business transformation. The entire retail sector is trying hard to cope with the fast and ever-changing customer shopping habits, and the shift from high street to Web. We have seen some of the retail giants closing stores showing moving customer demand while some of the retailers have moved their majority business to Web / online.

Across the board, the retail sector has started investing heavily and innovated with various technologies including AI, Robotics, Supply chain, and logistics automation, Order Forecasting automation, Data analytics to become more competitive, ability to become more responsive to demand and opportunity.

Due to the transactional business, Retailers in general generate high volumes of data related to sales, cost of product, customer histories, new trends, reports, geographical data, weather information etc. All of this data both current and historical can be put to use to deliver functional business can be put to use to deliver functional business insights and informed decision-making, reduced time to marked for new products and services, and improve success rates for initiatives. Automating and informing decision-making through AI can help retailers determine what to order and when, what products to merchandise at the front of the store, cross-selling and up-selling opportunities to individual customers based on previous purchases and current basket contents.

The process of planning, procurement, making, distributing, selling and gathering customer experience can take more than a year in the retail sector. It’s a long lead time that limits response to fast-moving trends.

AI can expedite the process, reducing the time from ideation to sale, reducing indecision and informing trading decisions through historical data and trend analysis.

Majority of Retailers cite improving customer experience as their main motivation in investing in AI and Automation. The automating includes services, in-store and online to highly personalise the shopping experience. This involves creating one-to-one interaction using vast customer history and trend data. It’s not just at the storefront, we can see AI being used in the large warehouse based retail and fulfillment centers. AI, robotics and automation systems are being used to increased operating capacity and enable more orders being processed per hour. Its helping retail cutting cost by allocating more warehouse floor space to be used for product storage through reducing the shelving aisles.

The adoption of AI represents an exciting leap forward in the retail sector.

AI's purpose is not necessarily to replace humans, but rather to assist them and increase technology-based jobs worldwide. This is expected to increase job opportunities by expanding the job market to make way for AI. From the creation of more jobs with specific skills never offered before to customer experiences, AI is the future of the retail industry that has sat on customer data for too long and justifies a chance to put this information to good use at last.

Part 2 of series "AWS cloud components -simplified - The Power of Lambda"

Continuing my series on AWS cloud components simplified, here I have made a pictorial representation of how Lambda works within AWS cloud platform.

Below diagram should be read in conjunction with my previous blog on The Power of Lambda.

Part 1 of series "AWS cloud components -simplified - The Power of Lambda"

AWS S3 bucket:

This component can be used to receive files coming from external sources. The file is routed via API Gateway in order to balance load, firewall unprecedented request for security and serve as a proxy server.

AWS Lambda:

This will monitor S3 bucket in order to record landing of a file. It will then provision the event of incoming file in bucket to different tasks like reception and transformation.

An example:

- S3 bucket can be structured to contain sub-folders named pending, processing, received, renamed, transformed, validated, error, exception and archive.

- Incoming Files can be filtered and sent to Pending folder on file creation event (marks file arrival). This will also instantiate AWS lambda.

- AWS lambda can then store filename and status in the table with status=’Received’. It will also invoke EC2 container (reception task) for further processing.

- Reception task is a docker instance running on EC2. It will identify the file that is ready and read the file from S3 and process it.

- Post job in ACW to invoke ProcessorLambda.

- On successful processing, the last step is to post an ACW job to invoke SubmissionLambda

- ACW will trigger based on time event (will trigger on specific time as per configuration) and invoke ExceptionTask via ExceptionLambda

AWS Cloudwatch:

ACW will perform orchestration role. It will ensure start of different tasks based on completion of previous tasks.

AWS SNS:

SNS is primarily used as a notification service. It will serve as an interface to report errors during processing of a task. It will send a notification in order to log the error using already existent loggly service.

AWS EC2 cluster:

EC2 cluster will comprise of several dockerized tasks. These are reception, transformation, submission and exception.

AWS RDS:

We can make use of AWS RDS (postgres) in order to hold DB objects. Broadly it needs to hold DB objects to preserve any processing or transnational tasks.

Jenkins and Continuous Integration

Jenkins is an open source tool which comes with built in plugin for continuous integration purpose. Jenkins primary functionality is to keep a track of version control system and to initiate build and monitor a build in case of any changes. Jenkins monitors the entire process and provides reports and alert notifications.

For using Jenkins we would require the source code repository (example Git repository, PVCS, SVN), a working build script (example Ant / Maven script), which is checked-in to the code repository.

Jenkins captures any build failures during integration stage, automatically generates build report notification for every code commit changes, sends notification to developers about build report success / failure, achieves continuous integration and test driven development. With simple steps, maven release project is automated. Jenkins comes with built in plugin for continuous integration like Maven 2 project, Amazon EC2, etc.

Jenkin is mainly integrated with two components, version control repositories like GIT, SVN and build tools like Ant, Maven.

Within Jenkins, builds can be triggered by source code management commits, can be triggered after completion of other builds, can be scheduled to run at specified time, or by manual build requests

In order for Jenkins to do a clean build, it’s advised that developer perform a successful clean install on local machine with all unit tests successful. Only then the code changes must be checked into the code repository.

In case a Jenkins build failure, open the console output for the build and debug to see if any file changes were missed. If not able to find the issue that way, then clean and update your local workspace to replicate the problem and try to resolve it.

DevOps explained

DevOps is a cultural shift that merges development and operations. Apart from having skills of web languages such as Python or Java, the ideal DevOps team should have some experience using infrastructure automation tools like Chef or Ansible. Organisations also think about the essential interpersonal skills that make DevOps successful.

Instead of releasing big bang release of features, companies are now trying to see if small features can be delivered in short and regular intervals. This enables many advantages like getting quick feedback from customers and better quality of software and higher customer satisfaction. For achieving this, there is a need for increasing the deployment frequency, reduce the failure rate of releases, reduced time gap between fixes.

Some of the popular DevOps tools:
Git - for version control system tool
Jenkins - continuous integration tool
Chef, Ansible - configuration management and deployment tools
Docker - Containerization tool
Selenium - continuous testing tool
Nagios - Continuous Monitoring tool

Developers develop the code and this source code can be managed by Version Control System tools like Git. Developers check-in this code into the Git repository and any changes made in the code is committed to this repository. Jenkins pulls this code from the Git repository using the Git plugin and build it using tools like Ant. Configuration management tools like Chef / puppet deploys & provisions testing environment and then Jenkins releases this code on the test environment on which testing is done using tools like selenium. Once the code is tested, Jenkins send it for deployment on the production server. Post deployment it is continuously monitored by tools like Nagios

Docker containers provides testing environment to test the build features.

Version control is a system that records changes to a file or set of files over time so that you can recall specific versions later. Version control systems consist of a central shared repository where team members can commit changes to a file or set of file. Git has a major advantage it has over other VCS tools like SVN is that it is a distributed version control system. Distributed VCS tools do not necessarily rely on a central server to store all the versions of a project’s files. Instead, every developer “clones” a copy of a repository and has the full history of the project on their own hard drive.Git is a Distributed Version Control system (DVCS). It can track changes to a file and allows you to revert back to any particular change. There is a central cloud repository as well where developers can commit changes and share it with other team members.


Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

Developers check out code into their local work spaces. Once done with code changes, they commit the changes to the Version Control Repository. CI server monitors the repository and checks out changes as and when they occur. CI server then pulls these changes and builds the system and also runs unit and integration tests. CI server will inform the team of the successful build. In case of build or tests failures, the CI server will alert the team for fixing.

Success factors for Continuous Integration would include maintaining a shared code repository, automated build, making build self-testing, everyone commits to the baseline, every commit to the baseline should be built, test in a production like environment, all team members can see the results of the latest build, automated deployment.

Usage of Jenkins for CI: move a job from one installation of Jenkins to another by simply copying the corresponding job directory. Make a copy of an existing job by making a clone of a job directory by a different name. Continuous Testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with in the latest build. In this way, each build is tested continuously, allowing Development teams to get fast feedback so that they can prevent those problems from progressing to the next stage of Software delivery life-cycle. Automation testing is a process of automating the manual process to test the application/system under test. Automation testing involves use of separate testing tools which lets you create test scripts which can be executed repeatedly and doesn’t require any manual intervention.

Selenium supports two types of testing 1. Regression Testing: re-testing a code around an area where a defect was fixed. 2. Functional Testing: refers to the testing of software features individually.

Infrastructure as Code IAC is a type of IT infrastructure that operations teams can use to automatically manage and provision through code, rather than using a manual process.

Puppet is a Configuration Management tool which is used to automate administration tasks.
Puppet has a Master-Slave architecture in which the Slave has to first send a Certificate signing request to Master and Master has to sign that Certificate in order to establish a secure connection between Puppet Master and Puppet Slave. Puppet Slave sends request to Puppet Master and Puppet Master then pushes configuration on Slave

Chef is an automation platform that transforms infrastructure into code. Chef is a tool for which you write scripts that are used to automate processes.Chef Server is the central store of your infrastructure’s configuration data. Chef Server stores the data necessary to configure your nodes and provides search. Chef Node is any host that is configured using Chef-client. Chef-client runs on your nodes, contacting the Chef Server for the information necessary to configure the node. Since a Node is a machine that runs the Chef-client software, nodes are sometimes referred to as “clients”. A Chef Workstation is the host you use to modify your cookbooks and other configuration data.

Containerization: containers are used to provide consistent computing environment from a developer’s laptop to a test environment, from a staging environment into production.
Container consists of an entire runtime environment, an application, plus all its dependencies, libraries and other binaries, and configuration files needed to run it, bundled into one package. Containerizing the application platform and its dependencies removes the differences in OS distributions and underlying infrastructure.

Docker image are used to create containers. Images are created with the build command, and they’ll produce a container when started with run. Images are stored in a Docker registry because they can become quite large, images are designed to be composed of layers of other images, allowing a minimal amount of data to be sent when transferring images over the network. Docker containers include the application and all of its dependencies but share the kernel with other containers, running as isolated processes in user space on the host operating system. Docker containers are not tied to any specific infrastructure: they run on any computer, on any infrastructure, and in any cloud. Docker containers can be created by either creating a Docker image and then running it or you can use Docker images that are present on the Dockerhub. Docker containers are basically runtime instances of Docker images.

Docker hub is a cloud-based registry service which allows you to link to code repositories, build your images and test them, stores manually pushed images, and links to Docker cloud so you can deploy images to your hosts. It provides a centralized resource for container image discovery, distribution and change management, user and team collaboration, and workflow automation throughout the development pipeline.

Docker Swarn is native clustering for Docker which turns a pool of Docker hosts into a single, virtual Docker host. Docker Swarm serves the standard Docker API, any tool that already communicates with a Docker daemon can use Swarm to transparently scale to multiple hosts.

DevOps - Adopting Continuous Deployment

Continuous Delivery and Deployment, there is a difference between the two. When adopting DevOps, Continuous Delivery is a core capability to adopt. Continuous Deployment to Production is an optional capability that you may or may not adopt, based on your needs and constraints.

Then what does one do ‘deliver’ when adopting Continuous Delivery or Deployment? We shall look at it from the perspective of  People, Process and Technology.

As mentioned in my previous blog, DevOps is a cultural movement. The people aspect of DevOps is where it all begins. While adopting Continuous Delivery, the most important part is the culture of continuous delivery and continuous collaboration between all the stakeholders who need to be onboard to enable and consume continuous delivery. This includes Dev and Ops but also stakeholders across the SDLC including Business, Analysts, Product owners, Architects and Design, Information Security, Quality Assurance and Management. There is importance of  creating a culture where all these stakeholders contribute to the continuous delivery, and also accept the feedback that comes from the continuous delivery at every stage. Dev teams uses the feedback to decide what to work on for the next Sprint, Quality Assurance or testing teams uses it to test and validate functionality, integrations and performance. The Ops team determines how the environment performed and where it needs enhancement or corrections. Project management team will manage their project and release plans, and so on.

The change in culture is of continuous collaboration, communication, trust and working towards common business goals.

When continuously delivering software, one is not only validating the functionality and performance of the software being delivered and the environments it is being delivered to, but also the process of deploying the software. Deployment of the code involves code deployment, file transfers, configuration changes to operating systems, databases and middleware (SOA / OSB, ODI etc). It also involves an orchestration of steps. The middleware processes may need to be restarted after configuration changes. Services may need to be stopped before file transfers and then restarted. Hence, continuous delivery allows for these processes to be tested and refined to ensure that when it comes to the final deployment to production, it is not the first time the team is executing the processes. They are tested and proven to work in production like environments.

Deployment requires set of tools that can automate the deployments and ensure continuous delivery of all changes from one environment in the SDLC to the next – to Dev, to testing, to Performance testing, to SIT integration testing, UAT user acceptance, Pre Prod and Production.

The key here is to start continuous delivery from the start of the project, from Sprint 0 all the way thru the project. In the beginning, the deployments may be simple, to much more complex orchestrated deployments later in the project. Continuous delivering changes – application, middleware, configuration, data and environment – in small pieces, using the right automation tools, reduces risk by validating the automation, the deployment processes, the configuration changes, the environments being deployed to and of course, the application being deployed.