Reduce food wastage with IoT Solution

Ethylene gas is produced by most plants, which use it as a hormone to stimulate growth & ripening. Fruits and flowers under stress can overproduce ethylene, leading them to ripen prematurely. 

Every year supermarkets lose considerable amount of their fruits and vegetables to spoilage. 

Now sensors are available that can detect ethylene gas that indicates when fruits ripen, which could be useful in preventing food spoilage. Once these sensors are integrated with IoT cloud, supermarkets will have the data on the expected impact on produce quality, as well as sensors integrated with IoT can provide alerts to ensure that products are handled & cooled correctly. Based on the data received in IoT, insights can provide dashboards/reports/alerting that enable smarter decisions to maximise shelf life of produce (longer lasting freshness for bananas) and improve the consumer experience. 

A data driven solution (ethylene sensors integrated with IoT) can expand the availability of fresh, high quality produce and minimise food wastage.

IoT and opportunities for Analytics, Artificial Intelligence & Machine Learning

IoT is not just for establishing connections between devices and systems. IoT is opening up opportunities for Analytics, AI Artificial Intelligence and Machine Learning ML. 

IoT tools can act on the numerous data sources available from various devices, sensors, and systems. Organisations can have comprehensive view of their customers’ requirements in a way that wasn’t previously possible and can then use those insights to improve the overall customer experience.

A great example would be, IoT has the potential to transform the way Healthcare is delivered. Patient’s vitals monitoring device such as Glucose Levels, Blood Pressure & Heart-Rate are being connected to the IoT which then enables the capability to share real-time patient information. Analysis of the data obtained from patient monitoring devices can be used to provide inputs to patient control devices like ventilators, defibrillators, etc.

- Use of IoT sensor enabled cameras at Airports to detect passenger queues at various check points

- Use of IoT sensors at Shopping centers to detect broken escalators, dirty washrooms

- Using IoT sensors, Supermarkets in UK can regulate the temperature of refrigerators & ovens to keep it at optimum temperatures which can reduce energy consumption & reduce food wastage by keeping food at the right temperatures. IoT can monitor the refrigerators temperature in real time & as soon as the temperature of a particular refrigerator goes above/below the optimum levels, IoT can assist in setting the temperature of refrigerator back to the optimum level. This will also help in reducing the greenhouse gas emissions from thousands of refrigerators used in Supermarkets across UK.

- IoT can play a big role in driving automation and controlling with building heating and cooling systems, which are heavy energy consumers. IoT sensors can detect how many people are there inside the store in real time, this can then be used to automatically turn off the system when customers have left, thus conserving energy. Again will help in reducing the greenhouse gas emissions

- Walmart installed IoT water monitoring systems to water the grass at 1,000 stores in 2009, and saw a 33% reduction in total water consumption over the next five years.

- A report by Ericsson states that, IoT could help reduce greenhouse gas emissions by up to 15%, across all industrial sectors by 2030

- Using the IoT and Cloud platform, fishermen with smartphones can now upload geo-located images of the sea mammals via a mobile app. This data can then be analysed to put together recommendations for future protection areas.

Edge Computing

Compared to cloud computing, where we access centralized services, edge computing refers to decentralized data processing where computing is done at or near the source of the data, instead of depending on the cloud which is diverts the traffic to one of the data centers to do all the processing. But that doesn’t mean the cloud will disappear, instead the cloud is coming closer to you.

 A simple example…for instance, if you have one Pressure sensor device which continuously sends pressure readings to the cloud. However if you have 100 x Pressure sensor devices across a factory/warehouse, you have a bandwidth problem. But if the sensor devices are “smart enough” (possibility for AI / ML) to only send the important readings and discard the rest, your internet bandwidth is saved.

One more example would be Alexa….with local speech recognition, Alexa only sends text to the cloud rather than voice recordings which results in much faster response, especially for commands that can be handled without leaving your home (e.g., turning on the light).

There is a view that there isn’t much growth left in the cloud space as nearly everything that could have been centralized has been already centralized, which then creates new opportunities for bringing the cloud closer to the edge.

The idea of edge computing is to process data on the spot if not fully at least partially at the end device to save resources (internet bandwidth, response times, etc.) while still benefiting from the advantages of the cloud.

At an enterprise level when we discuss edge to cloud, they typically are thinking about storing and processing data generated by employees or IoT devices, at the point where it is created and consumed (at the edge). Leveraging edge to cloud architectures can result in cost savings, improved productivity, without compromising on application responsiveness. This requires the use of resources that are not permanently connected to the network, such as sensors, tablets, controllers, etc.

IoT connected devices can best use the edge computing architecture. With remote sensors installed on machines or devices, they generate considerable amounts of data. If that data is sent across a long network link to be analyzed, tracked & processed, that takes much more time than if the data is processed at the edge, close to the source of the data.

With edge computing, security also improves as encrypted files are processed closer to the network core. Though edge computing supports real time requirements on the IoT, however for processing or storing large amount of data it needs the power of cloud computing.

The Amazon Web Service’s Snowball Edge device is a device with on-board storage and compute power for select AWS capabilities. Snowball Edge can carry out local processing and edge computing workloads and also transfer data between your local environment (devices, sensors, tablets, etc) and the AWS Cloud. AWS Snowball Edge brings the power of the AWS Cloud to your on premises location, which can cut down on the bandwidth because we can either do all of the processing on the device or pre-process it before sending it on to the cloud.

Edge to cloud file services enable companies to manage their data centrally in the cloud, while at the same time making that data instantly available to users at the edge. By having edge to cloud file services, enterprises can improve application responsiveness and availability. This enables agile collaboration, while also providing data security, governance and compliance. Edge to cloud file services will be a key enabler for business success by offering low latency, reliable access to data & power of cloud-scale economics.

Low Code Application Platforms - LCAP

The pandemic has forced businesses to rapidly transform their operating models. Businesses are adopting several Digital Transformation initiatives to address the challenges of the new normal. There is demand for businesses to be more flexible & agile and as a result businesses are looking to adopt Low Code Application Platforms LCAP. There is increasing demand for low-code platforms that fast-track app development and this demand is driven by the need for close collaboration between IT and business teams.

What is LCAP?

Low-Code application platform offers rapid application development and deployment using low or no techniques. Low-Code makes exploring & integrating latest emerging technologies like AI, ML and IoT accessible for business users and developers with drag & drop ease, enabling them to create functional prototypes. Low-Code platforms also enables business users to build Apps with little to no coding experience. With Low-code platforms, developers can work using existing templates and drag prebuilt elements, forms, and objects together to get a simple working app.

The components of Low-Code platform include: access to AI, ML, IoT without having the need for domain expertise, automated testing, fast integration, reusability, DevOps, one click deployment to the Cloud that automatically manages application reliability & scalability. For example, Low-code application platforms provide flexible integration options through which organisations can connect their disparate systems which may include legacy, on-premise systems or cloud based systems and have a unified view of their data.

Fintechs and Conventional Bankers

Fintechs companies have brought a wave of insecurity among conventional bankers. Fintech companies introduced dynamic payment systems that empowered their users to complete their financial transactions without the involvement of a middleman and limiting transaction charges applied on them by traditional banks. Fintechs have increased the speed of transactions. Technology has helped fintech organizations in improving transparency of transactions in their payment systems which led to improved user interaction and experience.

Conventional Banks have set financial services such as ATMs and online banking, their innovation skills are reliable. Building their services like the fintech companies can significantly help traditional banks in avoiding risks posed by fintechs. Developing & implementing similar services could be challenging but could yield ample benefits in the long run for traditional banks. 

Threat of Fintech to banks is real and banks have to start searching for viable options for working in the market alongside fintechs. Banks can think of partnering with fintech companies, acquire or develop a fintech portal that can create a digital value of banks.

Dark Data and ML

Data can be categorised into 3 categories:

• Critical Business Data: this is the data that is required for the day to day operations of the business, it allows the business to grow
• Trivial Data: this data is never used and has no importance to the business
• Dark Data: this is the data that is hiding within your internal systems, folders, sources and networks and can hold a large amount of information that can be useful for business and can be moved to Critical Data Set category.

According to a recent IBM study, over 80% of all data is dark and unstructured. IBM estimates that this will rise to 93% by 2020. Dark Data examples:

• Spreadsheets • Analytics Reports and Survey Data • Multiple old versions of documents • Email attachments that are downloaded and then ignored • Inactive databases with unused customer • Inactive databases with unused customer information • Project notes and learning's

Basically Dark Data is the data that is left behind from various processes, scattered across every level of business. Some people may consider it as unnecessary and ignore it, whereas it can be highly valuable for making business decisions.

To handle Dark Data spread across different types of spreadsheets, emails, zip files, documents and images stored on various servers, the power of Machine Learning algorithms can be used. AI, Machine Learning and Analytics can systematically identify the rarely used data and indicates that data is obsolete.

Aggregation of data may be required for queries which then would need integration to access the data from different sources. Machine Learning can make the process efficient by automatic mapping between the sources and data repository.

 

Post Corona - Path back to Growth

There is no single number or data that could tell what would be the impact of Covid-19 could be on the world economy. Reaction of firms, company's, and businesses are unknown.

 

We are seeing melt down in the global financial market and may indicate that the world economy is on the path of recession. Though the recession risk is there, but let us not take it as a forgone conclusion.

 

Path back to growth will depend on many factors:

 

World economy will be different after the pandemic like Covid-19 in a number of ways - crisis can spur the adoption of New Technologies and Business models.

• As Schools have closed in many parts of the world could e-learning mode of delivery of education see a smart break through. While online education is helping to cope with the current emergency, it is also preparing the world for the next future. It is a great idea to build your trusted platform for learning.
• Digital efforts to track coronavirus spread via smartphone trackers demonstrates a powerful new public health tool.
• There is need for a robust delivery management solution - finding ways to reach customers doorstep.
• Take grocery store online to ensure timely delivery of grocery orders at customers’ doorstep
• App based payments to avoid handling cash and reduce their potential exposure to COVID-19
• At this moment, pharmaceutical company's have an essential role to play. People quarantining themselves find it difficult to reach medical stores. Not only those affected by the virus, but others who are dealing with minor or major health issues, are failing to get medicines on time.  With right technology, online medicine ensure timely delivery of medicine's at peoples doorstep
• On-Demand Doctors , Nurses , Caretakers - build apps which simplifies the process of booking for medical professionals visiting home
• Fitness & Wellness App - people are increasingly turning to digital workout programs to maintain their exercise routines from home. From online training apps to yoga and meditation apps, the sector promises tremendous growth

Keep calm retain your confidence, communicate with people and teams, make them comfortable, actively collaborate with your partners, reach out to the community.

 

Work from home is the new normal -  worldwide there are millions of remote workers. As we settle down to new routines it is also making us question a constant need to travel during normal times, it is also making us understand new ways of keeping virtual teams engaged.

 

The transformation of work in organisations is being shaped by technology and talent. Faster computing power, connected devices, Cloud, Blockchain, RPAs are redefining the man machine interface.

 

Routine and mono skilled roles will be replaced with specialists. Skills like creativity, empathy, collaboration will gain more prominence, as will be there requirement for data scientists, design and systems thinking.

 

Post Covid-19 world would be led by key enablers like Technology, high level of social skills to connect virtual teams, it will require flexible work policies that allows for people to choose work times that work for them and their role, and leadership that solicits participation and not hierarchy and its purpose led and not power driven.

 

Acquiring New Skills, Continuous Learning, Expertise, Domain Skills, Leadership Skills - never ever lose sight of them.

Why do Big Companies Fail with Innovation?

Let’s first understand two things:

1.  What is Innovation - It is about doing the same things in a bit better way or doing new things

2.  Then what is Disruptive Innovation? - It is about making things that make the old things obsolete

Big Companies Fail with Innovation because:

Doing What We Have Always Done

When companies find successful business model, then it becomes their goal to exploit that advantage to its fullest extent. Big companies are then structurally organised to manage their currently successful business model. All the company structures, operations, process, tools and culture are geared towards doing what they have always done.

However the mistake companies make is in organising themselves to focus exclusively on exploitation. Every business model has a life-cycle. The decline of any business model is inevitable. When the business model eventually declines, the company will decline as well.

Companies that are structurally organized around a successful business model also find it more difficult to implement new innovations. A breakthrough innovation that fits a company's current structures is more likely to succeed. In contrast, an incremental innovation that does not fit into a company's current structures is likely to fail.

Companies are misguided when they continuously listen to their current customers.

Most big companies are not comfortable with disruptive innovations as these innovations do not satisfy their current customers. They only focus on the demands of the current customers. If the customer wants better products, they’ll keep evolving the same line of products.

Moreover, these kinds of in-house disruptive innovation can burn huge resources away from the sustaining innovation — which is required to compete against the current competitions.

Although the companies were continuously innovating to improve their product. But most of them went bankrupt continuously listening to their current customers and giving them what they demanded.

Disruptive innovations currently in progress that are already transforming the world are Machine Learning, Artificial Intelligence, Robotics, Automatic Vehicles, IoT Internet of things, Advanced Virtual Reality …

Disruptive innovation is a process where a smaller company with fewer resources are able to successfully challenge the big businesses. Big companies fail when they do not react to this disruption.

Importance of DevSecOps

DevSecOps is about introducing security earlier in the life cycle of application development, thus minimising vulnerabilities. DevSecOps aims to embed security in every part of the software development lifecycle process. It is about embedding security controls and processes early in the DevOps workflow.

With the move to Agile and DevOps methodologies and continuous delivery the ability to deploy applications in the Cloud has improved both scale and speed. Due to continuous change in technology and consumer demand, the application security was mostly an afterthought, and at times considered to be a roadblock to staying ahead in the race.

Automation from the start reduces the chance of errors. It’s about shifting security left in the SDLC lifecycle. Shifting security left is about building things that's innovative and also secure.

Integrating security into DevOps to deliver DevSecOps requires new mind-sets, processes, and tools. When developers are writing code, they need to have tools that checks for vulnerabilities during the local build process. Embed the checks for vulnerabilities within the continuous integration/continuous delivery CICD process using Jenkins so as to ensure that at each build process, there is a security element checking that the code is secure.

While a developer may do their best with regards to implementing basic security checks, nobody can know in this vast open-source world, how many software packages contain a security vulnerability and in which of its versions. An integrated DevSecOps solution or workflow which supports automation can help developers spot if they are unintentionally using any open-source libraries with known vulnerabilities, before they even begin coding the rest of the modules of a software project, only to realise they need to start over again.

Since the open-source community has always welcomed contributions from anyone, an unsuspecting developer already using one of the compromised components would have no means of knowing this, unless an automated tool was in place to be able to constantly scan their project and point out any malicious open-source components.

Make Developers security-aware

Developers are busy and tasked with implementing a certain functionality out of code for your users. Security may not always be their number one priority due to limitations imposed by meeting deadlines and even developer’s own lack of security expertise. However, with DevSecOps software solutions, the constant ‘reminders’ about excluding certain components from software builds, along with credible reasoning which warrants so, makes your developers a little more interested in and aware of security every time they see such an alert.

There is never a way of knowing whether your application or project is totally secure from all directions. But following best practices and automation - DevSecOps can drastically reduce risk arising from using software components with known vulnerabilities, right from the beginning.

Robotic Process Automation (RPA)

RPA = Automation

RPA would be probably the most efficient & effective way a company can move quickly in the Digital Transformation journey.

Automation by default is key for success of Digital Transformation journey of a company.

Idea behind Automation is about actively moving towards a world in which people will be freed from repetitive and low complexity tasks, instead having more time to dedicate to work that brings value. Basically the Robot can be focused on important but very tedious and repetitive tasks and making no errors. Which would obviously result in human getting almost entire time for value added activities which could be customer meetings, understanding business requirements in an ever changing world, face to face interactions with people and so on.

RPA is non-invasive and easy to scale. RPA offers a light weight integration into processes and IT assets. Using RPA solution, companies can easily adapt to changing business requirements by scaling the solution up or down, depending on the requirements.

What should be the approach for RPA?

-        Look to automate the routine repetitive tasks
-        Look at the every task and ask questions like : would automating the task save time, result in accuracy, reduce errors, improve productivity

The 2 types of Robots in a RPA world are: Attended Robots and Unattended Robots

Attended Robots: for carrying out automated tasks which require human interventions at various points – for example the Robot can start manually or automatically. Idea is that, Robot will do automated tasks and human will do the non-automated tasks.

Unattended Robots: for carrying out repetitive / rule based tasks which do not require any manual intervention

Automation by default approach is the key success factor : first step in this direction would be to identify what can be automated and what cannot be. 

Which then brings us the challenge of what can be automated? The four categories to look would be:

1. Cannot be automated : where most of the process tasks are performed by humans and every time the process is executed the tasks / activities can be different
2. Scope for automating some tasks : where some of the process tasks are performed by humans and every time the process is executed the some of the tasks / activities are the repetitive
3. Already have some level of automation : where some of the repetitive tasks / activities have already been automated
4. Fully Automated : a fully automated world or where every tasks can be automated

RPA implementation obviously has to be done in stages:

1.  Planning – define the processes, prioritisation, and detailed plan the implementation
2. Solution Design – document the solution design, document each process, review the solution, create test scenarios, prepare the infra and environments
3. Build – code for automating the processes, unit test the code
4.  Test - integration testing of the workflows, user acceptance testing of the workflows, and finally get the process signed off
5. Cutover to production – live transition, the process is deployed into production, monitor the process, and measure the performance
6. Hyper care, ongoing support, and continuous improvement – provide production support, track the business benefits delivered

RPA can be implemented across any domain, example Supply Chain, Logistics, Manufacturing, HR, Finance and Accounting

Best Practices of Continuous Integration

• Code Repository : maintain a main branch in the Code Repository which stores all of the Production ready code. From this main branch we can deploy code to Production at any time.

• Build Automation : create build environment in such way that even with one command build can be triggered.

• Self Testing Build : Every build should be self tested, which means that with every build there is a set of tests that runs to ensure high quality

• Daily Commit to Baseline : developers should commit all of their changes to baseline on a daily basis. This is to ensure that large amount of code is not waiting for integration with the main repository for a long time. 

• Do Not Check In Broken Build : developers tend to check in broken build. And when the build breaks, the same developers who broke the code has to spend time to build it and get it working quickly.

• Always Commit Tests Locally : Running a commit tests locally is a sanity check before actually committing to the action. This will ensure that what we believe to work actually works. 
  
• Production Like Environment Testing :  maintain a pre - prod environment which is very close to or nearly like a production environment. Perform testing on this pre-prod environment to check for any integration issues.

• Publish Build Results : publish the build results on a common platform or site so that everyone can access and see these results and take corrective actions immediately.
• Deployment Automation :  automate the deployment process to the extent that in a build process we can add the steps of deploying the code to a test environment. On this test environment the latest delivery can be tested by all stakeholders

Artificial Intelligence impact on Retail Industry

Do we know how technology is making an impact on the retail industry as of now? The retailers have massive amounts of data about their customers, their shopping patterns and experiences and much more. These vast collections of big data are often too difficult for some of the average companies to drill down into and analyse properly. Therefore, large amounts of this valuable information are left to rot, resulting in very low conversion rates overall.

Nevertheless, AI technology may change that for these retailers by using this data to create web shops that take customer information and turn it into targeted shopping experiences, online chatbots that will easily answer questions and assist customers, and in-store intelligence to make the experience even more interactive.

Some of the Retailers are in the process of transforming their contact center to include ‘chatbot agent’ with an AI program determining which calls to route to chatbot and which ones to the actual agent.

Web shops that customise their marketing and items based on customers' previous purchases, search patterns & habits, number of clicks, age, gender, and other variables by using self-learning algorithms could take what once was a lost customer and make their experience far more intimate, targeted and effective. This allows retailers to make more sales online and also provide more items on their online stores. The algorithm could easily bring up items it feels the user would want, as a start.

Some of the Retailers are looking to use Machine Learning algorithm to determine how promotional campaigns will work to predict the success of the next customer touch point.

In addition to the automated personal assistant chatbot, a customer could receive targeted marketing campaigns when they shop. Current chatbots rely on particular keywords and phrases to generate responses. However, through advancements in machine learning, a whole new form of chatbots, which will adapt to your questions to answer more complex things and think for themselves, will be deployed on retail sites in the future.

The way forward is clear, we must find the means for Humans and machines to work together.

Some of the retailers has also begun to replace staff with automated machines to make consistent products and reduce the time to serve customers.

The retail sector is in an immense state of change and business transformation. The entire retail sector is trying hard to cope with the fast and ever-changing customer shopping habits, and the shift from high street to Web. We have seen some of the retail giants closing stores showing moving customer demand while some of the retailers have moved their majority business to Web / online.

Across the board, the retail sector has started investing heavily and innovated with various technologies including AI, Robotics, Supply chain, and logistics automation, Order Forecasting automation, Data analytics to become more competitive, ability to become more responsive to demand and opportunity.

Due to the transactional business, Retailers in general generate high volumes of data related to sales, cost of product, customer histories, new trends, reports, geographical data, weather information etc. All of this data both current and historical can be put to use to deliver functional business can be put to use to deliver functional business insights and informed decision-making, reduced time to marked for new products and services, and improve success rates for initiatives. Automating and informing decision-making through AI can help retailers determine what to order and when, what products to merchandise at the front of the store, cross-selling and up-selling opportunities to individual customers based on previous purchases and current basket contents.

The process of planning, procurement, making, distributing, selling and gathering customer experience can take more than a year in the retail sector. It’s a long lead time that limits response to fast-moving trends.

AI can expedite the process, reducing the time from ideation to sale, reducing indecision and informing trading decisions through historical data and trend analysis.

Majority of Retailers cite improving customer experience as their main motivation in investing in AI and Automation. The automating includes services, in-store and online to highly personalise the shopping experience. This involves creating one-to-one interaction using vast customer history and trend data. It’s not just at the storefront, we can see AI being used in the large warehouse based retail and fulfillment centers. AI, robotics and automation systems are being used to increased operating capacity and enable more orders being processed per hour. Its helping retail cutting cost by allocating more warehouse floor space to be used for product storage through reducing the shelving aisles.

The adoption of AI represents an exciting leap forward in the retail sector.

AI's purpose is not necessarily to replace humans, but rather to assist them and increase technology-based jobs worldwide. This is expected to increase job opportunities by expanding the job market to make way for AI. From the creation of more jobs with specific skills never offered before to customer experiences, AI is the future of the retail industry that has sat on customer data for too long and justifies a chance to put this information to good use at last.

Part 2 of series "AWS cloud components -simplified - The Power of Lambda"

Continuing my series on AWS cloud components simplified, here I have made a pictorial representation of how Lambda works within AWS cloud platform.

Below diagram should be read in conjunction with my previous blog on The Power of Lambda.

Part 1 of series "AWS cloud components -simplified - The Power of Lambda"

AWS S3 bucket:

This component can be used to receive files coming from external sources. The file is routed via API Gateway in order to balance load, firewall unprecedented request for security and serve as a proxy server.

AWS Lambda:

This will monitor S3 bucket in order to record landing of a file. It will then provision the event of incoming file in bucket to different tasks like reception and transformation.

An example:

- S3 bucket can be structured to contain sub-folders named pending, processing, received, renamed, transformed, validated, error, exception and archive.

- Incoming Files can be filtered and sent to Pending folder on file creation event (marks file arrival). This will also instantiate AWS lambda.

- AWS lambda can then store filename and status in the table with status=’Received’. It will also invoke EC2 container (reception task) for further processing.

- Reception task is a docker instance running on EC2. It will identify the file that is ready and read the file from S3 and process it.

- Post job in ACW to invoke ProcessorLambda.

- On successful processing, the last step is to post an ACW job to invoke SubmissionLambda

- ACW will trigger based on time event (will trigger on specific time as per configuration) and invoke ExceptionTask via ExceptionLambda

AWS Cloudwatch:

ACW will perform orchestration role. It will ensure start of different tasks based on completion of previous tasks.

AWS SNS:

SNS is primarily used as a notification service. It will serve as an interface to report errors during processing of a task. It will send a notification in order to log the error using already existent loggly service.

AWS EC2 cluster:

EC2 cluster will comprise of several dockerized tasks. These are reception, transformation, submission and exception.

AWS RDS:

We can make use of AWS RDS (postgres) in order to hold DB objects. Broadly it needs to hold DB objects to preserve any processing or transnational tasks.

Jenkins and Continuous Integration

Jenkins is an open source tool which comes with built in plugin for continuous integration purpose. Jenkins primary functionality is to keep a track of version control system and to initiate build and monitor a build in case of any changes. Jenkins monitors the entire process and provides reports and alert notifications.

For using Jenkins we would require the source code repository (example Git repository, PVCS, SVN), a working build script (example Ant / Maven script), which is checked-in to the code repository.

Jenkins captures any build failures during integration stage, automatically generates build report notification for every code commit changes, sends notification to developers about build report success / failure, achieves continuous integration and test driven development. With simple steps, maven release project is automated. Jenkins comes with built in plugin for continuous integration like Maven 2 project, Amazon EC2, etc.

Jenkin is mainly integrated with two components, version control repositories like GIT, SVN and build tools like Ant, Maven.

Within Jenkins, builds can be triggered by source code management commits, can be triggered after completion of other builds, can be scheduled to run at specified time, or by manual build requests

In order for Jenkins to do a clean build, it’s advised that developer perform a successful clean install on local machine with all unit tests successful. Only then the code changes must be checked into the code repository.

In case a Jenkins build failure, open the console output for the build and debug to see if any file changes were missed. If not able to find the issue that way, then clean and update your local workspace to replicate the problem and try to resolve it.

DevOps explained

DevOps is a cultural shift that merges development and operations. Apart from having skills of web languages such as Python or Java, the ideal DevOps team should have some experience using infrastructure automation tools like Chef or Ansible. Organisations also think about the essential interpersonal skills that make DevOps successful.

Instead of releasing big bang release of features, companies are now trying to see if small features can be delivered in short and regular intervals. This enables many advantages like getting quick feedback from customers and better quality of software and higher customer satisfaction. For achieving this, there is a need for increasing the deployment frequency, reduce the failure rate of releases, reduced time gap between fixes.

Some of the popular DevOps tools:
Git - for version control system tool
Jenkins - continuous integration tool
Chef, Ansible - configuration management and deployment tools
Docker - Containerization tool
Selenium - continuous testing tool
Nagios - Continuous Monitoring tool

Developers develop the code and this source code can be managed by Version Control System tools like Git. Developers check-in this code into the Git repository and any changes made in the code is committed to this repository. Jenkins pulls this code from the Git repository using the Git plugin and build it using tools like Ant. Configuration management tools like Chef / puppet deploys & provisions testing environment and then Jenkins releases this code on the test environment on which testing is done using tools like selenium. Once the code is tested, Jenkins send it for deployment on the production server. Post deployment it is continuously monitored by tools like Nagios

Docker containers provides testing environment to test the build features.

Version control is a system that records changes to a file or set of files over time so that you can recall specific versions later. Version control systems consist of a central shared repository where team members can commit changes to a file or set of file. Git has a major advantage it has over other VCS tools like SVN is that it is a distributed version control system. Distributed VCS tools do not necessarily rely on a central server to store all the versions of a project’s files. Instead, every developer “clones” a copy of a repository and has the full history of the project on their own hard drive.Git is a Distributed Version Control system (DVCS). It can track changes to a file and allows you to revert back to any particular change. There is a central cloud repository as well where developers can commit changes and share it with other team members.


Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

Developers check out code into their local work spaces. Once done with code changes, they commit the changes to the Version Control Repository. CI server monitors the repository and checks out changes as and when they occur. CI server then pulls these changes and builds the system and also runs unit and integration tests. CI server will inform the team of the successful build. In case of build or tests failures, the CI server will alert the team for fixing.

Success factors for Continuous Integration would include maintaining a shared code repository, automated build, making build self-testing, everyone commits to the baseline, every commit to the baseline should be built, test in a production like environment, all team members can see the results of the latest build, automated deployment.

Usage of Jenkins for CI: move a job from one installation of Jenkins to another by simply copying the corresponding job directory. Make a copy of an existing job by making a clone of a job directory by a different name. Continuous Testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with in the latest build. In this way, each build is tested continuously, allowing Development teams to get fast feedback so that they can prevent those problems from progressing to the next stage of Software delivery life-cycle. Automation testing is a process of automating the manual process to test the application/system under test. Automation testing involves use of separate testing tools which lets you create test scripts which can be executed repeatedly and doesn’t require any manual intervention.

Selenium supports two types of testing 1. Regression Testing: re-testing a code around an area where a defect was fixed. 2. Functional Testing: refers to the testing of software features individually.

Infrastructure as Code IAC is a type of IT infrastructure that operations teams can use to automatically manage and provision through code, rather than using a manual process.

Puppet is a Configuration Management tool which is used to automate administration tasks.
Puppet has a Master-Slave architecture in which the Slave has to first send a Certificate signing request to Master and Master has to sign that Certificate in order to establish a secure connection between Puppet Master and Puppet Slave. Puppet Slave sends request to Puppet Master and Puppet Master then pushes configuration on Slave

Chef is an automation platform that transforms infrastructure into code. Chef is a tool for which you write scripts that are used to automate processes.Chef Server is the central store of your infrastructure’s configuration data. Chef Server stores the data necessary to configure your nodes and provides search. Chef Node is any host that is configured using Chef-client. Chef-client runs on your nodes, contacting the Chef Server for the information necessary to configure the node. Since a Node is a machine that runs the Chef-client software, nodes are sometimes referred to as “clients”. A Chef Workstation is the host you use to modify your cookbooks and other configuration data.

Containerization: containers are used to provide consistent computing environment from a developer’s laptop to a test environment, from a staging environment into production.
Container consists of an entire runtime environment, an application, plus all its dependencies, libraries and other binaries, and configuration files needed to run it, bundled into one package. Containerizing the application platform and its dependencies removes the differences in OS distributions and underlying infrastructure.

Docker image are used to create containers. Images are created with the build command, and they’ll produce a container when started with run. Images are stored in a Docker registry because they can become quite large, images are designed to be composed of layers of other images, allowing a minimal amount of data to be sent when transferring images over the network. Docker containers include the application and all of its dependencies but share the kernel with other containers, running as isolated processes in user space on the host operating system. Docker containers are not tied to any specific infrastructure: they run on any computer, on any infrastructure, and in any cloud. Docker containers can be created by either creating a Docker image and then running it or you can use Docker images that are present on the Dockerhub. Docker containers are basically runtime instances of Docker images.

Docker hub is a cloud-based registry service which allows you to link to code repositories, build your images and test them, stores manually pushed images, and links to Docker cloud so you can deploy images to your hosts. It provides a centralized resource for container image discovery, distribution and change management, user and team collaboration, and workflow automation throughout the development pipeline.

Docker Swarn is native clustering for Docker which turns a pool of Docker hosts into a single, virtual Docker host. Docker Swarm serves the standard Docker API, any tool that already communicates with a Docker daemon can use Swarm to transparently scale to multiple hosts.

DevOps - Adopting Continuous Deployment

Continuous Delivery and Deployment, there is a difference between the two. When adopting DevOps, Continuous Delivery is a core capability to adopt. Continuous Deployment to Production is an optional capability that you may or may not adopt, based on your needs and constraints.

Then what does one do ‘deliver’ when adopting Continuous Delivery or Deployment? We shall look at it from the perspective of  People, Process and Technology.

As mentioned in my previous blog, DevOps is a cultural movement. The people aspect of DevOps is where it all begins. While adopting Continuous Delivery, the most important part is the culture of continuous delivery and continuous collaboration between all the stakeholders who need to be onboard to enable and consume continuous delivery. This includes Dev and Ops but also stakeholders across the SDLC including Business, Analysts, Product owners, Architects and Design, Information Security, Quality Assurance and Management. There is importance of  creating a culture where all these stakeholders contribute to the continuous delivery, and also accept the feedback that comes from the continuous delivery at every stage. Dev teams uses the feedback to decide what to work on for the next Sprint, Quality Assurance or testing teams uses it to test and validate functionality, integrations and performance. The Ops team determines how the environment performed and where it needs enhancement or corrections. Project management team will manage their project and release plans, and so on.

The change in culture is of continuous collaboration, communication, trust and working towards common business goals.

When continuously delivering software, one is not only validating the functionality and performance of the software being delivered and the environments it is being delivered to, but also the process of deploying the software. Deployment of the code involves code deployment, file transfers, configuration changes to operating systems, databases and middleware (SOA / OSB, ODI etc). It also involves an orchestration of steps. The middleware processes may need to be restarted after configuration changes. Services may need to be stopped before file transfers and then restarted. Hence, continuous delivery allows for these processes to be tested and refined to ensure that when it comes to the final deployment to production, it is not the first time the team is executing the processes. They are tested and proven to work in production like environments.

Deployment requires set of tools that can automate the deployments and ensure continuous delivery of all changes from one environment in the SDLC to the next – to Dev, to testing, to Performance testing, to SIT integration testing, UAT user acceptance, Pre Prod and Production.

The key here is to start continuous delivery from the start of the project, from Sprint 0 all the way thru the project. In the beginning, the deployments may be simple, to much more complex orchestrated deployments later in the project. Continuous delivering changes – application, middleware, configuration, data and environment – in small pieces, using the right automation tools, reduces risk by validating the automation, the deployment processes, the configuration changes, the environments being deployed to and of course, the application being deployed.

Cloud Platform - Jenkins and Docker

Cloud platform has made the way for programmable infrastructure, which brought automation into SDLC. The ability to make provision of resources, configuring them dynamically, deploying applications and monitoring the entire process have led to the DevOps culture where developers and the operators are collaborating throughout the application delivery lifecycle.

Chef, Puppet, and Ansible are best of the tools for provisioning and configuration. Another software that is crucial for success of DevOps is Jenkins.

Application build, followed by packaging and deployment play a crucial role in release & configuration management. Developers relied on tools like Ant, Maven to compile. Jenkins comes to the rescue of Dev teams. By automating the build process, Jenkins can well coordinate complex workflows required by teams. By integrating with the source code control system, Jenkins can invoke multiple build processes targeting different environments. The output of this process would result in a set of executables exes, or jar files depending on the platform. Jenkins can invoke simple shell scripts to perform pre-build and post-build tasks. This option enables software like node.js, python to take benefit of build automation.

Docker is a preferred environment for software development and testing. DevOps teams find it to more be efficient to configure development and test environments based on Docker. So, instead of deploying the final set of artifacts such as exe / jar files to the target environment, ops teams can now package the entire application as a Docker Image. This image shares the same build version tag before being published to a central registry. It can then be picked up by various environments viz development, testing, and production for final deployment.

DevOps teams are greatly helped by using combination of Jenkins and Docker. The tight integration with source code control mechanisms such as Git, Jenkins can initiate a build process each time a developer commits his code. This process results in a new Docker image which is instantly available across environments.

You cannot ignore Docker, it’s time to include it in the DevOps strategy.

Dev Vs Ops

The Dev

Developers build software and deliver changes quickly, while the IT operations give priority on stability and reliability of the system. This mismatch of goals between the teams in an organisation can lead to conflict, and eventually more cost to the Business.

Developers deliver software after performing testing in development or PTL (path to live) environments. However these systems may not be same as complexly integrated production systems. While doing development, the developers are not really concerned about the production infrastructure as well as the deployment impact due to code changes. 

The Ops

The IT operations (service support) focus majorly on SLAs around uptime and stability. IT ops expect lesser turn around time for code deployment and testing as high volumes of development builds / code changes come their way.

In today’s work these old divisions within organisations are breaking down, with the IT operations and Developer roles merging and following efficient principles like:

• Automation
• Continuous deployment
• Monitoring
• Security

Benefits of bringing Dev and Ops together

We are here to ENABLE Business. Developers work in collaboration with Operations to impact assess the code change. Developers work more closely with production equivalent environments with similar integration points and live like data. The Operations have better clarity of infrastructure. As a result there is more focus on deployment automation and test automation. Also Monitoring provides more visibility of Dev to Test to Production pipeline for each deployment and better feedback mechanism.

DevOps is an approach to bridge the gap between Agile Software Development and Operations. DevOps doesn’t mean Fire-fighting, DevOps is to help.

Agile and DevOps

In the recent years, the agile software development methodology has started to move downstream towards DevOps. Agile software development primarily focuses on the collaboration between the business requirements and its developers, whereas DevOps focuses on the collaboration between developers and IT operations (DBAs, Network engineers, administrators, infrastructure architects, service support personnel and security). Agile software development provides business the much required agility, DevOps provides IT agility, enabling the deployment of applications that are more reliable, predictable, and cost efficient. With respect to application development, DevOps focuses on development of code, code coverage, unit or component testing, release packaging, and deployment. With respect to infrastructure, on the other hand, DevOps focuses on provisioning, configuration and deployment. Underlying principles include version management, deployment, re-startability, roll back and roll forward.

DevOps is what Agile is to software development

What do we do in DevOps

DevOps can be categorised as C.A.M.S (Culture, Automation, Measurement, Sharing)

Culture:
A successful DevOps is all about people relationships and communication.

Automation:

Machines are really good at doing the same task over and over. Automation guarantees Consistent and Known state. Automation provides Fast and Efficient delivery of products to the business.

What can be Automated?

Measurement

DevOps is an evolving process, hence to evolve there is a need of measurement at every stage to capture the learning's and enable continuous improvement. 

Mesurement assists in

• Capacity planning
• Trend analysis
• Fault finding
• Plotted on a graph over time

Sharing

Sharing of knowledge base creates redundancy that helps reducing dependencies and increase synergies within the team. Sharing can be in the form of:

• Share Ideas
• Share Metrics
• Ops: Give devs shell access
• Devs: See what technology can be leveraged

DevOps Lifecycle

Before:

• Talk about functional requirements

• Talk about non-functional requirements

           – Security

           – Backups

           – Availability

           – Upgradeability

           – Configuration management

           – Monitoring

           – Logging

           – Metrics

During:

           – Communication

           – Source control

           – Automate builds

           – Automate tests

           – Automate deployments (dev, test and production)

           – Collate metrics

After:

• Release

          – Retrospective Meetings

          – Continue to Run Tests

          – Monitor Applications and Systems

• Issues

          – Post mortem meetings

          – What went wrong, what could be done better