Dark Data and ML

Data can be categorised into 3 categories:

• Critical Business Data: this is the data that is required for the day to day operations of the business, it allows the business to grow
• Trivial Data: this data is never used and has no importance to the business
• Dark Data: this is the data that is hiding within your internal systems, folders, sources and networks and can hold a large amount of information that can be useful for business and can be moved to Critical Data Set category.

According to a recent IBM study, over 80% of all data is dark and unstructured. IBM estimates that this will rise to 93% by 2020. Dark Data examples:

• Spreadsheets • Analytics Reports and Survey Data • Multiple old versions of documents • Email attachments that are downloaded and then ignored • Inactive databases with unused customer • Inactive databases with unused customer information • Project notes and learning's

Basically Dark Data is the data that is left behind from various processes, scattered across every level of business. Some people may consider it as unnecessary and ignore it, whereas it can be highly valuable for making business decisions.

To handle Dark Data spread across different types of spreadsheets, emails, zip files, documents and images stored on various servers, the power of Machine Learning algorithms can be used. AI, Machine Learning and Analytics can systematically identify the rarely used data and indicates that data is obsolete.

Aggregation of data may be required for queries which then would need integration to access the data from different sources. Machine Learning can make the process efficient by automatic mapping between the sources and data repository.

 

Post Corona - Path back to Growth

There is no single number or data that could tell what would be the impact of Covid-19 could be on the world economy. Reaction of firms, company's, and businesses are unknown.

 

We are seeing melt down in the global financial market and may indicate that the world economy is on the path of recession. Though the recession risk is there, but let us not take it as a forgone conclusion.

 

Path back to growth will depend on many factors:

 

World economy will be different after the pandemic like Covid-19 in a number of ways - crisis can spur the adoption of New Technologies and Business models.

• As Schools have closed in many parts of the world could e-learning mode of delivery of education see a smart break through. While online education is helping to cope with the current emergency, it is also preparing the world for the next future. It is a great idea to build your trusted platform for learning.
• Digital efforts to track coronavirus spread via smartphone trackers demonstrates a powerful new public health tool.
• There is need for a robust delivery management solution - finding ways to reach customers doorstep.
• Take grocery store online to ensure timely delivery of grocery orders at customers’ doorstep
• App based payments to avoid handling cash and reduce their potential exposure to COVID-19
• At this moment, pharmaceutical company's have an essential role to play. People quarantining themselves find it difficult to reach medical stores. Not only those affected by the virus, but others who are dealing with minor or major health issues, are failing to get medicines on time.  With right technology, online medicine ensure timely delivery of medicine's at peoples doorstep
• On-Demand Doctors , Nurses , Caretakers - build apps which simplifies the process of booking for medical professionals visiting home
• Fitness & Wellness App - people are increasingly turning to digital workout programs to maintain their exercise routines from home. From online training apps to yoga and meditation apps, the sector promises tremendous growth

Keep calm retain your confidence, communicate with people and teams, make them comfortable, actively collaborate with your partners, reach out to the community.

 

Work from home is the new normal -  worldwide there are millions of remote workers. As we settle down to new routines it is also making us question a constant need to travel during normal times, it is also making us understand new ways of keeping virtual teams engaged.

 

The transformation of work in organisations is being shaped by technology and talent. Faster computing power, connected devices, Cloud, Blockchain, RPAs are redefining the man machine interface.

 

Routine and mono skilled roles will be replaced with specialists. Skills like creativity, empathy, collaboration will gain more prominence, as will be there requirement for data scientists, design and systems thinking.

 

Post Covid-19 world would be led by key enablers like Technology, high level of social skills to connect virtual teams, it will require flexible work policies that allows for people to choose work times that work for them and their role, and leadership that solicits participation and not hierarchy and its purpose led and not power driven.

 

Acquiring New Skills, Continuous Learning, Expertise, Domain Skills, Leadership Skills - never ever lose sight of them.

Why do Big Companies Fail with Innovation?

Let’s first understand two things:

1.  What is Innovation - It is about doing the same things in a bit better way or doing new things

2.  Then what is Disruptive Innovation? - It is about making things that make the old things obsolete

Big Companies Fail with Innovation because:

Doing What We Have Always Done

When companies find successful business model, then it becomes their goal to exploit that advantage to its fullest extent. Big companies are then structurally organised to manage their currently successful business model. All the company structures, operations, process, tools and culture are geared towards doing what they have always done.

However the mistake companies make is in organising themselves to focus exclusively on exploitation. Every business model has a life-cycle. The decline of any business model is inevitable. When the business model eventually declines, the company will decline as well.

Companies that are structurally organized around a successful business model also find it more difficult to implement new innovations. A breakthrough innovation that fits a company's current structures is more likely to succeed. In contrast, an incremental innovation that does not fit into a company's current structures is likely to fail.

Companies are misguided when they continuously listen to their current customers.

Most big companies are not comfortable with disruptive innovations as these innovations do not satisfy their current customers. They only focus on the demands of the current customers. If the customer wants better products, they’ll keep evolving the same line of products.

Moreover, these kinds of in-house disruptive innovation can burn huge resources away from the sustaining innovation — which is required to compete against the current competitions.

Although the companies were continuously innovating to improve their product. But most of them went bankrupt continuously listening to their current customers and giving them what they demanded.

Disruptive innovations currently in progress that are already transforming the world are Machine Learning, Artificial Intelligence, Robotics, Automatic Vehicles, IoT Internet of things, Advanced Virtual Reality …

Disruptive innovation is a process where a smaller company with fewer resources are able to successfully challenge the big businesses. Big companies fail when they do not react to this disruption.

Importance of DevSecOps

DevSecOps is about introducing security earlier in the life cycle of application development, thus minimising vulnerabilities. DevSecOps aims to embed security in every part of the software development lifecycle process. It is about embedding security controls and processes early in the DevOps workflow.

With the move to Agile and DevOps methodologies and continuous delivery the ability to deploy applications in the Cloud has improved both scale and speed. Due to continuous change in technology and consumer demand, the application security was mostly an afterthought, and at times considered to be a roadblock to staying ahead in the race.

Automation from the start reduces the chance of errors. It’s about shifting security left in the SDLC lifecycle. Shifting security left is about building things that's innovative and also secure.

Integrating security into DevOps to deliver DevSecOps requires new mind-sets, processes, and tools. When developers are writing code, they need to have tools that checks for vulnerabilities during the local build process. Embed the checks for vulnerabilities within the continuous integration/continuous delivery CICD process using Jenkins so as to ensure that at each build process, there is a security element checking that the code is secure.

While a developer may do their best with regards to implementing basic security checks, nobody can know in this vast open-source world, how many software packages contain a security vulnerability and in which of its versions. An integrated DevSecOps solution or workflow which supports automation can help developers spot if they are unintentionally using any open-source libraries with known vulnerabilities, before they even begin coding the rest of the modules of a software project, only to realise they need to start over again.

Since the open-source community has always welcomed contributions from anyone, an unsuspecting developer already using one of the compromised components would have no means of knowing this, unless an automated tool was in place to be able to constantly scan their project and point out any malicious open-source components.

Make Developers security-aware

Developers are busy and tasked with implementing a certain functionality out of code for your users. Security may not always be their number one priority due to limitations imposed by meeting deadlines and even developer’s own lack of security expertise. However, with DevSecOps software solutions, the constant ‘reminders’ about excluding certain components from software builds, along with credible reasoning which warrants so, makes your developers a little more interested in and aware of security every time they see such an alert.

There is never a way of knowing whether your application or project is totally secure from all directions. But following best practices and automation - DevSecOps can drastically reduce risk arising from using software components with known vulnerabilities, right from the beginning.

Robotic Process Automation (RPA)

RPA = Automation

RPA would be probably the most efficient & effective way a company can move quickly in the Digital Transformation journey.

Automation by default is key for success of Digital Transformation journey of a company.

Idea behind Automation is about actively moving towards a world in which people will be freed from repetitive and low complexity tasks, instead having more time to dedicate to work that brings value. Basically the Robot can be focused on important but very tedious and repetitive tasks and making no errors. Which would obviously result in human getting almost entire time for value added activities which could be customer meetings, understanding business requirements in an ever changing world, face to face interactions with people and so on.

RPA is non-invasive and easy to scale. RPA offers a light weight integration into processes and IT assets. Using RPA solution, companies can easily adapt to changing business requirements by scaling the solution up or down, depending on the requirements.

What should be the approach for RPA?

-        Look to automate the routine repetitive tasks
-        Look at the every task and ask questions like : would automating the task save time, result in accuracy, reduce errors, improve productivity

The 2 types of Robots in a RPA world are: Attended Robots and Unattended Robots

Attended Robots: for carrying out automated tasks which require human interventions at various points – for example the Robot can start manually or automatically. Idea is that, Robot will do automated tasks and human will do the non-automated tasks.

Unattended Robots: for carrying out repetitive / rule based tasks which do not require any manual intervention

Automation by default approach is the key success factor : first step in this direction would be to identify what can be automated and what cannot be. 

Which then brings us the challenge of what can be automated? The four categories to look would be:

1. Cannot be automated : where most of the process tasks are performed by humans and every time the process is executed the tasks / activities can be different
2. Scope for automating some tasks : where some of the process tasks are performed by humans and every time the process is executed the some of the tasks / activities are the repetitive
3. Already have some level of automation : where some of the repetitive tasks / activities have already been automated
4. Fully Automated : a fully automated world or where every tasks can be automated

RPA implementation obviously has to be done in stages:

1.  Planning – define the processes, prioritisation, and detailed plan the implementation
2. Solution Design – document the solution design, document each process, review the solution, create test scenarios, prepare the infra and environments
3. Build – code for automating the processes, unit test the code
4.  Test - integration testing of the workflows, user acceptance testing of the workflows, and finally get the process signed off
5. Cutover to production – live transition, the process is deployed into production, monitor the process, and measure the performance
6. Hyper care, ongoing support, and continuous improvement – provide production support, track the business benefits delivered

RPA can be implemented across any domain, example Supply Chain, Logistics, Manufacturing, HR, Finance and Accounting

Best Practices of Continuous Integration

• Code Repository : maintain a main branch in the Code Repository which stores all of the Production ready code. From this main branch we can deploy code to Production at any time.

• Build Automation : create build environment in such way that even with one command build can be triggered.

• Self Testing Build : Every build should be self tested, which means that with every build there is a set of tests that runs to ensure high quality

• Daily Commit to Baseline : developers should commit all of their changes to baseline on a daily basis. This is to ensure that large amount of code is not waiting for integration with the main repository for a long time. 

• Do Not Check In Broken Build : developers tend to check in broken build. And when the build breaks, the same developers who broke the code has to spend time to build it and get it working quickly.

• Always Commit Tests Locally : Running a commit tests locally is a sanity check before actually committing to the action. This will ensure that what we believe to work actually works. 
  
• Production Like Environment Testing :  maintain a pre - prod environment which is very close to or nearly like a production environment. Perform testing on this pre-prod environment to check for any integration issues.

• Publish Build Results : publish the build results on a common platform or site so that everyone can access and see these results and take corrective actions immediately.
• Deployment Automation :  automate the deployment process to the extent that in a build process we can add the steps of deploying the code to a test environment. On this test environment the latest delivery can be tested by all stakeholders

Artificial Intelligence impact on Retail Industry

Do we know how technology is making an impact on the retail industry as of now? The retailers have massive amounts of data about their customers, their shopping patterns and experiences and much more. These vast collections of big data are often too difficult for some of the average companies to drill down into and analyse properly. Therefore, large amounts of this valuable information are left to rot, resulting in very low conversion rates overall.

Nevertheless, AI technology may change that for these retailers by using this data to create web shops that take customer information and turn it into targeted shopping experiences, online chatbots that will easily answer questions and assist customers, and in-store intelligence to make the experience even more interactive.

Some of the Retailers are in the process of transforming their contact center to include ‘chatbot agent’ with an AI program determining which calls to route to chatbot and which ones to the actual agent.

Web shops that customise their marketing and items based on customers' previous purchases, search patterns & habits, number of clicks, age, gender, and other variables by using self-learning algorithms could take what once was a lost customer and make their experience far more intimate, targeted and effective. This allows retailers to make more sales online and also provide more items on their online stores. The algorithm could easily bring up items it feels the user would want, as a start.

Some of the Retailers are looking to use Machine Learning algorithm to determine how promotional campaigns will work to predict the success of the next customer touch point.

In addition to the automated personal assistant chatbot, a customer could receive targeted marketing campaigns when they shop. Current chatbots rely on particular keywords and phrases to generate responses. However, through advancements in machine learning, a whole new form of chatbots, which will adapt to your questions to answer more complex things and think for themselves, will be deployed on retail sites in the future.

The way forward is clear, we must find the means for Humans and machines to work together.

Some of the retailers has also begun to replace staff with automated machines to make consistent products and reduce the time to serve customers.

The retail sector is in an immense state of change and business transformation. The entire retail sector is trying hard to cope with the fast and ever-changing customer shopping habits, and the shift from high street to Web. We have seen some of the retail giants closing stores showing moving customer demand while some of the retailers have moved their majority business to Web / online.

Across the board, the retail sector has started investing heavily and innovated with various technologies including AI, Robotics, Supply chain, and logistics automation, Order Forecasting automation, Data analytics to become more competitive, ability to become more responsive to demand and opportunity.

Due to the transactional business, Retailers in general generate high volumes of data related to sales, cost of product, customer histories, new trends, reports, geographical data, weather information etc. All of this data both current and historical can be put to use to deliver functional business can be put to use to deliver functional business insights and informed decision-making, reduced time to marked for new products and services, and improve success rates for initiatives. Automating and informing decision-making through AI can help retailers determine what to order and when, what products to merchandise at the front of the store, cross-selling and up-selling opportunities to individual customers based on previous purchases and current basket contents.

The process of planning, procurement, making, distributing, selling and gathering customer experience can take more than a year in the retail sector. It’s a long lead time that limits response to fast-moving trends.

AI can expedite the process, reducing the time from ideation to sale, reducing indecision and informing trading decisions through historical data and trend analysis.

Majority of Retailers cite improving customer experience as their main motivation in investing in AI and Automation. The automating includes services, in-store and online to highly personalise the shopping experience. This involves creating one-to-one interaction using vast customer history and trend data. It’s not just at the storefront, we can see AI being used in the large warehouse based retail and fulfillment centers. AI, robotics and automation systems are being used to increased operating capacity and enable more orders being processed per hour. Its helping retail cutting cost by allocating more warehouse floor space to be used for product storage through reducing the shelving aisles.

The adoption of AI represents an exciting leap forward in the retail sector.

AI's purpose is not necessarily to replace humans, but rather to assist them and increase technology-based jobs worldwide. This is expected to increase job opportunities by expanding the job market to make way for AI. From the creation of more jobs with specific skills never offered before to customer experiences, AI is the future of the retail industry that has sat on customer data for too long and justifies a chance to put this information to good use at last.