Data Migration and Cloud Silos

The Need for Migration

To keep up with current trends as well as industry best practices, a business will find the need to upgrade its enterprise applications to achieve higher ROI by utilizing the latest technologies and features. In these situations, moving data from one system to another may become necessary and will become a major part of the upgrading or integration project.

Surprisingly, data migration is not given the importance it deserves in these projects, and therefore is not planned well enough and given the right resources to guarantee success. Perhaps that’s the reason why the failure rate is so high. According to one survey by Boor Research, about two third of data migration projects end up overrunning the time and the budget, and according to another research by Gartner, 83% of data migrations fail or exceed their allotted budgets. 

Common Migration Challenges

Many companies don’t realize the true extent of the deficiencies related to their data management and quality until they engage in a migration project. The absence of a practical data management and governance policy, degradation of data quality and chaos in the organizational structure, access rights, user roles and other governance issues create serious obstacles in a data migration effort.

Low quality of data is the most immediate challenge encountered when starting a migration project. Duplicate data, poor data entry, missing data, misplaced data, data that is not normalized to conform to the system, all are data quality problems that must be corrected through a careful data cleaning process.

In addition to the above, final testing and validation could be a huge effort in large and complex data models, forcing the organization to allocate precious resources and causing unplanned time consumption. Integrating testing and validation of data into every phase of the project takes a huge burden off the final validation phase and could save valuable time and resources.

What Could Go Wrong?

There are many factors at play during a data migration project, and just as many opportunities for error. Two of the biggest issues arise from a lack of proper planning as well as the absence of a qualified data migration team. Existence or absence of a team of experienced data engineers and Salesforce experts handling the migration project could mean the difference between project success or project failure. It is not uncommon to run into serious issues and errors during migration, but with a qualified staff, using best practices and the knowledge they have gained from previous projects, they will find a timely and cost-efficient solution; avoiding delays and any fear of budget overrun.

Additionally, there are problems related to the integrity of the data itself. There always is a risk of data loss or corruption with every operation you perform on your data, and this problem increases exponentially when the data comes from different sources and the sizes are large. The mismatch between the format of the migrating data and the fields in the new system could render the data unusable.  Lowering these risks amounts to the careful identification of any errors in the data and difference in the format between corresponding fields during the mapping operation; and then fixing all errors and inconsistencies before any attempt at migration or transfer.

The final hurdle is related to data security and privacy. Depending on the nature of the data that is being transferred, whether it is for compliance purposes regarding personal or healthcare data, or for protecting business data, implementing measures to prevent data privacy and protection violations could become a crucial part of a data migration project. In such cases, allocating necessary resources to methods such as data masking could save the organization time and money.

Planning a Migration Project for Salesforce

It is said that the way you take the first step decides how you will take the last one. A successful migration project begins with preparing a well thought out plan. At Cetrix, we divide every data migration plan into six phases:

1. Preparation: discovery, planning, assessment of data and defining the procedures to follow in each step of the project and assigning tasks to project team members.
2. Extraction: exporting data from the legacy system, analysis and validation, creating backups, etc.
3. Cleaning: data quality definition, data profiling, and data discovery. Error identification and fixing, appending missing data, deduplication, etc.
4. Mapping: mapping fields between the two systems and adding missing fields. Creating a mapping template, creating a migration workbook that holds the data mapping for each object involved in the process, completing the mapping based on the workbook, and saving (exporting) the final mapping.
5. Loading: importing data into the new org in increments and in a top-down order (e.g. loading master objects before details), testing each import before starting the next increment.
6. Testing and Validation: profiling and discovery of the data in the new system, identifying errors and fixing, finding format mismatches and fixing, paying special attention to custom fields, objects, and codes, and finally issuing final approval to close the migration operation. 

Table of Contents

1. Customer Enterprise Data Integration Best Practices
2. Agile MDM Cloud Integration and Enterprise Service Bus Platforms
3. Challenges when Introducing an Enterprise Service Bus Platform

Success Factors

Each phase of a migration project is key when measuring overall success , but the Critical Success Factor is the mapping of the data. If the mapping operation is performed well, all other problems can be taken care of with proper assessment and planning. However, if the mapping is not done correctly, everything must be deleted from the new system and the mapping and loading operation repeated, which will cause a considerable delay and cost overrun.

Cetrix places special care on the mapping operation in every migration project it executes. In larger projects, experienced engineers are assigned to this task to avoid trial-and-error methodology often used by some IT staff that do not have the required qualifications and experience.

At the end of the day, the success of a Salesforce migration project depends on proper planning and careful cleaning and mapping of data, all performed by qualified and experienced data engineers.

Customer data silo challenge from the customer-centric standpoint  

Customer lifecycle management in the B2B marketplace

Introduction

Over the last 20 years, technology evolution has modified the environment where electronic transactions through a B2B or B2C marketplace take place. Transactions that used to be posted from a stable, rigid office environment have been replaced with ones that can now be posted from anywhere using any device.

This is an ongoing process, since there are further adjustments and improvements to be made in B2B marketplaces in how customer services are delivered throughout the whole customer journey. Recent studies have shown that mobile and social media marketing budgets are increasing, with figures for mobile marketing budgets expected to exceed 127% within three years, and marketing analytics budget is expected to increase by up to 376% within three years.

Despite this evolution in how business is done and the positive benefits seen from deploying robust CRM solutions, isolated customer data silos have increased exponentially which has led to disparate information about customers and their buying habits. In fact, on a routine basis, data is copied from one system to another in a manual fashion and as a result, ERP and/or CRM tools are updated only when necessary. This leads to major issues in understanding customer journeys, their buying habits, and the ability to analyse and understand these things.

In this article we will try to present why having customer data silos is a major problem for your business, and what the best practices are to overcome this issue.

Technology evolution in the B2B marketplace

Technology evolution has a prime role in B2B marketplace transformation. It is most notable in the cloud technology, big data analysis, AI realization and mobile technology areas. Let us consider these one by one.

Cloud technology

Cloud technology is the foundation of B2B transformation. Due to the relatively low costs associated with cloud technology, there is very little up-front initial investment. For example, prior to the evolution of cloud technology, a data analysis environment would require storage and processing power that only a large enterprise could afford. With cloud technology, a data analysis system is provided as a service with no initial investment required.

Cloud’s service nature has two major consequences:

• Agility. Companies are able to adjust to and adopt new trends, modify internal processes easily and effectively and do so with minimized cost. Today, agility has proven to be one of the strategic advantages in the B2B market.
• Decentralized IT procedures. Since applications are everywhere in the cloud, and don't require hardware resources or strict internal procedures to follow, the management of applications has partially passed from IT to individual departments. This has led to increased independence of departments when choosing satellite applications, and, as a result, increasing the spread of overall customer information to various data ‘islands’.

Big data analytics

Customer data is everywhere. It can be stored in an ERP or CRM database, it can be in social media profiles, and customer activity can even be monitored in real time. But in order to be useful, customer data should be collected, processed, and analyzed.

Big data analysis techniques provide the means for achieving the above: data is collected from several sources, it is then cleansed, consolidated, and processed. In B2B organizations, data analysis provides information regarding customer profiles, trends and behaviors. Based on this information, marketing decides on the right way in which to engage prospects and customers.

Data silos are the major cause for not having good quality data, and, as a result, have a direct impact on data analysis.  

Artificial intelligence realization

Artificial intelligence has moved from the academic environment to real life. Throughout the buyer journey, customers and prospects are exposed to a plethora of AI technology: bots, recommender systems and sophisticated models that predict behaviors and trends. All of this technology is aimed at helping businesses understand their prospects and customers in order to make decisions that will effectively support the customer or prospect better.  

Mobile technology

With the abundance of mobile technology available, B2B transactions can now be performed from anywhere using mobile phones, tablets and even sensors! Since devices such as mobiles have a simple man-machine interface, but do not have the presentation capabilities of a desktop computer, placing an order using a mobile device should be simple and straightforward, and specifically designed for mobile. Furthermore, the regular features of a webstore, such as pricing or product selection, will have to be be kept simple, providing only important information for those accessing using a mobile device.  

Behavioral changes

But this technology evolution has not only affected the means that a B2B user is doing business, but also the business itself. Consider the case of a customer visiting an online web store. The customer has all the related information available, including product features, logistics, pricing and support information. Content that is delivered to the user is personalized, driving then from visiting to purchasing. Bots can provide 1st level support, while recommendations guide similar alternative or related products.

Recent Google surveys have shown that the average B2B user is already two-thirds of the way along their buyer journey before purchasing. Everything is available and easily accessible. It is the B2B user that is looking for companies to satisfy his/her needs, and it is technology that provides the means for this. But it is critical for a B2B marketplace to be able to satisfy the customers needs in this two-thirds part of the customer journey. Marketplaces that are not able to cover these needs online are losing opportunities.

Data integration challenge

The impact isolated customer data silos has  

Having described technologies and behaviors that govern marketing and sales processes today, it would be logical to conclude that there is much room for improvement in the whole customer journey process. But in reality, things are different:

• According to the DMA Statistical factbook, understanding and integrating cross-channel customer data is the prime marketing and sales challenge.
• The CMO Survey reveals that integration of customer information through different channels have shown no improvement.

Within a company, it is very common for departments to adopt solutions that do not change habits and procedures. If, in this high level scenario, we add possible data sources that support applications referring to the same entity (customer), as well as legacy sources, this may result in a chaotic customer data landscape. Furthermore, since no IT analysis is required to adopt an application, departments are free to develop other complementary applications that manage the same customer entity without exchanging any information between them.

It is quite common that the sales, marketing, and support departments work independently, having separate budgets and metrics to measure performance. In this situation, leads come to sales without being qualified and with no prior knowledge of who these leads are, or a member of the sales team has to deal with an existing customer who has had a negative support experience.

With that in mind, acquiring new customers could be up to 25 times more cost-intensive than keeping existing ones. Furthermore, a company should focus on extending customer retention, since it will result in an increase in ROI.

In order to help illustrate the significance of having a single and consolidated view of customer data, consider a simplified view of the customer journey:

Figure 1:Customer journey

Customer journey components, as listed above, may be grouped into categories:

• Identification components: These may include simple identification (e.g. sign in) and/or user intent analysis processes, web crawling and other processes that will form a picture of who the B2B user is.  
• Marketing and marketing automation components: These can help adjust content, campaigns, etc according to the customer persona.
• Sales and sales automation components: These components help assign customers to a salesperson, or adjust prices and discounts, manage deliveries, etc.
• Support components that will provide after sales support, etc.

All processes in the above example require a single consolidated view of customer data in order for the B2B business to be efficient and effective. Data used as customer identification, or sent to scoring and recommender systems should point to the same customer regardless of the data source they come from. The lack of master data integration causes malfunctioning in the overall process, with a direct impact on effectiveness. Today, having isolated systems with individual customer data silos, is the most significant cause of B2Bs not being able to increase their ROI.

Isolated data sources are the most significant obstacles to having maximum optimization of the overall customer journey. Lack of effective integration often results in erroneous conclusions and subsequent actions:

• How can user intent be interpreted correctly if the wrong customer is pointed to?
• How can a customer be scored if his/her profile does not correspond to the actual one?
• What aspects of the marketing strategy apply to the customer and how effective will they be?

With data on the prospect or customer stored in so many different places, it is nearly impossible to form a cohesive, single view of who the prospect or customer is, what their behaviour is, and how they should be marketed and sold to. This, in turn, has a dramatic impact on the buyer journey, which in turn has an impact on the businesses bottom line and ROI. Not only this, but by not having a single 360 degree view of the customer in one place it affects cross-departmental organization and effectiveness.

Application Variety

Cloud technology, with low initial investments costs, drives companies to use software applications as a service. Companies are able to have optimum services from the very best vendors with minimal cost. Furthermore, entire business processes may consist of composite applications that run completely in the cloud through different software service modules.    

SaaS solutions are the building blocks of the B2B user journey:

• 3rd party data providers are used to provide B2B user behaviors, market trends, and basic customer segmentation.
• Business intelligence and analytics engines are used to gather B2B user information from several sources, and then clean and consolidate it. Sources may be both structured and unstructured.
• Smart systems make use of artificial intelligence and they are used to model and score customers, hence providing the marketing infrastructure for the customer journey.  
• Marketing automation systems transform results to actions. While sales automation helps accelerate conversions.
• CRM and/or ERP systems provide the backoffice for any financial transactions that take place.

All of the above sub processes may be offered as a service, and in the worst case, each of these will have their own customer data silo also.     

Figure 2: Customer data silos

With the huge availability of applications, data has spread into several data silos. Furthermore, due to the agility that the cloud offers, it has accelerated the adoption of individual applications that in turn, manage their own pieces of customer data.

This trend is going to increase in the near future according to recent studies:   

• A move from hierarchical data architectures to decentralized ones is anticipated.
• In order to reduce cost and optimize space usage, tier based storage will be used and several data management functions like analytics and BI will be outsourced. This will apply not only to smaller or medium businesses but also to larger ones.
• Data privacy, security and governance assurance (i.e. GDPR).
• Private-public cloud hybrid solutions will be used for increased productivity and efficiency across organizations.

Data Integration and the lack of orchestration

Today, sales and marketing application vendors, regardless of being cloud based or not, offer integration capabilities in master data and transaction management. Application interfaces (APIs) using industry standards along with advanced integration modules with logging, monitoring, and reporting features have been developed in order to manage these data interface flows.  

But despite the technical capacity that enterprise application and master data integration platforms offer, the problem still persists, mainly due to two reasons:

• Lack of proper master data integration architecture. Often they have not fulfilled basic design principles such as:
• There should be a single customer reference. This should be stored in a single repository and be synchronized to all satellite application. 
• Each application manages data that is relevant to it. For example, customer financial information is managed by an ERP, whereas sales information is managed by a CRM.
• Often integrations and synchronizations are not in real time and result in overall process execution delays. For example, customer financial transaction information may be sent to other modules through a batch process on a day-to-day basis..  

The role of an orchestrator

The architecture and performance issues that are listed above, describe the lack of an application or framework that will orchestrate the customer data integration process. This may include the following:

• Manages customer data, providing the single point of truth for customer data.
• Governs the integration process, delivering customer related information to all subsystems.
• Logs and monitors the integration process, giving tracing and reporting capabilities.
• Have analysis and self improvement margins.

 Figure 3: Orchestration of customer master data

Marketing and sales alignment

Having marketing and sales alignment as an objective, one should consider what tools are needed to achieve it. Though integration and data orchestration define a framework that someone should move within, the real world proves that this isn’t always adequate:

• Management making a strong commitment to the success of the project has proven to be more important than the tools being used.
• Cloud technology and SaaS architecture is here to stay. Departmental independence can be managed only through an integration hub, which usually is the IT department. New applications can only be deployed if they have real time integration capabilities that are reliable and maintainable.
• Integration platforms can offer agile and easy to implement integrations, with reporting and logging capabilities. Integration architecture strategies should be applied in order to ensure that a customer entity is described in a unique way throughout the marketing and sales departments.

Conclusion

Despite the fact that companies have realized the performance impact of having isolated customer data silos, these still continue to be created and to grow. A cross department strategy having at its center IT policies that require integration capabilities maintained centrally should be enough to break this trend.  

If you are seeing this trend in your organization, and are struggling to overcome this difficult obstacle, please contact us. We are experts in digital transformation and integration. One of our consultants will kindly chat with you to discuss your needs and requirements and then outline a cost effective solution.