Challenges of a Large Data Migration with Millions of SKUs

Today we’re going to be talking about migrating an eCommerce store that has over a million-plus SKUs. We’ve done tons of eCommerce migrations moving from BigCommerce to Shopify or Volusion to BigCommerce.

eCommerce Migration: The Difference With Million-Plus SKUs

What makes the difference?

Having over a million-plus SKUs. This isn’t just a typical migration.

Can you tell me a little bit about why someone would ever need to migrate once they reached that amount of SKUs?

Usually, the main cause of this migration is because you know you have reached the limit of the current platform. So usually that happens when you’re on a platform that you started when you have like probably 20000 SKUs, but over the course of the years you’ve been adding new products and new vendors and new things and you get to a limit. That the platform can not handle that.

At that time you start having issues with page loads. You start having issues with you know, monitoring the back end of the store because it can’t, it’s not ready for that. The product forms for different levels. And whenever you could reach that limit, you need to start thinking about scalable enterprise-level solutions.

What would be a platform that I can go to once I reach that level of SKUs?

We have done a lot of them with BigCommerce, Shopify might be another solution. You can start looking at, you know, enterprise-level solutions, like that.

How does the migration process get started? You know, what’s the first step?

We take a look at the entire, operations, how everything functions on the existing platform. We check the data and how clean it is. Data makes all the difference for the next few steps.

If they have any integrations with existing CRMs or accounting system, we then take a look at the existing platforms that can handle this. And we make like a list of pros and cons of each platform, so we can suggest the best platform for each business. Everybody has different needs.

We basically take a look at integrations that they currently have that can be replaced with built-in solutions on these platforms. And we find the best, you know, cost benefits.

Suppose I’ve found the right solution. I found the right platform. How do I get started? What’s step one?

We look at data migration. You know, that’s one of the most important parts. So whenever you do a migration of the platform, you want to keep everything, you know that you currently have. Data migration is tricky when you’re dealing with one million-plus SKUs.

You have many moving pieces. Data migration is pretty simple when you’re migrating a 10000 SKU store. You’re going to run the migration multiple times until you get it done. But with a million-plus SKUs, you know, these processes might take hours, if not days, depending on the API limitations that you’re dealing with.

Migration Checklist: Preserving Data

For us, for developers, our team would take a look at all the data. They want to map it to the new system in the most efficient way so that we consume fewer resources, you know, using this API, some playing with these API limitations.

Data migration is very important. So is order history. We keep all the data working in the same way.

In this new platform, when an existing customer comes back after the migration that they log in, they can see all the orders that they had through the years in the previous platform. A lot of things need to happen. Sometimes these platforms do not store the data in the same way. So we need to do but adjustments in order to make it work in this new platform.

Let’s say I get all the data migrated over, what’s the next step after that?

When you get 1 million-plus SKUs, you also have a lot of integrations with third parties, like accounting systems and CRMs. We then focus on how are we going to, how are we going to migrate these systems?

Usually, we tried to find, you know, a third party that integrates these already. Sometime there might not be a solution with the new platform. So we build in house, you know, connectors between the eCommerce platform and the third party, you know, that they’re using for like a CRM for example.

That process might take a long time depending on how many integrations and customs solutions you have. That’s basically it. And then after that, of course, we have the ACO processes with the 301 redirects and all that.

Migrating Custom Functionalities During an eCommerce Replatform

What about functionality? If I have a custom functionality on my existing site, how will it migrate and how’s it going to function?

Most of the time, we have to do that from scratch because these custom functionalities are built to the platform it currently occupies. When we migrate that, we get the idea and we might use any server-side thing that’s not tied to the platform, but most of the time it needs to be built out completely from scratch.

In our case, what we have found out is like, you know, if you are getting out of a platform it’s because you have a lot of things that you’re not happy with. And then what we usually do is like we take that since we’re going to rebuild the functionality, we try to, you know, also improve it, you know, try to make it better. Taking into account that we’re going to take all the parts of this new platform.

During the overview where we do an initial deep dive, there are some functionalities that will be native to the new platform. The current platform that you’re on right now didn’t have that. So you have to get it done custom.

Now you’re going to move to a platform that’s already built-in. We do that during the scope out process. We take a lot of time upfront to understand your business and to understand why you have specific functionality or how can that be replaced on this new platform.

Then we make a choice. Many decent platforms exist, but that doesn’t mean that all of them are good for your business. So we do a lot of upfronts and, deep-dives just to make sure that we’re choosing the right platform.

SEO And Successful Migrations

The SEO process of the migration is time-consuming. You’ve got a million-plus SKUs. That’s a million-plus 301 redirects that have to get implemented. There’s a lot of room for error. The more URLs that are indexed, the more URLs in their sitemap.

With a larger potential for error, you have to take more care. Stay on top of the process. That’s why for migration this size, you need a full team working on this and multiple eyes. I would say perform at least five, six to 10 rounds of QA.

For a project of this size, we need to work very closely with the customer and their team as well. Any store that has 1 million-plus SKU has a lot of processes, and it’s very important that this migration doesn’t break anything else.

Many development companies just focus on the technical part that involves the switch. But we also take a look at the big picture, you know, from the business’ standpoint.

“How is this going to affect anything else?” we ask. Then we answer to prevent migration issues.

The design needs to get then migrated over as well.

If you’re on an eCommerce platform that you feel that you’ve outgrown and you have a million-plus SKUs and you’re looking to migrate, you’re going to need a team of developers working on this. And this is very important. It could make or break your migration.

You know your site if something goes wrong. Let us know if you have any questions and we’ll talk to you guys next week.