AWS Summit Online ASEAN – The Art of the ML Deal With Arne Jeroschewski

Our CEO Arne Jeroschewski was invited to present with Parijat Mishra from Amazon Web Services at the AWS Summit Online ASEAN in May to share our experience working with AWS to improve the e-commerce checkout experience with Machine Learning.

Watch Arne’s presentation on YouTube here or read a transcript below.

Parijat: Now I hand over to Arne from Parcel Perform to discuss how they have worked with our ML Solutions Lab in an interesting problem on ecommerce logistics. Over to you, Arne.

Arne: Thank you Parijat. I am Arne, the founder of Parcel Perform. Appreciate the opportunity to share our machine learning journey at Parcel Perform with you today. 

Let me start off with giving you a quick overview of our work at Parcel Perform. We have set out to help e-commerce merchants to support them with an easy-to-use SaaS solution for their e-commerce logistics. Our solution allows them to significantly improve their post-purchase customer experience and therefore drive customer satisfaction. Our logistics data integrations and tools make it very easy for any merchant to deliver a fantastic experience after the checkout.

E-commerce could be so easy: A consumer places an order, the merchant hands over the parcel to the carrier and the carrier delivers it to the customer. What could possibly go wrong?

But reality looks very different from this.

Merchants need to work with several carriers as they have different product categories, ship to different locations or do not want to rely just on one service provider. That’s where the problem starts… Merchants need to integrate with many carriers with all their different requirements and different levels of tech sophistication. Merchants have little visibility on what is happening to the parcel after it is handed over. Neither does the consumer, who in most cases does not know where to get updates. He then calls the merchant and the carrier creates high customer service cost all around. A frustrating experience for everyone involved with high friction and high cost. We at Parcel Perform aim to change that!

Headquartered in Singapore, our more 50 team members are now distributed across our 4 offices in Singapore, Vietnam, Germany and the UK. We have a powerful tech team with over 25 team members being either developers or Product Managers.

With this, we already won several remarkable customers in our short 4 year history as a company.

We are working with e-commerce marketplaces like Zalando, Wayfair and Catch Group, brands like Nespresso and Decathlon, logistics companies like DHL, bpost and Schenker and B2B supply chain players like BMW. Every one of them seeking to manage logistics a little bit better.

On our journey to create a better e-commerce experience, we stumbled across one area that continues to be a horrible experience for consumers… tracking. Carriers with challenging to use tracking pages, expose incomprehensible tracking data to consumers worldwide…. “Parcel is now at this depot and then at depot”… asking the consumer to figure out from these cryptic data, when a parcel will likely arrive. 

Wouldn’t it be much better if we could just tell our customers when the parcel will likely arrive… saving them from having to become an expert in parcel tracking data interpretation. 

Makes sense right… but it will surprise you that only less than 5% of parcel carriers worldwide offer this.

And the ones that do … may not even have such an excellent track record at meeting this delivery promise. This is where we at Parcel Perform thought this is something we can change and fix. 

We believe that this a problem that one can fix with machine learning. Logistics processes are complex and different for every carrier and player in the supply chain. They follow logical patterns that one can try to understand. Second, there are enough data available for us to understand these processes and make predictions. Thus, we truly believe that this problem is solvable with machine learning. 

If it is solvable, then everyone should be able to do it. Right?

We do not think so as we at Parcel Perform bring a few unique things to the table.

First, we are seeing hundreds of millions of parcels around the world giving us a unique set of data to start with.

Second, we are standardising the data for us to understand every step of the logistics process. Turning random events into structured data… something nobody in the industry does to the same level of granularity. This was significant news for us… it can be solved with machine learning and we are the best people to do it. So we got started. 

Well… what we thought to be an easy problem to solve, turned out to be much more complex given logistics processes. One deals with cut off times, delays, weekends and public holidays, poor weather, customs, missing scan events and much more. 

Thus, we needed to acknowledge that building a ML product around predicting the Date of Arrival for parcels will be much more challenging.

We reached out to the AWS ML solutions lab to bundle our respective expertise and crack the nut together.

This is when our collaboration started. 

First, we set up a joint project team around Nabeel and Sahithya, our machine learning product team at Parcel Perform, and Sujoy, Phil and Bernard from AWS to tackle the problem. 

With the team in place, we kicked off the collaboration in September last year and worked together as a team over the past months to solve the task at hand. In this collaboration, we conducted a number of workshops and had weekly check-in calls between all team members.

The AWS team helped us with sharing ideas and recommendations but also doing hand-on simulations and predictions themselves. As much as we learned a lot, our team very much enjoyed the collaborative working style between our teams. 

In this collaboration, we can set up and refine our Parcel Perform machine learning process and tackle all elements that allow us to build a great product around Date of Arrival predictions for us:

  • First, we needed to understand and transform our data for this prediction. We could resolve many errors that us have found in the predictions through feature engineering and data imputation.
  • As a second step, our team picked the right model and build the machine learning and data pipeline based on AWS broad range of machine learning tools, foremost Sagemaker.
  • Lastly a step we underestimated at first – we build the business logic that allows us to interpret the modelling results and make it a usable product for the customer experience of our merchants. Here, we realised that the highest accuracy model does not yield the best experience for our customers. Defining the right measures of success and refining them over time has proven to key to our product. We ultimately spend about 60% of our time working on the business logic – something that took us by surprise. 

In summary, iteratively improving this process of 6 months was key for our success.

Nobody gets machine learning right first time.

And successful we were. Our Parcel Perform Date of Arrival prediction platform is now in a marketable state.

I have included a few prediction results here from a large customer of ours across millions of parcels and over 20 carriers and countries. You see that we could accurately predict the delivery date in about 91% of the cases and then with our update notifications move this number up to 96% right before delivery. These are numbers that are significantly better than the numbers that the few carriers with Date of Arrival predictions can offer…. all independent of the carriers just based on machine learning and historic data. This collaboration has been a tremendous success for us as a business and for our customers. Please reach me if our Date of Arrival engine can be of help to you. We are happy to set up a proof of concept for you. 

Let me summarise a few of the key lessons that we at Parcel Perform had on our journey to developing a machine learning product.

First, just hiring a few data scientists and throwing them at a problem is not enough. It is an excellent start, but you need a lot more to succeed. They have to be deeply integrated into the business as machine learning is merely a tool to solve a business problem. The business needs to be front and center of your machine learning efforts.

Second, do not use model accuracy to determine if a machine learning model is good for your business applications. We painstakingly learned that success for our use case is determined by 10 unique business KPIs that may or may not be correlated to accuracy. We had to make several tradeoffs on accuracy to get to the best model for us. 

Third, do not expect the machine learning model to solve all your problems. Abstracting the modelling results and putting overriding business rules on top made our solution improve from good to great. Be ready to augment your machine learning model with a lot more to make it right for you. That’s also when your machine learning efforts are in need for business input the most.

Last, there are many things you need to make machine learning work for you to solve your business challenges…. the right team with the right skills, the right tools and processes and a deep integration of the team in the business to give them access to all the expertise you have your organization. 

Our collaboration with the AWS ML Solutions Lab made us realize that and helped us to develop a marketable key product for us. We are very grateful for this and appreciate all the support we have gotten on this journey. Thanks a lot.