The Future of Retail Ops

Working with Google's retail industry team to build an experience for their larger presence at the National Retail Federation's 2019 show in New York City, we prototyped and built a machine learning-powered smart mirror to demonstrate the ease of use for large retailers and brands to incorporate Google's industry-leading machine learning and data analysis solutions. Starting with knowing the need to show how the experience can benefit both the customer and the employee, we built a smart mirror system that delivers real-time product reviews and recommendations while an employee dashboard used the same information to display up-to-date analytics about the products interacted with and traffic visualizations.

Initial prototypes were built with a mixture of tfjs and AutoML, where we used custom models to track emotion, extract faces, classify objects, and capture poses. As the concept was developed further, we switched to prototyping custom models using AutoML, creating a tool that allowed us to classify specific products. Due to latency concerns, a local copy of the data is stored in a SQL database, while also being sent to the cloud to run through an analytics pipeline using BigQuery and Dataflow to determine and send recommendations.

Moving into production, we came up with a method to ensure correct personalization across users by using facial detection to create "sessions." Because the mirror itself sets up a 1:1 interaction, we were able to avoid the development overhead of tracking personalization for many users at once.