Street-to-Shop

Research Project by Dr. Julia Lasserre & Dr. Katharina Rasch

Have you ever seen somebody out on the street and thought “Wow, those are some great shoes, wonder where I can buy those?” Or were inspired by your favourite fashion blogger and looking for similar articles (for example on Zalando)? The task of identifying fashion articles in an image and finding them in an online store is called street-to-shop [Liu et al] and has been the subject of several scientific articles. And yes, there are already apps out there that implement this, but there is still room for improvement.

The picture below illustrates the scenario we are currently working on. On the left side is the image of a person wearing an outfit, on the right side are, on the top, the items composing this outfit; on the bottom, random items not present in the outfit. The algorithm should find all the top items and reject all the bottom ones. In practice, a satisfying output would be something like the boxes displayed around some of the items.

street-2-shop: results

Everyday Zalando creates such outfits for its website, photographs them and stores the list of articles used in the outfit, thereby providing us with large amounts of curated data, which keep growing at a healthy pace. This means that we can use state-of-the-art learning techniques such as deep nets which have revolutionized computer vision. Taking advantage of our in-house fashion expertise and of scientific advances from our own lab that are tailored for fashion, such as Fashion DNA, Zalando has the potential to become a strong player in the field.

Some interesting questions street-to-shop poses are:

  • How can we deal with variations in image quality, lighting, background?
  • How can we deal with different human poses and article distorsion?
  • How can we find the right articles in such a large database in real-time?
  • If we can’t find the actual article in the image, our customer might also be happy with a similar article. But what does it mean for two clothing items to be similar? This is one of the most critical aspects because it affects customer satisfaction. And one of the least studied in the field because it is extremely hard to evaluate performance. In theory Zalando could use its platform to run experiments, gather real customer feedback and provide invaluable insights.

We think this project is exciting because it is really about the customers needs.  Solving street-to-shop promises new means of making fashion searchable, i.e. of helping our customers find the fashion items they are really looking for. It also includes a long list of computer vision and machine-learning problems to solve, some of which could lead to contributions to the scientific community.

[Liu et al., 2012] Liu, S., Song, Z., Liu, G., Xu, C., Lu, H., and Yan, S. Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 3330–3337.