July 17, 2024
Illustration on-line issues: sensible end-to-end diversification in search and recommender programs | by Pinterest Engineering | Pinterest Engineering Weblog |
Pinterest Engineering
Pinterest Engineering Blog

Bhawna Juneja | Senior Machine Studying Engineer; Pedro Silva | Senior Machine Studying Engineer; Shloka Desai | Machine Studying Engineer II; Ashudeep Singh | Machine Studying Engineer II; Nadia Fawaz | (former) Inclusive AI Tech Lead

Different women in white button down shirts and blue jeans

Pinterest is a platform designed to carry everybody the inspiration to create a life they love. This isn’t solely our firm’s core mission however one thing that has turn out to be more and more necessary in as we speak’s interconnected world. As know-how turns into more and more built-in into the every day lives of billions of individuals globally, it’s essential for on-line platforms to mirror the various communities they serve. Enhancing illustration on-line can facilitate content material discovery for a extra various person base by reflecting their inclusion on the platform. This, in flip, demonstrates the platform’s means to satisfy their wants and preferences. Along with improved person expertise and satisfaction, this could have a optimistic enterprise affect via elevated engagement, retention, and belief within the platform.

On this submit, we present how we improved diversification on Pinterest for 3 completely different surfaces: Search, Associated Merchandise, and New Person Homefeed. Particularly, we’ve developed and deployed scalable diversification mechanisms that make the most of a visible pores and skin tone sign to assist illustration of a variety of pores and skin tones in suggestions, as proven in Determine 1 for style suggestions within the Associated Merchandise floor.

Difference in recommendations of women in white button downs and blue pants
Fig. 1. Facet-by-side Associated Merchandise suggestions for the question Pin “Shirt Tail Button Down” proven on the left. Prime proper: earlier expertise with out diversification. Decrease proper: present diversified expertise.

The top-to-end diversification course of consists of a number of parts. First, requests that may set off diversification must be detected throughout completely different classes and locales. Second, the diversification mechanism should be certain that various content material is retrieved from the massive content material corpus. Lastly, the diversity-aware rating stage must stability the diversity-utility trade-off when rating content material and to accommodate diversification throughout a number of dimensions, such because the pores and skin tone seen within the picture in addition to the person’s numerous pursuits. Multi-stage diversification permits the mechanism to function all through the pipeline, from retrieval to rating, to make sure that various content material passes via all of the phases of a recommender system, from billions of things to a small set that’s surfaced within the software.


Superior search and recommender programs, which function on the large-scale of tons of of tens of millions of energetic customers and billions of things, are usually very complicated and have a number of parts. These programs usually comprise two main phases: retrieval and rating. That is generally adopted by extra enterprise logic: Objects are retrieved and ranked, then the listing is surfaced to the person.

Item Corpus arrow 10⁶-10¹⁰ to Candidate Retrieval arrow 10²-10³ to Ranking arrow 10–100 to User interface/surface
Fig. 2. Massive-scale recommender programs can broadly be categorized into two phases going from an merchandise corpus to suggestions: retrieval and rating.
  • Retrieval: The retrieval stage consists of a number of candidate turbines that slim down the set of candidates from a big corpus of things (within the vary of 10⁶ to 10¹⁰) to a a lot narrower set (within the vary of 10² to 10⁴) primarily based on some predicted scores, such because the relevance of the objects to the question and the person.
  • Rating: Within the rating stage, the aim is to seek out an ordering of the candidates that maximizes a mixture of targets, which can embrace utility metrics, variety targets, and extra enterprise targets. That is normally achieved by way of one or many Machine Studying (ML) fashions that generate rating(s) for every merchandise. These scores are then mixed (e.g. utilizing a weighted sum) to generate a ranked listing.

Variety in suggestions

Variety Dimension: Diversification goals to make sure that the ranked listing of things surfaced by the system is various with respect to a related variety dimension, which might embrace express dimensions reminiscent of demographics (e.g., age, gender), geographic or cultural attributes (e.g., nation, language), domain-specific dimensions (e.g., pores and skin tone ranges in magnificence, delicacies sort in meals), business-specific dimensions (e.g., service provider sizes), and likewise different implicit dimensions that will not be expressed straight however will be modeled utilizing latent representations (e.g., embedding, clustering). Whereas on this work we current an instance of pores and skin tone diversification, the proposed strategies usually are not restricted to this single dimension and might assist diversification extra broadly, together with intersectionality of a number of variety dimensions. We denote the set of teams underneath a variety dimension as D, and every particular person group is denoted by 𝑑.

Variety Metric: For a given question, we outline the top-k variety of a rating system because the fraction of queries the place all teams underneath the variety dimension are represented within the prime ok ranked outcomes for which the variety dimension is outlined. As an example, within the case of pores and skin tone ranges, an merchandise whose picture doesn’t embrace any pores and skin tone wouldn’t contribute to visible pores and skin tone variety. Thus it is not going to be counted within the top-𝑘 and shall be skipped within the variety metric computation.

Multi-stage diversification: Each retrieval and rating phases straight affect the variety of the ultimate content material surfaced within the software. The range metric on the output of retrieval stage upper-bounds the variety on the output of rating. Therefore, the retrieval layer must generate a sufficiently various set of candidates to make sure that the rating stage has sufficient candidates in every group to generate a remaining various rating set. Nevertheless, variety on the retrieval stage shouldn’t be a enough situation to ensure {that a} utility-focused ranker will floor a various ordering on the prime of the rating the place customers usually tend to focus their consideration and to work together with objects. Therefore, each the retrieval stage and ranker additionally must be diversity-aware.

Triggering logic: An actual-world system might obtain requests that span a variety of classes, reminiscent of style, magnificence, residence decor, meals, journey, and so forth. The range dimension of curiosity depends upon the applying. For instance, pores and skin tone vary diversification is relevant to style and wonder, however to not residence decor. Thus, upon receiving a request, the system wants to find out whether or not to set off diversification in keeping with the dimension of curiosity. The triggering logic must account for the variety dimension, the applying, the manufacturing floor, and the native context, reminiscent of nation and language, and will be primarily based on heuristics or ML fashions, reminiscent of fashions that predict the class of a question. On these components, together with person analysis and knowledge evaluation on pores and skin tone associated Search question modifiers that spotlight a necessity for variety in comparable requests, we resolve to solely set off skintone diversification for magnificence and style classes in Search, Associated Merchandise, and New Person Homefeed.

We begin with a deal with the rating stage to attain diversification of outcomes since it’s the final stage of a recommender system. As an alternative of utilizing boosters or discounting scores, which have a tendency so as to add vital tech debt in the long run, we leverage a diversity-aware rating stage that takes as enter an inventory of things with utility scores and their variety dimensions and produces a rating in keeping with a mixture of each targets. The primary method we used is a category of easy grasping rerankers, e.g. Spherical Robin (RR). Given an ordered listing of things 𝑦, . . .,yₙ, we assemble |D| variety of ordered sub-lists corresponding to every pores and skin tone vary and containing objects which have a utility rating above the brink. Then, we re-build a ranked listing by greedily choosing the highest merchandise of every sub-list. All of the candidates that don’t belong to a sub-list, as an illustration as a result of they don’t have a pores and skin tone vary or have utility scores under the brink, will be left on the similar place as within the authentic listing or assigned to a random sub-list.

Fig. 3. Illustrative examples of Spherical Robin and DPP utilized to a utility-ranked listing. Every block is an merchandise within the ranked listing and the colour denotes the pores and skin tone vary of the merchandise picture. (a) Ranked listing obtained after making use of Spherical Robin is re-ordered such that the distribution of pores and skin tones is extra uniform within the prime positions. (b) Ranked listing obtained after DPP (for a selected worth of 𝜃) permits for optimizing a list-wise goal to commerce off utility and variety of the preliminary ranked listing.

RR is an easy, intuitive, and environment friendly method to diversification; nevertheless, it doesn’t all the time stability variety and utility. As well as, it doesn’t simply generalize to a number of completely different variety dimensions or a number of utility rating thresholds. To keep away from these limitations, we suggest a multi-objective optimization framework, i.e. Determinantal Level Course of (DPP). A DPP is a machine learnable probabilistic mannequin utilized in physics for repulsion modeling and extra not too long ago in recommender programs. DPPs are notably helpful in ML for duties reminiscent of subset choice, the place the aim is to pick a subset of factors from a bigger set which can be various or consultant in some sense. The fundamental concept behind a DPP is to mannequin the chance of choosing a set of things 𝑌 from a set of measurement 𝑁 because the determinant of a kernel matrix 𝐿ᵧ, the place 𝐿 is a kernel perform that encodes the utility of the objects and the similarity between pairs of things, and 𝐿 is the kernel matrix of the subset 𝑌. The determinant of 𝐿ᵧcan be regarded as a measure of how unfold out the factors in 𝑌 are within the characteristic area outlined by the kernel perform 𝐿. The diagonal entry 𝐿ᵢᵢ represents the utility of the 𝑖ᵀᴴ merchandise, in our case the rating with which the objects had been initially ranked. The off-diagonal entry 𝐿ᵢⱼ, nevertheless, represents the similarity between the objects, which in our case depends upon the variety dimension (e.g. the pores and skin tone vary within the merchandise picture). The kernel is chosen such that 𝐿 is a optimistic semi-definite (PSD) kernel matrix and has a Cholesky decomposition, and therefore 𝐿 will be written as:

L = U phi phi^TU^T = USU^T

the place 𝑈 = diag(𝑒^(𝜃𝑢1)), . . .,𝑒^(𝜃𝑢𝑁 )) is a diagonal matrix that encodes the utility uᵢ of every merchandise, 𝜃 is a parameter that governs the trade-off between utility and variety, and Φ = [Φ₁, Φ₂, Φ₃, …, Φₙ ], the place Φᵢ is the characteristic vector for the 𝑖ᵗʰ merchandise.

For our use case, ΦΦᵀ is the symmetric similarity matrix, which we henceforth denote by 𝑆. Lastly, given a price of 𝜃 and kernel matrix 𝐿, the aim is to discover a subset Y that maximizes the determinant of 𝐿ᵧ:

Y = argmaxY  y1, …, yN det(Ly)

Using determinant implies that, primarily based on the selection of kernel matrix, 𝑌 would come with objects with excessive utility scores whereas avoiding ones which can be much like others within the subset. Discovering such a subset 𝑌 of a given measurement 𝑘 is an NP-hard downside. Nevertheless, due to its submodular property, it may be effectively approximated utilizing a grasping algorithm.

Determine 3(a) exhibits an instance the place RR is used to diversify a ranked listing of things with respect to 4 teams 𝑑,𝑑,𝑑,𝑑₄. Determine 3(b) exhibits a hypothetical instance of how DPP would rerank as in comparison with RR given an applicable worth of parameter 𝜃.

Compared to RR, DPP takes into consideration each the utility scores and similarity and is ready to stability their trade-off. For a number of variety dimensions, DPP will be operationalized with a joint similarity matrix 𝑆𝑌 to account for the intersectionality between completely different dimensions. This may be additional prolonged to a perform the place, for every merchandise, all variety dimensions (pores and skin tone, merchandise classes, and so forth.) are offered and the return is a mixed worth that represents the joint dimensions. A less complicated possibility is so as to add a variety time period within the weighted sum proven in equation 4 for every dimension. Within the case of numerous variety dimensions, dimensionality discount strategies can be utilized.

Diversifying in the course of the rating stage will be difficult as a result of restricted availability of candidates from all teams within the retrieved set. The strategies proposed above reminiscent of RR and DPP are restricted to the set of candidates retrieved by completely different sources within the first stage. Subsequently, it could not all the time be attainable to diversify the rating stage for particular queries. To beat this limitation, we’ve developed three strategies to extend the variety of candidates on the retrieval layer. These strategies enhance the flexibility of rerankers to diversify at a later stage and are appropriate for various setups.

Overfetch-and-Rerank at retrieval: To extend candidate set variety, the Overfetch technique fetches a bigger set of candidates, which will be outlined to comprise a minimal variety of candidates from every pores and skin tone vary. For instance, if a candidate set of measurement Ok is desired, the neighborhood measurement will be expanded to Ok’ (Ok’ > Ok) to satisfy the variety criterion. To scale back latency, a hyperparameter Kmax is chosen in order that the overfetched set by no means exceeds Kₘₐₓ. The rerank technique selects a subset of measurement Ok from the overfetched set by performing a Spherical Robin choice of one candidate at a time from every pores and skin tone vary till Ok objects are chosen. Overfetching stops when the minimal threshold for every pores and skin tone vary is met or Kmax is reached.

Segments to Leaf to Root — Aggregate + Bucketize
Fig.4. A diagram of distributed ANN retrieval aggregating candidates from segments to leaves to the foundation primarily based on the gap metric whereas assigning prime Pins with every pores and skin tone to their corresponding buckets

Bucketized ANN retrieval: Approximate nearest neighbor (ANN) search is a broadly used retrieval technique in embedding-based search indexes. In such programs, customers, objects, and queries are embedded into the identical area, and the system retrieves the objects closest to the question or person embedding primarily based on a selected distance metric. Since computing pairwise distances for all query-item pairs shouldn’t be possible, approximation algorithms like k-Dimensional Tree, Locality-sensitive Hashing (LSH), and Hierarchical Navigable Small Worlds (HNSW) are used to carry out nearest neighbor search effectively. In large-scale recommender programs, these strategies are applied as a distributed system. The final structure of an ANN search system incorporates a root node that sends a request to some leaf nodes, which additional request a number of segments to carry out a nearest neighbor search in several subregions of the embedding area. To seek out 𝐾 nearest neighbors for a given question embedding, every section returns 𝐾 potential candidates to the corresponding leaf, which then aggregates these 𝑀 × 𝐾 variety of candidates to retain solely the highest 𝐾 candidates to the foundation. The foundation selects the highest 𝐾 candidates from 𝐾 × 𝐿 × 𝑀 candidates whose distances are computed in the course of the course of. Within the bucketization method, the aggregation step is modified to pick the top-𝐾 candidates and mixture the highest 𝐾𝑑𝑖 candidates from every pores and skin tone 𝑑𝑖 right into a bucket with top-𝐾𝑑𝑖 candidates for every pores and skin tone 𝑑𝑖. This helps protect prime candidates belonging to every pores and skin tone vary with out increasing all the aggregation graph.

Strong-Or to term1 > 3–1, 6, 9 and to term 2–1,2,3,4,5,7,8,9,10
Fig.5. An instance of how the Robust-OR operator ensures variety throughout retrieval for 2 question phrases, one with a minimal threshold situation

Robust OR retrieval: Within the Search course of, the retrieval stage includes changing textual content queries to structured queries utilizing logical operators like AND, OR, and XOR to slim or broaden the set of outcomes. To extend the variety of outcomes, a specialised logical operator known as Robust-OR is used. Robust-OR prioritizes a set of candidates that fulfill a number of standards concurrently, permitting us to specify what proportion of candidates ought to match every criterion. Robust-OR scans a restricted variety of objects and retrieves candidates that meet the required standards. If there are inadequate objects to meet the standards, it matches as many as attainable. Robust-OR acts as a daily OR at first, however promotes a criterion to be a essential situation throughout scanning to retrieve extra related outcomes. Candidates that fulfill the standards and wouldn’t have been retrieved in any other case will be added to devoted buckets to make sure they aren’t dropped within the latter phases of retrieval.

We deployed diversification approaches on three completely different surfaces on Pinterest primarily based on person suggestions to diversify particular experiences — particularly Search, New Person Homefeed, and Associated Merchandise. These surfaces had been consciously chosen retaining in thoughts person analysis and knowledge evaluation of person wants. On this part we current a number of sensible issues to deploy diversification approaches in an actual world manufacturing system. First, deploying diversification algorithms at retrieval requires indexing the variety dimension of Pins (e.g. the Pin pores and skin tone vary) in each embedding-based and token-based indices. Particulars about our method will be discovered within the paper. Second is affect on latency and scaling. For RR we discovered it had a minimal affect on latency as a result of linear time complexity but it surely was arduous to scale when utilizing a number of dimensions. For DPP, we minimal impact on latency via numerous strategies (for instance tuning the batch measurement, window measurement, and depth measurement), all of which will be optimized and evaluated via offline replay, shadow testing, or A/B experiments for every floor. Further strategies to cut back the affect on latency for DPP will be discovered within the paper. Third, to judge the diversification of outcomes utilizing pores and skin tone, we collected qualitative suggestions from a various set of inner members for each iteration, along with relevance evaluations via skilled knowledge labeling. To account for the native context in worldwide markets, we collaborated carefully with the internationalization workforce for a qualitative evaluation of diversification and its outcomes.

To enhance pores and skin tone illustration, we launched pores and skin tone diversification in Search, Associated Merchandise, and New Person Homefeed. For search, diversification was launched for queries within the magnificence and style classes. For Associated Merchandise, it was added for style and marriage ceremony requests and in New Person Homefeed as a part of the brand new person expertise. There are a number of nuances that have to be considered when measuring the success and implications of those approaches in search and recommender programs. First, applicable metrics and guardrails have to be set in place earlier than performing diversification. Second, whereas a few of the learnings are transferable between surfaces, every floor presents distinctive challenges and should differ drastically from previous use instances. We regularly noticed optimistic features in variety metrics coupled with impartial or optimistic affect in guardrail enterprise metrics for all of the strategies described above. All metrics reported listed here are the results of a number of A/B experiments we ran in manufacturing for not less than three weeks, and Desk 1 offers a quick overview of the affect of those.

In the remainder of this part, we give a quick overview of the affect of those strategies on person engagement metrics and the variety metric (DIV@ok(R)) (we offer extra particulars on the selection of ok within the paper). We report the affect to those metrics as the proportion distinction relative to regulate.

Surface Skin tone diversification technique Percentage improvement skin tone diversity Search RR with score threshold 250%* DPP Parity* Strong-OR in retrieval + DPP 14%* Related Products RR 270%** DPP Small decrease* Bucketized ANN Retrieval 1% ** (8% increase at the retrieval stage) New User Homefeed Two dimensional RR using priority queues with Pin category and skintone 109%** Single skin tone based RR 650%** Overfetch-and-Rerank during retrieval 63%** DPP for reranking 462%**
Desk 1: On this desk we summarize the outcomes of a number of A/B experiments in manufacturing that enhance pores and skin tone diversification in Search, Associated Merchandise and New Person Homefeed both post-ranking or at retrieval. * signifies optimistic affect to engagement metrics, and ** signifies impartial affect to engagement metrics. Extra particulars will be discovered within the paper.
Pins of painted pink nails going from less diverse to more diverse
Fig. 6. For the question “pink nails matte” on Search, (a) exhibits search outcomes with none variety, (b) exhibits diversified search outcomes utilizing RR with a rating threshold, and (c)exhibits the diversified rating for a similar question utilizing DPP.

We tackled the problem of diversification to enhance illustration in Search and recommender programs utilizing scalable diversification approaches at rating and retrieval. We deployed multi-stage diversification on a number of Pinterest surfaces and thru in depth empirical proof confirmed that it’s attainable to create an inclusive product expertise that positively impacts enterprise metrics reminiscent of engagement. Our strategies are scalable for a number of simultaneous variety dimensions and might assist intersectionality. Whereas these approaches had been profitable we goal to maintain bettering upon them. Future work contains however shouldn’t be restricted to:

  • Creating extra superior and scalable triggering mechanisms for diversification
  • Automating weight adjustment for the multi-objective optimization weights that stability completely different targets
  • Testing some latest developments in debiasing phrase embeddings and honest illustration studying for retrieval diversification
  • Analyzing how diversified search outcomes and proposals can assist mitigate serving bias in programs that generate their very own coaching knowledge

Pores and skin tone diversification goals at bettering illustration by surfacing all pores and skin tone ranges within the prime outcomes when attainable. Whereas the seen pores and skin tone ranges in Pin photos are leveraged to floor all pores and skin tone ranges within the prime outcomes at serving time, they aren’t used as inputs to coach ML rating fashions. It is very important be aware that pores and skin tone ranges are Pin options, not person options. We respect the person’s privateness and don’t try to predict the person’s private info, reminiscent of their ethnicity.

This endeavor wouldn’t have been attainable with out a number of rounds of debate and iterations with our colleagues Vinod Bakthavachalam, Somnath Banerjee, Kevin Bannerman-Hutchful, Josh Beal, Larkin Brown, Hayder Casey, Yaron Greif, Will Hamlin, Edmarc Hedrick, Felicia Heng, Dmitry Kislyuk, Anna Kiyantseva, Tim Koh, Helene Labriet-Gross, Van Lam, Weiran Li, Daniel Liu, Dan Lurie, Jason Madeano, Rohan Mahadev, Nidhi Mastey, Candice Morgan, AJ Oxendine, Monica Pangilinan, Susanna Park, Rajat Raina, Chuck Rosenberg, Marta Scotto, Altay Sendil, Julia Starostenko, Kurchi Subhra Hazra, Eric Sung, Annie Ta, Abhishek Tayal, Yuting Wang, Dylan Wang, Jiajing Xu, David Xue, Saadia Kaffo Yaya, Duo Zhang, Liang Zhang, and Ruimin Zhu. We want to thank them for his or her assist and contributions alongside the way in which.

For extra particulars on the approaches introduced on this article please refer to our paper printed at FAccT 2023.

To be taught extra about engineering at Pinterest, try the remainder of our Engineering Weblog and go to our Pinterest Labs web site. To discover life at Pinterest, go to our Careers web page.