May 18, 2024

At Slack, we’ve lengthy been conservative technologists. In different phrases, after we put money into leveraging a brand new class of infrastructure, we do it rigorously. We’ve accomplished this since we debuted machine learning-powered options in 2016, and we’ve developed a sturdy course of and expert workforce within the area.

Regardless of that, over the previous 12 months we’ve been blown away by the rise in functionality of commercially out there massive language fashions (LLMs) — and extra importantly, the distinction they might make for our customers’ greatest ache factors. An excessive amount of to learn? Too laborious to search out stuff? Not anymore — 90% of users who adopted AI reported the next stage of productiveness than those that didn’t.

However as with all new know-how, our means to launch a product with AI is based on discovering an implementation that meets Slack’s rigorous requirements for buyer information stewardship. So we got down to construct not simply superior AI options, however superior and trusted AI.

The generative mannequin trade is kind of younger; it’s nonetheless largely research-focused, and never enterprise-customer targeted. There have been few present enterprise-grade safety and privateness patterns for us to leverage when constructing out the brand new Slack AI structure.

As a substitute, to tell how we constructed out Slack AI, we began from first ideas. We started with our necessities: upholding our present safety and compliance choices, in addition to our privacy principles like “Buyer Information is sacrosanct.” Then, by means of the precise lens of generative AI, our workforce created a brand new set of Slack AI ideas to information us.

  • Buyer information by no means leaves Slack.
  • We don’t practice massive language fashions (LLMs) on buyer information.
  • Slack AI solely operates on the info that the person can already see.
  • Slack AI upholds all of Slack’s enterprise-grade safety and compliance necessities.

These ideas made designing our structure clearer, though generally more difficult. We’ll stroll by means of how every of those knowledgeable what Slack AI appears like right this moment.

Buyer information by no means leaves Slack

The primary, and maybe most essential, determination we confronted was how to make sure that we may use a top-tier foundational mannequin whereas by no means permitting buyer information to depart Slack-controlled VPCs. Within the generative mannequin trade, most clients of foundational fashions have been calling the hosted companies straight, and various choices have been scarce.

We knew this strategy wouldn’t work for us. Slack, and our clients, have excessive expectations round data ownership. Particularly, Slack is FedRAMP High authorized, which confers particular compliance necessities, together with not sending buyer information exterior of our belief boundary. We needed to make sure our information didn’t depart our AWS Digital Personal Cloud (VPC) in order that we may assure that third events wouldn’t have the flexibility to retain it or practice on it.

So we started to search for inventive options the place we may host a foundational mannequin on our personal infrastructure. Nonetheless, most foundational fashions are closed-source: Their fashions are their secret sauce, they usually don’t like at hand them to clients to deploy on their very own {hardware}.

Thankfully, AWS has an providing the place it may be the trusted dealer between foundational mannequin supplier and buyer: AWS SageMaker. Through the use of SageMaker, we’re capable of host and deploy closed-source massive language fashions (LLMs) in an escrow VPC, permitting us to manage the lifecycle of our clients’ information and make sure the mannequin supplier has no entry to Slack’s clients’ information. For extra on how Slack is utilizing SageMaker, check out this post on the AWS blog.

And there we had it: We had entry to a prime tier foundational mannequin, hosted in our personal AWS VPC, giving us assurances on our buyer information.

Slack AI architecture diagram

We don’t practice massive language fashions (LLMs) on buyer information

The following determination was additionally key: We selected to make use of off-the-shelf fashions as an alternative of coaching or fine-tuning fashions. We’ve had privacy principles in place since we started using extra conventional machine studying (ML) fashions in Slack, like those that rank search outcomes. Amongst these ideas are that information is not going to leak throughout workspaces, and that we provide clients a alternative round these practices; we felt that, with the present, younger state of this trade and know-how, we couldn’t make sturdy sufficient ensures on these practices if we skilled a generative AI mannequin utilizing Slack’s clients’ information.

So we made the selection to make use of off-the-shelf fashions in a stateless method by using Retrieval Augmented Era (RAG). With RAG, you embody all the context wanted to carry out a job inside every request, so the mannequin doesn’t retain any of that information. For instance, when summarizing a channel, we’ll ship the LLM a immediate containing the messages to be summarized, together with directions for the way to take action. The statelessness of RAG is a large privateness profit, but it surely’s a product profit as nicely. All of Slack AI’s outcomes are grounded in your organization’s data base — not the general public Web – which makes the outcomes extra related and correct. You get the advantage of incorporating your proprietary and particular person information set with out the chance of a mannequin retaining that information.

Utilizing RAG can slender down the set of fashions you should use; they should have “context home windows” massive sufficient so that you can cross in all the info you wish to use in your job. Moreover, the extra context you ship an LLM, the slower your request can be, because the mannequin must course of extra information. As you possibly can think about, the duty of summarizing all messages in a channel can contain fairly a bit of knowledge.

This posed a problem for us: Discover a top-tier mannequin with a big context window with pretty low latency. We evaluated quite a lot of fashions and located one which suited our first use circumstances, summarization and search, nicely. There was room for enchancment, although, and we started an extended journey of each immediate tuning and chaining extra conventional ML fashions with the generative fashions to enhance the outcomes.

RAG is getting simpler and sooner with every iteration of fashions: Context home windows are rising, as is the fashions’ means to synthesize information throughout a big context window. We’re assured that this strategy can get us each the standard we’re aiming for whereas serving to guarantee our clients’ information is protected.

Slack AI solely operates on the info that the person can already see

It’s one among our core tenets that Slack AI can solely see the identical information that the requesting person can see. Slack AI’s search function, for instance, won’t ever floor any outcomes to the person that commonplace search wouldn’t. Summaries won’t ever summarize content material that the person couldn’t in any other case see whereas studying channels.

We guarantee this by utilizing the requesting person’s Entry Management Record (ACLs) when fetching the info to summarize or search and by leveraging our present libraries that fetch the info to show in channel or on the search outcomes web page.

This wasn’t laborious to do, technically talking, but it surely wanted to be an specific alternative; one of the best ways to ensure this was to construct on prime of, and reuse, Slack’s core function units whereas including some AI magic on the finish.

It’s price noting, too, that solely the person who invokes Slack AI can see the AI-generated output. This builds confidence that Slack is your trusted AI companion: Solely the info you could see goes in, after which solely you possibly can see the output.

Slack AI upholds all of Slack’s enterprise-grade safety and compliance necessities

There’s no Slack AI with out Slack, so we ensured that we built-in all of our enterprise grade compliance and safety choices. We comply with the precept of least information: We retailer solely the info wanted to finish the duty, and just for the length crucial.

Generally the least information is: None. The place attainable, Slack AI’s outputs are ephemeral: Dialog summaries and search solutions all generate point-in-time responses that aren’t saved on disk.

The place that’s not attainable, we reused as a lot of Slack’s present compliance infrastructure as attainable, and constructed new help the place we needed to. Lots of our compliance choices come inbuilt with our present infrastructure, equivalent to Encryption Key Administration and Worldwide Information Residency. For others, we inbuilt particular help to guarantee that derived content material, like summaries, are conscious of the messages that went into them; for instance, if a message is tombstoned due to Information Loss Safety (DLP), any summaries derived from that message are invalidated. This makes DLP and different administrative controls highly effective with Slack AI: The place these controls have been already energetic on Slack’s message content material, they’re additionally energetic Slack AI outputs.


Whew — that was an extended journey! And I didn’t even get to take you thru how we construct prompts, consider fashions, or deal with spiky demand; we’ll save that for subsequent time. However I’m glad we began right here, with safety and privateness: We wish our clients to know the way severely we take defending their information, and the way we’re safeguarding it every step of the best way.

 

Thinking about serving to us construct Slack’s AI capabilities? We’re hiring! Apply now