May 18, 2024

The Knowledge Engineering group is liable for Slack’s information lake, analytics dashboards, and different information companies. The group’s mission is to empower customers to leverage information to make selections shortly, precisely, and simply. Slack’s information lake grew in measurement from sub-petabyte to over 100 petabytes lately and it now spans hundreds of thousands of tables. Because the complexity of managing this information grew, so did a various group of Slack engineers devoted to supporting the ecosystem. 

We’ve got sturdy feminine illustration amongst engineers in Knowledge at Slack. Our Knowledge Engineering tradition celebrates variety in views and experiences. As information complexity intensifies, having a mosaic of inventive drawback solvers from totally different backgrounds is the important thing to navigating intricate challenges with agility and perception.

Let’s dive into the non-public tales of ladies who’re redefining Slack’s information panorama.

  • Hearken to Jessica, a latest rent, as she navigates the complexities of Pinot and Tableau expertise.
  • Observe Nilanjana, Ramya, Shrushti, and Nathalie, seasoned engineers, as they lead large-scale engineering tasks utilizing applied sciences like Spark, Merlin, DataHub, and Secor.
  • Lastly, hear from Suzanna, Lakshmi, and Beate, feminine engineering leaders, as they make clear the panorama of development and alternatives for ladies in information engineering. 

Hello, I’m Jessica Stewart, and I’m a Senior Software program Engineer on the Knowledge Orchestration group. I joined in Might 2023.

My group oversees the interior implementations of information workflows utilizing Apache Airflow and Apache Pinot. Working inside our main cluster, which shops terabytes of information, we preserve a system that boasts sub-second question latency and an almost 99.95% question success price service-level settlement (SLA). This datastore serves us internally by supporting instruments for workers and externally because the spine for Slack’s user-facing analytics dashboards.

A present technical problem we face revolves round migrating from a digital machine setup to a cloud-native Kubernetes infrastructure. Digital machines imply greater infrastructure prices and upkeep overhead, so we have been keen to extend effectivity with a containerized set-up. We’ve got an inside Slack Kubernetes platform that doesn’t totally assist Helm charts or Kubernetes companies, so we needed to make some customizations to the open-source answer. With a purpose to preserve efficiency necessities, we configured a buyer networking set-up that integrates Pinot with our inside Kubernetes platform. Moreover, working with Pinot permits us to delve into each software program and infrastructure layers. We’ve developed customized Python tooling that wraps round Pinot to standardize operations, and we’ve streamlined information ingestion by way of Airflow pipelines. On the infrastructure facet, we’ve automated deployments and upkeep duties utilizing Ansible, Kubernetes, and Jenkins.

When interviewing with Slack, my potential co-workers have been supportive and type, and this continues to be the case as we work on tasks, deal with incidents, and plan future work. I’ve the chance to personal initiatives, work cross-functionally and broaden my talent set — all whereas working with co-workers I belief and luxuriate in.

Hello, I’m Nilanjana Mukherjee, and I’m a Workers Software program Engineer on the Metrics Foundations group. I joined in October 2021.

My group is liable for producing actionable datasets and metrics for data-driven decision-making. We additionally preserve excessive information accuracy and desired touchdown time SLAs. 

Becoming a member of the Knowledge Engineering group two years in the past has been an unbelievable expertise with alternatives and challenges. Amidst my involvement in lots of information modeling tasks, I led a pivotal strategic initiative: a migration of workloads from Hive to Spark 3. Being one of many earliest adopters of Spark at Slack made it tough however rewarding. I turned the Spark subject material professional to over 40 groups throughout Slack.

The journey was difficult however considerably contributed to my private {and professional} growth. I owe a lot of my success to the assets and unwavering assist from my managers and group members. Lately, I used to be thrilled to be promoted from Senior Engineer to Workers Engineer a recognition of my development, and a testomony to the alternatives for development at Slack in information engineering.

Hello, I’m Ramya Sundaresan, and I’m a Senior Software program Engineer on the Metrics Foundations group. I joined in Might 2022.

I’ve had the chance to collaborate with two information groups throughout my tenure at Slack: the Knowledge Ingestion group and the Metrics Foundations group. On the Knowledge Ingestion group, we have been entrusted with the essential job of accumulating, processing, and ingesting information into our information warehouse from quite a lot of sources. This included extracting information from Slack’s Vitess utility database, managing custom-made log pipelines, and integrating information retrieval pipelines from Google Sheets. Throughout the Metrics Foundations group, I discovered frequent floor with Nilanjana as we centered on comparable areas of curiosity and experience.

My contributions to Slack’s information engineering ecosystem embody implementing Apache Iceberg inside Kafka Join clusters, orchestrating the migration of Airflow to Kubernetes from AWS EC2 (Amazon Net Providers Elastic Compute Cloud) cases, and migrating jobs to Spark3 on AWS EMR6 (Elastic MapReduce) clusters. We closely leverage AWS. Transitioning from Spark 2 on AWS EMR 5 to Spark 3 upgrades on AWS EMR 6 was a strategic endeavor. It was in keeping with a company-wide goal to ascertain a unified tech stack and diminish reliance on legacy techniques like Hive and MapReduce. I led a number of groups by way of automated workflows and complete documentation leading to a assured and profitable migration. We achieved our migration purpose in underneath a yr utilizing parallel construct pipelines and rigorous testing. This unlocked higher efficiency and fortified our techniques with improved compatibility, interoperability, and long-term assist.

My journey as a software program engineer at Slack has given me alternatives to steer, innovate, and contribute meaningfully to my group’s targets. Our on-call rotation and incident administration prioritizes group member well-being with group and automation. My work offers me lots of satisfaction, however I additionally have to acknowledge a pleasant fixed in my life — my toddler son, who usually contains the colourful Slack emblem each time he doodles, and jogs my memory of the fun past code.

My toddler’s drawing of Mario and Luigi holding the Slack emblem as an alternative of stars

Hello, I’m Shrushti Patel, and I’m a Senior Software program Engineer on the Knowledge Infrastructure group. I joined in August 2020.

My group owns all of the infrastructure and companies required to ship dependable and well timed information. Being a part of this group, I’ve labored on companies like AWS EMR, Airflow, Trino, Secor, and Ranger to handle infrastructure.

Once I joined, Slack had an information ingestion setup reliant on Secor for transferring Kafka information to S3 hosted on EC2. This was inflicting points once we tried to include new information subjects. My group decided that Secor fell brief when it comes to industry-standard recognition, ongoing growth, and assist for rising information codecs. I assumed duty for this Secor setup, and I spearheaded its migration to Bedrock (an internally-developed Kubernetes framework). This transition simplified the addition of recent subjects, harnessed the advantages of Kubernetes, and resulted in value financial savings by way of useful resource optimization.

To additional improve our information infrastructure, we’re migrating from Secor to Kafka Join: a widely-adopted {industry} commonplace for streaming techniques. Kafka Join affords out-of-the-box assist for state administration, fault tolerance, and scaling. In contrast to Secor, Kafka Join operates as an abstraction and leverages connectors (executable JARs) throughout the Kafka-Join ecosystem. This strategic shift is transferring us in direction of real-time streaming capabilities. The work aligns with our dedication to {industry} requirements, ongoing growth, and adaptableness to evolving information codecs. 

My time with Slack’s Knowledge Infrastructure group has been a transformative journey. I’ve delved into mentorship, participating tasks, and a tradition of steady studying. Slack’s dedication to mobility has enriched my development by offering me the chance to work with all of the groups underneath information and cross-organizational groups like Cloud and Observability. Once I returned from maternity depart, the group supported me by offering context about ongoing tasks and onboarding me again into the move. This expertise showcased Slack’s compassion by creating an surroundings the place skilled excellence is nurtured alongside empathy for private milestones.

Hello, I’m Nathalie Kaligirwa, a Senior Software program Engineer on the Metrics Platform & Governance group. My group builds scalable and standardized instruments to reinforce the information expertise. I joined in November 2021.

The previous two years have been eventful. I had a wholesome child, took six months of maternity depart, and labored on complicated tasks with a number of stakeholders and distributed groups. I aimed to stability private development with work commitments by aligning duties with my vitality stage and busy schedule.

Metrics standardization is a significant ongoing initiative at Slack, and it encompasses tasks like Merlin, a framework for abstracting metrics and huge tables creation. Throughout parenthood, I centered on particular parts inside this initiative that may be manageable throughout a time of elevated private calls for. This allowed me to contribute to the bigger imaginative and prescient of the undertaking whereas balancing new dad or mum duties. For instance, I up to date a service to dynamically add new metrics created with Merlin and eradicated the necessity for guide migration. I then switched gears to work on bettering the search expertise. This set the groundwork for a brand new metadata service that gives lineage throughout a number of information instruments and scalable search.

Whether or not I used to be consolidating metrics, studying to vary diapers, integrating information companies, placing collectively a nursery, engaged on metadata companies, or navigating a compromised sleep schedule, I grew professionally and personally. Wanting again, I’m pleased with my work and grateful for my group’s assist.

Suzanna Khatchatrian, Senior Director, Knowledge Engineering. Joined October 2018
Lakshmi Mohan, Director, Knowledge Platform. Joined February 2021
Beate Porst, Group Product Supervisor, Knowledge Engineering. Joined November 2022

As Slack’s Knowledge Engineering management, we’re dedicated to mentorship and offering alternatives to be taught and develop. The group can be dedicated to mobility we encourage inside transfers for skilled development and diversification of abilities. Slack’s steady studying philosophy not solely enriches particular person careers but in addition creates a dynamic and collaborative office. 

We mannequin Slack’s worth of compassion in our assist for group members taking maternity depart. The group is organized to make reintegration after a hiatus a seamless expertise. We genuinely care not nearly our teammate’s skilled efficiency but in addition their private milestones. 

As leaders at Slack, we actively nurture an surroundings the place ladies in information engineering thrive and attain their full potential. We do that by way of:

  • Visibility: We amplify ladies’s voices by encouraging them to tackle management roles and converse at conferences. We have a good time their achievements and showcase them as position fashions by way of participation in {industry} occasions.
  • Mentorship: We provide each formal and casual mentorship applications the place we join skilled ladies leaders with mentees for steering and assist. This fosters confidence, management abilities and a powerful community of friends.
  • Advocacy: We actively problem biases and advocate for honest practices in all features of our work surroundings.
  • Empowerment: We assist versatile work preparations, variety and inclusion initiatives, and entry to coaching assets. We create a welcoming and inclusive area for all.

Embracing variety: our dedication to inclusion

Our private experiences at Slack display the affect of those inclusion efforts. That is the Slack we envisioned: a spot the place concepts are heard, contributions are valued, and particular person journeys are fueled by a collaborative and supportive tradition. Our success is obvious within the rising variety of ladies selecting to hitch our groups and within the legacy of compassion and inclusion we’re proud to domesticate.

All in favour of becoming a member of our Knowledge Engineering group? Apply now