Incident tracking in Slack

ChatOps illustration

Problem

Let’s say your team has to serve a lot of ad-hoc requests from other teams in your company. Typically, if a company has already adopted Slack, these requests are coming via dedicated Slack channel, e.g.:

  • #people
    — a channel for all HR questions
  • #dev-ops
     — all dev infra requests go here
  • #helpdesk
     — broken laptops, VPN issues, etc
  • #sales-ops
     — missing CRM records, not working integrations
  • #any-other-ops

We’ll take a dev-infrastructure team as an example with an

#infra-ops
channel. This channel is used for the public announcements and discussions, but mostly it’s used by the infra team clients. They all come to the
#infra-ops
to resolve their day-to-day issues.

Every time someone has a problem with the infrastructure, they post it on the

#infra-ops
, and someone from the infra team has to respond to it. A typical channel log looks somewhat like this:

Requests in Slack A typical Slack channel where people report a lot of issues

It’s a very convenient model for infra-team clients because they don’t have to file tickets in external systems, or even put their thoughts in order: just dump a problem to the infra channel, and someone will figure out the rest. It is also pretty easy for the infra-team members to react to these incidents right in Slack: threads, emojis, mentions — all work great for quick responses and initial triaging. However, as the traffic on the channel grows, and issues start coming in dozens per day, the lack of order is starting to hurt:

  • People on the infra-team start forgetting who works on what
  • It’s not clear which items are in progress and which await attention
  • Duplicate reports and similar questions start clogging the channel
  • There’s no visibility into how much time the team spends working on ad-hoc issues vs. planned work
  • It’s not clear which areas generate the most number of problems and demand extra attention
  • There are no performance metrics at all

Someone would say it’s simple: let’s use a tracking system like JIRA Helpdesk or Zendesk. It has tickets, owners, labels, reports, knowledge base, etc., and it would solve all our problems!… if only we could force people to create tickets, instead of asking on Slack.
The reality is that going to the #infra-ops is utmost the easiest and the most efficient way of getting help for people, so they will just keep doing that. Trying to change this behavior is like pushing a giant rock uphill. Is there a better solution? Why can’t you turn your Slack into a full-fledged incident tracking system? Below you will see how you can combine Slack and OneBar to achieve exactly that!

Solution

High-level idea

We will continue using the

#infra-ops
channel as we were, but now someone is going to sync it to OneBar periodically. This way we won’t change anything for the infra clients: they will still be sending their issues directly to the Slack channel, and infra staff will keep responding in place if they will. At the same time, in OneBar we will be accumulating a structured log of all the events that happened on the #infra-ops, which we can later analyze, turn parts of it into Knowledge Base articles, assign individual issues to the team members and more.

Data flow High level overview of the OneBar & Slack integration

OneBar Slack import

The primary tool we’re going to be using here is the OneBar “import from Slack” interface. There are two ways how you can get to it:

  • In your Slack channel run
    /onebar
    save command or just say
    @onebar
    save in the thread
  • From the Web App go to the “Slack” tab, find your channel in the list and click on it

OneBar import interface solves a big problem for you that a typical Slack-actions based integration can not: you can create tickets out of arbitrarily structured conversations, be that a single message, multiple messages, a thread or even several threads. Right from the import interface, you can:

  • Create tickets (we call them Problems)
  • Merge duplicate reports
  • Reply with KB articles (similar solved Problems)
  • Assign people to unresolved Problems
  • Attach Tags
  • See on the minimap what’s already in OneBar and what’s pending
Import Dialog OneBar import dialog

Tags

Tags are multifunctional, but mainly they help you to group related information together. For our infra-ops use case, we could use the following tags:

  • infrastructure
    — to group together all the infra related Problems
  • incidents
     — for labeling all incidents
  • infra-faq
     — for labeling KB articles, HOWTOs, FAQs, etc
  • jenkins
    ,
    aws
    ,
    hardware
    , etc. — for labeling different infra sub-a
  • reas
  • p0
    ,
    p1
    ,
    p2
    — for tracking the severity

It’s important to attach tags, because later on when viewing the reports, they will help you slice and dice information in any way you want.

Import animation Working with OneBar import interface

Ok, it’s in OneBar, now what?

After you’ve moved your channel history into OneBar, you can start doing things that you couldn’t do in Slack.

View Problems filtered by any combination of tags, statuses, time periods, people, etc.

Tagged problems Solved Problems tagged #infrastructure

Analyze aggregated reports to detect trends and anomalies

Analytics A Created vs Solved graph for the #infra-oncall tag

Assign unresolved problems to your team members, and track the backlog

Analytics Assigning people to an unresolved Problem

Merge repetitive issues and see the occurrence history

Analytics Duplicate manager

You gain a lot of visibility into what’s going on in the channel, and what your team is working on. You are also automatically building up a Knowledge Base of typical solutions (in our example, it’s everything that’s tagged

infra-faq
), and you can start directing people there for self-service.

You can even configure the

@onebar
bot to automatically reply in the
#infra-ops
channel using the information tagged with the
infra-faq
tag.

Settings animation Configuring the @onebar bot verbosity

Another little trick is to put “

<your OneBar workspace>/tags/infra-faq
” in the title of your channel and tell everyone that it’s now an official channel F.A.Q. Even though it’s one step away from Slack, it can hold much more information than a “Pinned items” section.

Pinned OneBar link Title of the #infra-ops channel

Give it a try!

Slack logo color Add to Slack