A Tail of Two Task Trackers

On this little outing I'll be migrating a Trello board to a Storyboard project. The purpose of this effort is to move all cards, columns, and assets from an active Trello board into Storyboard creating new stories, worklists, tasks, and a board.^[1]

Why is this happening?

While Trello is wonderful, it does have some rough edges, especially when it comes to on-boarding new people into a project.

I want folks working on and consuming TripleO to see TripleO-Ansible as an easy place to contribute

Where we ran into issues

I created a Trello board which was intended to foster collaboration across communities but we ran into a wall when it came time to add folks to the board. While I could include a team, I couldn't make the board fully open; available to the public. Folks outside of the TripleO-Ansible team were unable to contribute to the board and during the first TripleO Transformation Squad meeting it was determined that the managerial overhead required to maintain our use of Trello was a non-starter.

Why Storyboard

Storyboard is part of OpenDev and anyone in the OpenDev community can add, modify, or close issues without any additional managerial overhead. With Storyboard being completely open-source and available to the public, it was quickly identified as the ideal tool for our needs; the TripleO-Ansible project. By moving from Trello to Storyboard folks will be better able to contribute to TripleO-Ansible and by extension TripleO. This move will also raise awareness about everything going on in the project, both in and outside of the OpenDev community.

While it should go without saying, I want folks working on and consuming TripleO to see TripleO-Ansible as an easy place to contribute, and Storyboard is just one part of that greater mission.

Getting Started

Before beginning any migration we need to map concepts between the two tools.

Terminology Breakdown

Thankfully the mapping of terminologies is quite simple between Trello and Storyboard.

Dealing with checklist and task state

Checklists and tasks, while quite different, provide for much of the same purposes. In this case, the state properties are different. Storyboard has many more state options than Trello, however, the map is quite simple given the limited optionality found within Trello.

Using the REST API

With the concepts well understood, it's time to dive into the REST API's for both services. The first thing we'll do is collect all of the information required from Trello, then we'll assemble the data into a Storyboard compatible format, and finally transfer the content. For this operation everything will be executed via python. If you'd like to just plugin information into a trashy script I've made one, which can be seen here^[2].

In our python interpreter, in my case ipython, we need to import a couple modules, which will make this migration process simple.

If the python environment does not have requests, you will need to install it.

import json
import urllib.parse as urlparse
import requests

Using the Trello API^[3]

There is one piece of information we'll need to pull all of the assets from the active Trello board, the TRELLOBOARD_ID. This ID can be seen in the board URL.

The Trello board that will be migrated is tEjJlT3k.

Armed with the board ID, store it as an object.

TRELLO_BOARD_ID = 'tEjJlT3k'

We'll now go grab the board details and store them in an object.

# Trello Constants
TRELLO_BOARD_JSON = requests.get(
    'https://api.trello.com/1/boards/{}/lists'.format(TRELLO_BOARD_ID)
).json()

You may need to extend the trello URL with your API and token keys: TRELLO_BOARD_JSON = requests.get('https://api.trello.com/1/boards/{}/lists?key={}&token={}'.format(TRELLO_BOARD_ID, TRELLO_KEY, TRELLO_TOKEN)).json()

With the board details stored, we'll set the list IDs and create a map of list names to an IDs.

TRELLO_COLUMN_IDS = set([i['id'] for i in TRELLO_BOARD_JSON])
TRELLO_COLUMNS_HASH = dict(
    [
        (
            i,
            requests.get('https://api.trello.com/1/lists/{}'.format(i)).json()['name']
        )
        for i in TRELLO_COLUMN_IDS
    ]
)

You may need to extend the trello URL with your API and token keys: requests.get('https://api.trello.com/1/lists/{}?key={}&token={}'.format(i, TRELLO_KEY, TRELLO_TOKEN)).json()['name']

Once all of the list names are known, we'll map all of the cards to a given list ID.

TRELLO_CARD_HASH = dict(
    [
        (
            i,
            requests.get('https://api.trello.com/1/lists/{}/cards'.format(i)).json()
        )
        for i in TRELLO_COLUMN_IDS
    ]
)

You may need to extend the trello URL with your API and token keys: requests.get('https://api.trello.com/1/lists/{}/cards?key={}&token={}'.format(i, TRELLO_KEY, TRELLO_TOKEN)).json()

Finally we'll create a map of Trello checklist state options to Storyboard status options.

TRELLO_TO_STORYBOARD_STATUS_MAP = {
    'incomplete': 'todo',
    'complete': 'merged'
}

Reviewing the stored data

With these REST API calls all wrapped up, it's time to review what we have. At this point in time we have several Trello constants which have everything we need from Trello to run a successful migration. The two interesting constants are TRELLO_COLUMNS_HASH and TRELLO_CARD_HASH which will be used later when its time to import data into Storyboard.

Using the Storyboard API^[4]

While the Storyboard API is public, an authentication token is required to create content. A Token can be generated using the Authentication Tokens panel.

With the token generated, store it as an object.

STORYBOARD_ACCESS_TOKEN = "TOKEN_STRING"

Now store the API URL we'll be using for this migration.

STORYBOARD_API_URL = "https://storyboard.openstack.org/api/v1"

Set the name for the board and project we'll be migrating to.

STORYBOARD_WORKBOARD = "TripleO-Ansible Transformation"
STORYBOARD_PROJECT_NAME = 'openstack/tripleo-ansible'

With all of our reqired information set, create a session object which will assist with client authentication.

STORYBOARD_CLIENT = requests.Session()
STORYBOARD_CLIENT.headers = {
    'Authorization': 'Bearer {}'.format(STORYBOARD_ACCESS_TOKEN),
    'Content-Type': 'application/json;charset=UTF-8'
}

Now set the project information as an object. The following request will get all projects from Storyboard and match on the defined STORYBOARD_PROJECT_NAME.

The following function makes the assumption that the defined project name has already been configured for use with Storyboard as documnented by OpenDev Infra.^[5]

STORYBOARD_PROJECTS = STORYBOARD_CLIENT.get(url=STORYBOARD_API_URL + '/projects').json()
STORYBOARD_PROJECT = [
    i for i in STORYBOARD_PROJECTS
    if i['name'] == STORYBOARD_PROJECT_NAME
][0]

The last thing we need is an empty dictionary which will be used to map our Storyboard story IDs to Trello card IDs.

STORYBOARD_TASK_HASH = dict()

Running the migration

At this point all of the required information has been gathered, and the Storyboard API is ready to be used. All we need to do now is to begin looping through the Trello Content and posting to Storyboard.

The following functions are ugly. They've been written for easy of understanding not efficiency.

Here's the story creation overview

1. loop though all of the lists of cards and create stories from the discovered Trello data.
2. If a given card is using a checklist, loop through all of the checklist IDs, returning data back from the Trello API about a given item.
3. Create a task from the checklist item details.

# loop though all of the *lists* of *cards* and create *stories* from the
# discovered Trello data.
for cards in TRELLO_CARD_HASH.values():
    for card in cards:
        story = STORYBOARD_CLIENT.post(
            url=STORYBOARD_API_URL + '/stories',
            data=json.dumps(
                {
                    "title": card['name'],
                    "description": card['desc']
                }
            )
        )
        _story = story.json()
        print('Story ID "{}" has been created'.format(_story['id']))
        STORYBOARD_TASK_HASH[_story['id']] = card['idList']
        # If a given *card* is using a *checklist*, loop through all of the
        # *checklist* IDs, returning data back from the Trello API about a
        # given item.
        for checklist in card['idChecklists']:
            print('Importing trello checklist "{}" into story'.format(checklist))
            for item in requests.get('https://api.trello.com/1/checklists/{}'.format(checklist)).json()['checkItems']:
                # Create a *task* from the *checklist* item details.
                STORYBOARD_CLIENT.post(
                    url=STORYBOARD_API_URL + '/stories/{}/tasks'.format(_story['id']),
                    data=json.dumps(
                        {
                            "title": item['name'],
                            "project_id": STORYBOARD_PROJECT['id'],
                            "key": TRELLO_TO_STORYBOARD_STATUS_MAP[item['state']]
                        }
                    )
                )
        print('imported story - "{}"'.format(card['name']))

You may need to extend the trello URL with your API and token keys: requests.get('https://api.trello.com/1/checklists/{}?key={}&token={}'.format(checklist, TRELLO_KEY, TRELLO_TOKEN)).json()['checkItems']

After the stories are all created, it's time to import Trello lists as Storyboard worklists and assign all of the created stories to their respective worklist.

Here's the worklist creation overview

1. Loop over all of the Trello lists.
2. Create a Storyboard worklist using the list name from Trello.
3. Loop over the Storyboard story ID to Trello list ID mapping.
4. Add all stories to the corresponding Storyboard worklist based on its original placement using the the Trello list information.

#  Loop over all of the Trello *lists*.
STORYBOARD_WORKLIST_IDS = list()
for trello_k, trello_v in TRELLO_COLUMNS_HASH.items():
    # Create a Storyboard *worklist* using the *list* name from Trello.
    worklist = STORYBOARD_CLIENT.post(
        url=STORYBOARD_API_URL + '/worklists',
        data=json.dumps(
            {
                "title": trello_v,
                "automatic": False
            }
        )
    ).json()

    STORYBOARD_WORKLIST_IDS.append(worklist['id'])
    print('created worklist "{}"'.format(worklist['id']))
    # Loop over the Storyboard *story* ID to Trello *list* ID mapping.
    for storyboard_k, storyboard_v in STORYBOARD_TASK_HASH.items():
        # Add all *stories* to the corresponding Storyboard *worklist* based on its
        # original placement using the the Trello *card* and *list* information.
        if trello_k == storyboard_v:
            STORYBOARD_CLIENT.post(
                STORYBOARD_API_URL + '/worklists/{}/items'.format(worklist['id']),
                data=json.dumps(
                    {
                        'item_id': storyboard_k,
                        'item_type': "story",
                        'list_position': 0
                    }
                )
            )
            print('Story "{}" assigned to worklist {}'.format(storyboard_k, worklist['id']))

The last and final step in the migration process is to create a new board and associate the newly created worklists to it.

board = STORYBOARD_CLIENT.post(
    STORYBOARD_API_URL + '/boards',
    data=json.dumps(
        {
            "title": STORYBOARD_WORKBOARD,
            "lanes": [
                dict(
                    list_id=v,
                    position=k
                )
                for k, v in enumerate(STORYBOARD_WORKLIST_IDS)
            ]
        }
    )
)
_board = board.json()
print('Story board created. This board can be found here: "https://storyboard.openstack.org/#!/board/{}"'.format(_board['id']))

Post Migration Review

Now that the migration has been completed, it's time to review everything we've done in the Storyboard UI (the place where most folks are going to be working).

As you can see the board within Storyboard is active and looks great.

This is the old Trello board for comparison. As you can see, the layouts are very similar so developers should be able to work from Storyboard without skipping a beat.

Shutting Down Trello

With the Trello board migrated it's time to lock-down the old Trello board and start working from Storyboard. Before closing the board I renamed it so folks see where to go for work from this point forward.

All Trello board operations can be easily handled in the Trello UI.

Why this is awesome

One of the greatest features of Storyboard is its hooks into Gerrit, the OpenDev review system. Now as developers are working on roles and services within TripleO-Ansible folks can track their work using commit message tags which will automatically update the Storyboard tasks and stories.

The openness of Storyboard and it's built-in developer workflow is something to get excited about, and I'm really looking forward to collaborating with folks using our new task tracker.

That's all folks

I hope you've enjoyed this post covering the magical adventure of migrating from Trello to Storyboard. I hope this post highlights some of the amazing capabilities available to the OpenDev communities and shows what's possible to new folks looking to get involved.

If this is the first time you're reading about Storyboard and are interested in all of the things it can do, or wanting to help make it even better, check it out. It's a very cool project with an awesome community who's always looking for new contributors.