The Atmosera DevOps Toolkit is a set of building blocks that can be leveraged to simplify operations involving various tools commonly found in software development environments. These involve Work/Bug tracking systems, Source Control Systems, Build Systems, and others. It can be considered along the lines of being a bunch of Legos™ with many different blocks of various shapes and sizes, along with the ability to create custom blocks that can be built up to perform simple or complex operations on data stored in various tools, including migrations, transformations, archiving, etc.
Typical engagements have unique considerations so the set of building blocks is continually expanded and built upon. This block-based approach has a much higher re-use rate [and therefore the potential for cost amortization] than attempting to build a complete “tool” [see the history section for more information].
CAPABILITIES AND LIMITATIONS
An important consideration in any migration/transformation effort is the ability to accurately represent the data in the new system, this is referred to as the “fidelity” of the effort. Determining the cost and ROI for differing levels of fidelity is a key part of ensuring level expectations on the outcome [and also why trial migrations are so important for empirical validation].
In many cases, the target system simply does not expose APIs or other means to import certain data or limits the quality of the data. A simple example is “Build Runs” in nearly every tool…While it is often possible to migrate the build definition, the history of executions simply does not have a means of being imported. In many cases, it is perfectly acceptable to “leave this behind” [ow fidelity], but in other cases, it may be desirable to create a type of archive of the information.
Even when API(s) exist there may be differences that are “reductive” and can not be accurately mapped. Consider an input with values 1 through 5 as a priority, and a target of only high, medium, and low…. The toolkit can be configured to map these values but resolution will be lost as there are only 3 possible outputs for 5 inputs.
TECHNICAL INTRO
The toolkit uses several C# projects written using the latest .NET 8.0 with many C# 11 and 12 features. Each tool has an adapter assembly, there is a persistence assembly [currently Entity Framework-based backed by SQL Server, along with the core assembly and appropriate harness and testing assemblies.
A key goal of the design is to encapsulate the details of each tool and the mechanics of communications [often ReST APIs] from the higher-level goal-focused custom application. While C# programming skills are needed, significant focus has been on keeping this aspect as simple as possible to reduce the barrier to usage.
The block diagram below provides a high-level view of the typical data flow when used in a migration situation:
HISTORY
Microsoft Team Foundation Server was introduced in 2005. At the time a wide variety of source control systems were in use, including VSS, CVS, SVN, Mercurial, Perforce, and others. Importing from these systems, as well as various sources of work planning and tracking, was a very active area. Microsoft created the “TFS Integration Platform” [TIP] to address this. Unfortunately, the tool had several issues and quickly collapsed on itself.
David Corbin was an independent consultant performing significant work in this area and quickly realized that while something was needed that could amortize development costs [compared to full custom solutions], it was also important to introduce an intermediate generalized/abstracted datastore. Since adapters for each tool interacted with this store, and not with other tools, they could be configured in a manner that facilitated tool->tool endpoints and the introduction of both new capabilities and tool support.
That codebase was used on many engagements over the next (approximately) 13 years. Like many efforts, it accumulated technical debt [quick fixes for a given engagement that were never refactored into stable designs]. It was based on old (often deprecated but still operational) APIs. It was retired (used for the last time in a production capacity) in 2019.
SAMPLE USE-CASES
Migration between tools
Jira Issues -> Azure DevOps
Azure DevOps -> GitHub
GitHub -> Azure DevOps
Import of Other Tools
Transformation within a Tool
Azure DevOps Inter-Organization Operations
Archiving of Historical Information
Source Migration TFVC ->GIT, SVN->GIT
TOOLKIT STATUS
The initial focus was migrating Jira Issues to Azure DevOps Work Items, and some test transformations were performed on internal Atmosera Data. Recently a focus on the splitting of Azure DevOps Organizations [something that was possible using the on-premise server versions, but not supported with the cloud service version] has been being investigated to determine the fidelity and possibilities for each of the data elements. GitHub issues are being monitored, but have not had significant investment.