Details
The core vision of CAI was to create a system that continuously monitors Input Events (input of spreadsheet risk data) and then semi-automates, via workflow processes, the tasks of cleansing, transforming, and generating output documents. The application was first developed on-premise, and then later migrated to the Azure cloud using the App Service Migration tool. Once in production, monitoring and alerting capabilities were added using App Insights, and geo replication was used to provided failover and uptime guarantees.
The high-level conceptual architecture is as follows:
- Front end UI built using modern web technologies such as React and ASP.NET MVC.
- Hosting and services environments built using Microsoft Azure cloud technologies, including Azure AD, Azure Web Apps, and Application Insight to capture telemetry
- Data for the system was hosted and processed using a combination of storage technologies, including Azure Blob storage, Cosmos DB, and Azure Key Vault
- In addition Azure Cognitive Services was used for language Translation, and major use was made of Azure Search for fuzzy logic searches.
- SignalR was used to enable real-time functionality for the system
The high-level system functionality included the following:
Basic Document Flow Management (data and document ingest and output). This was driven by an Azure Logic app that triggered workflows based on the receipt of incoming emails. End-User Application Core Features were extensive, and included:
- Application Sign In to the Application Portal
- User Management
- Client Account Management
- Data Import (Triage Interface)
- Task Management
- Preliminary Analysis of Documents
- Document Cleansing
- Data Transformation
- Geocoding Interface
- Output Generation
- Data Management
- Schema Management
Figure 1 – High-Level Architecture
The core tools used consisted of the following:
- Power BI. Power BI is a suite of business analytics tools that can be used to
analyze data and share insights. Power BI reports can be connected to many different data sources, produce reports, and publish them for an organization to consume on the web and across mobile devices.
- Azure Blob Storage. Azure Blob storage supports storing unstructured data in the cloud and can be used to store any type of text or binary data.
- Azure Cosmos DB. Azure Cosmos DB is a globally distributed, multi-model database service that offers storage across several well-defined consistency levels.
- React. A framework used for creating browser applications.
- ASP.NET MVC. A framework for building web APIs that are used to support the front-end browser application.
- Also: Azure Search, Cognitive Services for Translation, and SignalR.
Azure DevOps for Application Lifecycle Management
To ensure the application could be rapidly built, iterated on, and deployed, Azure DevOps was used for application lifecycle management, automating the build and release cycle. The team implemented continuous integration and continuous delivery processes using Azure DevOps Pipelines, ensuring code could be built, tested, and released on demand. Finally, Azure ARM templates were used as the solution for infrastructure-as-code, ensuring the entire environment could be released to Azure in a fully automated manner.
Fig 2 – Data Selection: Selecting data of interest from ingest worksheets
Fig 3 – Transforming origin data into standard form using transformation formulas
Fig 4 – Capturing geocoding information for an individual property
Fig 5 – Mapping relationships between source columns and target columns
Figure 6 – Decision Tree Builder allows full configuration of processing logic