Alabama based Intergraph (now part of Stockholm-based Hexagon) is a global leader in providing information technology for industrial and geospatial applications to create autonomous connected systems. With over 20,000 employees across the globe, they are a major provider of asset life cycle solutions for the design, construction, and operation of hyperscale industrial facilities such as nuclear submarines and plants.
Intergraph engaged Atmosera to assist in the envisioning and implementation of an Azure Network Infrastructure that would scale to support the unique needs of Intergraph while minimizing Network Administration and Management.
An underlying issue in the Intergraph environment was the potential for significant subscription expansion. Further, the use of Virtual Network Peering (vNet Peering) is non-transitive, so the administration and coordination of traffic increases significantly as more networks are added. Finally, based upon tooling choices and the desire to facilitate large-scale deployments, network limitations such as maximum number of addresses in a Virtual Network needed be considered.
The existing network connection was a single site-to-site Virtual Private Network (VPN) node from the Intergraph headquarters in Huntsville, AL. This connection was to a single transit vNet within the South-Central US Region, which was experiencing resource constraints. Intergraph needed to ensure that teams had connectivity to at least two additional Azure regions to maximize availability, with an ability to fallback to VPNs in the event of an ExpressRoute failure. While the primary purpose of this network was to support non-production workloads, the pattern needed to be repeatable to enable the corresponding regional pairs to be brought online in a similar manner.
While the Huntsville headquarters was the primary focus, Intergraph wanted to ensure that the approaches utilized could be repeated for multiple other facilities around the globe. To that end, the process had to be executed using infrastructure-as-code techniques to ensure that deployments were repeatable and could be managed in the future as code. Appropriate permissions needed to be applied to the deployed environment to ensure that only specific staff could modify the deployed network infrastructure resources.
An additional challenge was the diversity of product groups and technologies used internally, as Intergraph relies on multiple Microsoft and non-Microsoft technologies, all of which required the same level of support from the cloud environment. This meant supporting multiple flavors of Linux (RedHat, CentOS, Debian), Windows Server (2008 R2 – 2016, with plans for 2019), and Windows client (Windows 7 – 10) rehosted from on-premise to Azure. Database servers were also migrated and rehosted, including MySQL, PostgreSQL, Oracle, and SQL Server (2008 R2 – 2016). Atmosera worked with Intergraph’s teams to help them understand how to create and support these environments in Azure, how to network them to existing on-premises environments, and how to optimize these environments to maximize performance and minimize costs.
Atmosera worked with the Intergraph team to consolidate subscriptions, provide connectivity to Azure Locations in three locations around the United States, and provide isolation for individual products while at the same time scaling to Intergraph’s needs.
High Level Deployment Architecture
Atmosera recommended a configuration in which the Corporate IT Subscription would host all ExpressRoute, FortiGate, and Shared Security systems. The default routing topology would force routing through the FortiGate devices so communication could exist between all systems in a single region or in multiple regions. Privileges to configure this environment would be granted using IAM Roles that correspond with Intergraph IT responsibilities.
The second tier of configuration is for Product Group Shared Services. These subscriptions are isolated for “shared services” systems, such as Domain Controllers, Build Servers, and Shared Database Servers and resources. The Virtual Network resource in this subscription is the handoff point between Business Unit networks and the Corporate IT Networks. It was recommended that an IPSEC VPN be leveraged to support communication between this network and the Corporate IT Network. Peering will facilitate communications between the Business Units and the Product Group Virtual Networks.
The final tier of this configuration is the collection of Business Unit Subscriptions for individual Products. Each Product resides in a Business Unit Subscription that consolidates billing and resource management as much as possible while still enabling the flexibility that Intergraph requires.
Logical Network Architecture (U.S.)
Azure ExpressRoute Premium was deployed and peered with three North America Azure regions. A secondary connection to the regions was leveraged from the Huntsville Office using VPNs to ensure direct-connection support in the event of a failure (VPN failover).
The corporate network was based upon a hub/spoke model that allows the corporate environments to be static and controlled only by IT while facilitating connectivity to Product Group networks (PPM, SI, GEO) as well as individual Business Units (the 3D Business Unit of PPM, for example).
Atmosera recommend consolidating site networks as much as possible to within a single CIDR block. A final network was defined with each Azure region allocated a single /16 CIDR block with 65,536 total addresses.
Regional Network Design
Each region was modeled similarly, with subscriptions defining network boundaries. Each Business Unit is an individual subscription, with each Business Unit allocated a vNet in each region required. The Business Unit networks will connect to a Product Group Transit vNet specific to each region that connects to the Corporate IT hub (Figure 2).
Each Business Unit vNet was allocated 2,048 IP addresses in a /21 CIDR block (with up to 2,043 usable addresses). This resulted in 31 individual networks for the business units, and one block dedicated to the Corporate IT Group.
Figure 3 provides a prototype of an example in a single Azure location based upon this design.
Figure 3 – Regional Network Footprint
Intergraph needed a partner that was expert at providing the Azure networking and infrastructure skills needed to take the business requirements and translate that into a technical design that was flexible and performant, as well as the implementation expertise to develop the production system.
Atmosera designed and implemented a hybrid network architecture, leveraging ExpressRoute to connect their on-premises networks to an Azure-based network architecture that maximized performance, provided isolation where required, and enabled flexibility for future growth and adaptability. The site-to-site VPN was redeployed, connecting to Azure through a VPN Gateway to support emergency failover in the event of a catastrophic ExpressRoute provider failure. To ensure optimal performance and minimize maintenance costs, network virtual appliances (NVAs) were utilized for routing and firewall support. This allowed Intergraph’s networking team to continue to utilize existing technologies and skillsets, including sharing configuration data between on-premises FortiGate systems and Azure-hosted appliances.
A hub-and-spoke model was used to ensure that subscriptions and resources could be appropriately isolated in Azure, allowing business units to consolidate resources, grant appropriate permissions, and scale their environments to support their development teams as they build and test their respective software products. This model also enables Intergraph to add additional business units and product groups in the future, ensuring that the company has a repeatable model that supports continued growth.
User-defined routes (UDRs) were deployed in each spoke to ensure that traffic was managed by the FortiGate NVA. Individual subnets were further secured with Network Security Groups (NSGs).
The solution was implemented using Azure Resource Manager (ARM) templates. This created a configurable infrastructure that could be managed and maintained in source control. As Intergraph enhanced additional global data centers with Azure connectivity, these templates could be automatically deployed to create the necessary configurations with minimal training. The Microsoft Cloud Adoption Framework (CAF) was used as the base methodology for the migration in terms of planning and strategy.
Intergraph was able to significantly reduce networking TCO while increasing the flexibility and adaptability of their complex networking infrastructure – leveraging the power of the Azure cloud. The initial system was delivered in 2018, with continual additions and enhancements continuing into 2020 to account for integration with new or different data centers to optimize performance across Intergraph’s global locations.