Introduction

NashTech’s Modern Data Solutions Accelerator provides an end-to-end cloud-based data solution with a project skeleton and Command Line Interface (CLI) support for efficient development. The solution is available on two major cloud platforms (AWS and Azure), providing robust support for data ingestion, data Lakehouse (Medalion architecture), DataOps, Data visualization, and is compliant with some data governance policies.

By leveraging Terraform, the template supports the provisioning of environments based on the following cloud services: Azure Synapse, Azure Data Lake Storage Gen 2, Azure Data Factory, Azure Databricks, Azure Event Hub, Streaming analytics, Power BI. It enables DataOps practices through Infrastructure as Code (IaC) with Terraform and Continuous Integration/Continuous Deployment (CI/CD) using Azure DevOps. Leveraging Terraform ensures consistent deployment across multi-cloud environments.

Use cases

It’s beneficial to use this template to quickly provision a self-service analytics platform on the cloud platform (currently supporting Azure Synapse, Databricks and it will soon support Microsoft Fabric) with a ready-to-use code base (in python & T-SQL). This allows you to focus on business data outcomes rather than on developing a data solution from scratch.

Benefits

Our data accelerator provides tangible benefits by markedly reducing the time required for provisioning, developing and deploying data solutions. Built on industry best practices, it helps to shorten your project delivery timelines and ensures the quality of your deliverables. Engineered for scalability, it effortlessly accommodates expanding data volumes and evolving processing demands without sacrificing performance, leveraging Terraform’s infrastructure as code capabilities.

Key features:

Data Ingestion:

• Enables dynamic ingestion from diverse sources including SQL, CSV, and JSON formats.
• Facilitates batch processing via Azure Data Factory, Synapse Integrate, and Databricks.
• Supports real-time data streaming through Event Hubs and Stream Analytics.

Data Lakehouse:

• Constructs a Lakehouse solution leveraging Synapse (serverless & dedicated), or Databricks.
• Compatible with Parquet and Delta Lake formats for enhanced data storage.
• Enhances data analytics capabilities with Spark processing.

DataOps:

• Utilizes Terraform for seamless infrastructure provisioning.
• Establishes CI/CD pipelines using Azure DevOps for automated deployment.
• Elevates code quality and traceability through GitFlow integration for improved DevOps practices.

Data Governance Compliance:

• Monitors database activities to ensure compliance.
• Implements robust data protection and encryption measures.
• Manages access control and enforces data policies to maintain governance standards.

Visualization:

• Empowers data visualization using Power BI.
• Seamlessly integrates with Synapse & Databricks for comprehensive visualization capabilities.
• Supports real-time analytics to provide up-to-date insights.