The cloud and data platforms the rest of this site runs on: AWS, Databricks, Snowflake and Azure, plus the DevOps layer that deploys and orchestrates the work.
Apache Spark is an open-source, distributed computing framework for big data processing and analytics. Designed for speed , ease of use , and scalability…
Snowflake is a cloud-based data warehouse platform that uses a unique multi-cluster, shared data architecture. It separates storage and compute resources…
Microsoft's AI stack on Azure spans three layers: Azure OpenAI Service for direct access to OpenAI models, Azure AI Foundry for multi-model generative AI…
Software delivery refers to the process of developing, testing, and deploying software applications to end-users or production environments. It encompasses…
This section documents the platforms that the data engineering, analytics and AI material elsewhere on the site is built on. The cloud providers, AWS and Azure, supply the raw building blocks: virtual machines and serverless compute, object storage, networking, identity and managed services ranging from databases to foundation-model APIs. Databricks and Snowflake are data platforms that run inside those clouds and add a managed analytics engine, governance and increasingly their own AI features. The DevOps area covers the tooling that turns code into running systems: version control, CI/CD pipelines, containers, Kubernetes and workflow orchestration.
In a typical architecture, raw data lands in cloud object storage (S3 or ADLS), is transformed by Spark jobs on Databricks or SQL pipelines in Snowflake, and is served to dashboards, applications and machine learning models. Both data platforms are available on AWS and Azure, so the cloud choice is often driven by existing contracts, identity systems and the managed AI services a team wants to use, while the data platform choice is driven by workload style and team skills.
| Area | Choose it for | Section |
|---|---|---|
| Amazon Web Services | broadest service catalogue: compute, storage, databases, Bedrock, SageMaker | AWS |
| Microsoft Azure | Azure OpenAI, AI Foundry, Azure ML, Microsoft identity integration | Azure |
| Databricks | Spark, Delta Lake, streaming, Python-first data engineering and ML | Databricks |
| Snowflake | SQL-first warehousing, separation of storage and compute, Cortex AI functions | Snowflake |
| DevOps | CI/CD, Docker, Kubernetes, Airflow, configuration | DevOps |
Databricks and Snowflake overlap more each year: both support SQL warehousing, Python, open table formats and AI features. Databricks remains the more natural fit for engineering-heavy, notebook and Spark workloads; Snowflake for analyst-heavy SQL workloads that value minimal administration. Security guidance for these platforms lives under Security.