AWS Analytics

AWS Glue Data CataloganalyticsThe AWS Glue Data Catalog is a centralized, Hive-compatible metadata repository that stores information about… AWS Glue ETL ServiceanalyticsAWS Glue is a fully managed, serverless ETL service that runs Apache Spark (and Python shell) jobs to… AWS Glue WorkflowanalyticsAWS Glue Workflow is the native orchestration layer inside AWS Glue. AWS Lake Formation vs AWS CloudFormationanalyticsHere are the key differences between AWS Lake Formation and AWS CloudFormation: AWS Lake Formation: Detailed StepsanalyticsAWS Lake Formation is a fully managed service that simplifies the process of setting up, securing, and… Amazon AthenaanalyticsAmazon Athena is a serverless, interactive query service that runs standard SQL directly against data in… Amazon QuickSightanalyticsAmazon QuickSight is AWS's serverless, cloud-native business intelligence (BI) service. Amazon Data FirehoseanalyticsAmazon Data Firehose is the serverless delivery service for streaming data. Amazon Redshift ArchitectureSectionAmazon Redshift is a fully managed, massively parallel processing (MPP) cloud data warehouse.

The managed analytics layer on AWS — cataloguing, transforming, querying and visualising data without running the infrastructure underneath it.

‹ AWS

1. Overview

AWS analytics services are designed around a data lake in Amazon S3. Rather than loading everything into one database, data lands in S3 as files (ideally Parquet or another columnar format), a shared catalog describes those files as tables, and several serverless engines read the same tables. This section covers the services that make that pattern work: delivery (Amazon Data Firehose), cataloguing and transformation (AWS Glue Data Catalog, Glue ETL, Glue Workflows), governance (Lake Formation), querying (Athena) and visualisation (QuickSight). One page also clarifies the frequent confusion between Lake Formation and CloudFormation, which share a name pattern but solve unrelated problems.

2. How the services fit together

AWS Analytics Data Flow Sourcesapps, logs, DBs Data Firehosestream delivery Amazon S3data lake Glue Crawler / ETLschema + transform Glue Data Catalogtables Lake Formationpermissions AthenaSQL on S3 QuickSightdashboards Firehose lands data in S3, Glue describes and transforms it, Lake Formation governs access, Athena queries it and QuickSight visualises the results. Glue Workflows orchestrate the ETL steps.

The Glue Data Catalog is the hub: it is a Hive-compatible metastore that Athena, Glue jobs, EMR and Redshift Spectrum all use to find table schemas and partitions. Crawlers or ETL jobs keep it current. Lake Formation sits on top of the catalog and grants table, column and row-level permissions, so the same governance applies whichever engine reads the data. Athena runs SQL directly against S3 and charges by data scanned, which is why partitioning and columnar formats translate directly into lower cost. QuickSight connects to Athena (among other sources) for dashboards.

3. When to use which

NeedService
Continuously deliver streams or logs to S3, Redshift or OpenSearchAmazon Data Firehose
Discover and register schemas for files in S3Glue Data Catalog and crawlers
Serverless Spark or Python ETLGlue ETL jobs
Chain crawlers and jobs with triggersGlue Workflows (or Step Functions)
Fine-grained access control on lake tablesLake Formation
Ad-hoc SQL over S3 without clustersAthena
Business dashboards and embedded analyticsQuickSight
Provision infrastructure as codeCloudFormation (not an analytics service)

For sustained, high-concurrency warehouse workloads, Amazon Redshift is usually a better fit than Athena; for large custom Spark or Hadoop jobs, EMR offers more control than Glue.

4. Common pitfalls

Related: AWS networking.