This page is a hub for the languages and tools used across this site's data and platform engineering notes. The link tables below point to the detailed pages; the sections around them explain where each language fits in a modern data platform and how to choose between them.
A typical data platform is polyglot. Python is the general-purpose language: ingestion scripts, REST APIs with Flask or FastAPI, orchestration code in Airflow, PySpark jobs and machine learning. SQL is where most business logic for analytics lives, from warehouse transformations to dimensional models, because a query optimizer can execute set-based logic far more efficiently than procedural code. Bash glues systems together: provisioning, deployment hooks, cron jobs and quick file processing. JVM languages (Scala and Java) power Spark, Kafka, Flink and NiFi internals, and Go dominates infrastructure tooling such as Docker, Kubernetes and Terraform. Declarative configuration (YAML for CI/CD and Kubernetes, HCL for Terraform) is the final layer that describes how everything is deployed.
|
|
The diagram summarises the trade-offs. Python's strength is its ecosystem and speed of development, but pure Python loops are slow, so heavy numeric or data work should be pushed into vectorised libraries (NumPy, pandas, Polars) or distributed engines (Spark). SQL is the best choice whenever data already sits in a database or warehouse: move the computation to the data instead of pulling data into an application. Go is a good fit for small, fast, easily deployed services and command-line tools. Scala and Java are mostly needed when extending JVM-based frameworks or writing performance-critical stream processing.
Count orders per region from a CSV file with columns order_id,region,amount:
# Bash: quick one-off check on a small file
tail -n +2 orders.csv | cut -d, -f2 | sort | uniq -c | sort -rn
# Python (pandas): when the result feeds further processing
import pandas as pd
df = pd.read_csv("orders.csv")
print(df.groupby("region")["amount"].agg(["count", "sum"]).sort_values("count", ascending=False))
-- SQL: when the data is already in a table, and for production pipelines
SELECT region, COUNT(*) AS orders, SUM(amount) AS revenue
FROM orders
GROUP BY region
ORDER BY orders DESC;
The Bash version breaks on quoted fields containing commas, the pandas version is limited by one machine's memory, and the SQL version scales with the engine. Each is the right answer in a different context, which is the main point of being comfortable in all three.
| Language | Typing / execution | Main uses on this site | Main weakness |
|---|---|---|---|
| Python | Dynamic, interpreted | APIs, ETL, PySpark, ML, automation | CPU-bound loops; packaging |
| SQL | Declarative, engine-optimised | Warehouse transforms, analytics, dimensional models | Procedural logic, testing |
| Bash | Shell, interpreted | Deployment, cron jobs, file utilities | Error handling, quoting, portability |
| Go | Static, compiled | CLI tools, small services, infrastructure | Smaller data and ML ecosystem |
| Scala / Java | Static, JVM | Spark, Kafka, Flink, NiFi extensions | Verbosity, build complexity |
A practical learning path for data engineering is SQL first (it is used everywhere and its concepts, joins, grouping and window functions, carry over to Spark and pandas), then Python for everything around the database, then enough Bash to automate servers and pipelines. Learn a JVM language or Go when a specific system requires it rather than in advance. Across all of them, the habits that matter most are the same: version control, automated tests, readable code reviewed by others, and reproducible environments built by a CI/CD pipeline rather than by hand on a server. Type hints in Python and linters for SQL and shell scripts (for example ShellCheck) catch many errors before the code runs.