$ show pyathena-dev/PyAthena
PyAthena is a Python client library for Amazon Athena that implements the PEP 249 DB-API 2.0 specification with both synchronous and asynchronous execution models. It provides multiple cursor implementations (CSV, Arrow, Pandas, Polars) and an UNLOAD-based Parquet path for large-scale ETL, plus a SQLAlchemy dialect for ORM-style access. The library handles Athena's S3-backed query lifecycle, result pagination, type conversion heuristics for complex types, and the documented null-vs-empty-string semantics that differ between CSV (GetQueryResults) and Parquet (UNLOAD) paths. The architecture is a layered client: a core connection/cursor layer implementing the DB-API contract, an async sibling layer (AioCursor and related classes), and optional integration layers for pandas, Polars, Arrow, and SQLAlchemy. Key entry points are the connection factory, cursor classes, and the SQLAlchemy dialect. The library depends on boto3/botocore for AWS API calls, with optional dependencies for pyarrow, pandas, polars, and SQLAlchemy. It also ships an S3 file system abstraction (sync and async) with caching, multipart uploads, and versioning, plus a benchmark framework that spawns isolated subprocesses with an Observer pattern for SDK event monitoring, memory/thread sampling, and incremental result reporting. Security is treated seriously: the codebase maintains Trino-vs-Hive escaping distinctions for identifiers and string literals, and tests explicitly verify injection resistance. The project is actively maintained with a defined contribution workflow, CI running lint and Sphinx builds, and a worktree-based PR process. Documentation covers Athena-specific limitations including Iceberg type constraints, naming rules, and the fact that Athena does not return element-level type information for complex types, so the converter applies documented heuristics. The project uses hatch-vcs for versioning (pyathena/_version.py must never be edited manually) and supports schema-qualified queries against ENV.schema.
The LLM identified 6 files worth highlighting for reuse:
S3FileSystem and AioS3FileSystem provide sync and async file operation abstractions over S3 with local caching, multipart upload support, and versioning. This is a self-contained, reusable pattern for any project needing S3-backed file I/O with an async twin, demonstrating non-obvious handling of S3 consistency and large-object uploads.
Defines AthenaQueryExecution (query state, statistics, S3 output config, encryption, result reuse, S3 access grants) plus table metadata, file format, SerDe, compression, and partition transform constants. This is the canonical domain model for Athena query metadata and is directly reusable in any Athena tooling that needs to reason about query execution state or table DDL.
Implements the SQLAlchemy dialect for Athena, including schema-qualified query compilation against ENV.schema, LIMIT/OFFSET handling, and Athena-specific type mappings. This is a complete, extractable dialect implementation that any project can reuse to add Athena support to SQLAlchemy-based tooling.
ArrowCursor and AsyncArrowCursor convert Athena's GetQueryResults output into PyArrow tables, implementing columnar type conversion heuristics for Athena's complex types. This is a clean, extractable pattern for any Athena-to-Arrow data path and demonstrates careful handling of Athena's lack of element-level type information for complex types.
PolarsCursor and AsyncPolarsCursor return Polars DataFrames and LazyFrames from Athena queries, including support for the UNLOAD-to-Parquet path. This shows how a modern dataframe library can be integrated with Athena's execution model and is a reusable reference for Polars-based Athena access.
AsyncCursor and related async cursor classes implement the DB-API contract over asyncio using botocore's async client. This is a clean pattern for wrapping a synchronous DB-API driver with an async twin, including async pagination and result fetching.