odpscmd-tunnel

[Interactive] Download table to local file

$ tunnel download [table_name] [path/to/file];

$ tunnel upload [path/to/file] [table_name]/[partition_spec];

$ tunnel upload [path/to/file] [table_name] -fd [field_delim] -rd [record_delim];

$ tunnel upload [path/to/file] [table_name] -threads [num];

odpscmd tunnel upload|download|ls|du [options] [args]

-h, --help
    Display help and exit

-f, --file FILE
    Local file or directory path (required for upload/download)

-t, --table TABLE
    Remote table name (required for upload/download)

-p, --partition PARTITION
    Table partition spec, e.g., ds=20230101

-D, --delimiter DELIMITER
    Field delimiter (default: comma for upload)

--header
    Treat first line as column headers

--async
    Run in asynchronous mode

--overwrite
    Overwrite existing data

--compression {NONE,LZO,SNAPPY}
    Compression type (default: auto)

-s, --sql SQL
    SQL query for download (alternative to -t)

-d, --dir DIR
    Local output directory for download

--project PROJECT
    MaxCompute project name

--endpoint ENDPOINT
    Tunnel service endpoint

--accessid ID
    AccessKey ID

--accesskey KEY
    AccessKey Secret

--verbose
    Enable verbose logging

--lifecycle DAYS
    Tunnel expiration in days (ls/du)

odpscmd tunnel is a subcommand of the odpscmd client for Alibaba Cloud MaxCompute (formerly ODPS). It enables efficient, secure transfer of large datasets between local filesystems and MaxCompute tables via the Tunnel service. Supports upload for loading data into tables/partitions, download for exporting table data or SQL results, ls to list active tunnels, and du for usage stats.

Key advantages include parallel processing for high throughput, resumable transfers, custom formats (CSV, etc.), compression (NONE, LZO, SNAPPY), header handling, and partition support. Ideal for ETL pipelines, migrations, and batch jobs. Requires Tunnel upload/download permissions and respects quotas (e.g., 5GB per file upload). Authentication via AccessKey or STS token.

Handles directories recursively, overwrites with caution, and supports async mode for non-blocking operations. Integrates with odpscmd config for endpoints/projects.

Requires Tunnel permissions and service enabled; file size limits (e.g., 5GB upload); no support for complex types like ARRAY/MAP; rate-limited by quotas; ensure consistent delimiters to avoid data errors.

Upload: odpscmd tunnel upload -f data.csv -t pt.table -p ds=20230101 --header
Download: odpscmd tunnel download -t pt.table -d ./output --compression SNAPPY

Install via MaxCompute console download; configure ~/.odpsconf; Tunnel quota via project settings.

Introduced with ODPS (2013), evolved in MaxCompute (2016+). Enhanced for resumability/compression in v2.0+; integrated STS auth in recent updates for security.

odpscmd(1), scp(1), rsync(1)