odpscmd-tunnel

Bulk data transfer for MaxCompute tables

TLDR

Upload a file to a table

$ odpscmd tunnel upload [file.csv] [project.table]

Download a table to a file

$ odpscmd tunnel download [project.table] [output.csv]

Upload to a specific partition

$ odpscmd tunnel upload [file.csv] [table/pt=value]

Resume a failed upload session

$ odpscmd tunnel resume [session_id]

Upload with custom delimiters and multiple threads

$ odpscmd tunnel upload [file.csv] [table] -fd [,] -rd [\n] -t [4]

Download a limited number of records

$ odpscmd tunnel download [project.table] [output.csv] -limit [1000]

SYNOPSIS

odpscmd tunnel [command] [options]

odpscmd tunnel provides high-speed bulk data transfer between local files and MaxCompute (formerly ODPS) tables. It supports uploading and downloading large datasets with configurable parallelism, compression, and delimiters.Each block of data corresponds to an HTTP request. Multiple blocks can be uploaded in parallel and atomically -- success or failure of one block does not affect others. Transfer speeds typically range from 1 MB/s to 20 MB/s depending on network conditions.

PARAMETERS

upload FILE TABLE

Upload local file data to a MaxCompute table.

download TABLE FILE

Download table data to a local file.

resume ID

Resume an interrupted upload session.

-fd CHAR

Field delimiter (default: comma).

-rd CHAR

Record delimiter (default: newline).

-t NUM

Number of threads for parallel transfer (default: 1).

-bs SIZE

Block size in MiB (default: 100).

-cp BOOL

Enable compression (default: true).

-cf BOOL

Treat the file as CSV format.

-h BOOL

Include a header row in output.

-limit NUM

Limit the number of records to download.

-dbr BOOL

Discard bad records instead of failing.

-mbr NUM

Maximum number of bad records allowed (default: 1000).

-acp BOOL

Auto-create partition if it does not exist.

-te ENDPOINT

Specify a custom Tunnel endpoint.

CAVEATS

Resumable uploads require the session ID from the original upload. Bandwidth limits may apply based on the MaxCompute project configuration. The tool is part of the MaxCompute Console (odpscmd) suite.

HISTORY

odpscmd tunnel is part of the MaxCompute Console provided by Alibaba Cloud for bulk data operations on MaxCompute (originally known as ODPS).