Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(mcs): MCOL-5618: change list option to --list and refactor. #3439

Merged
merged 1 commit into from
Mar 20, 2025
Merged

fix(mcs): MCOL-5618: change list option to --list and refactor. #3439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 19 additions & 14 deletions cmapi/mcs_cluster_tool/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,26 +53,28 @@ Name of the bucket to store the columnstore backups.
Example: "s3://my-cs-backups"
* `-url, --endpoint-url TEXT`: Used by on premise S3 vendors.
Example: "http://127.0.0.1:8000"
* `-nv-ssl, --no-verify-ssl / -v-ssl, --verify-ssl`: Skips verifying ssl certs, useful for onpremise s3 storage. [default: v-ssl]
* `-s, --storage TEXT`: What storage topogoly is being used by Columnstore - found in /etc/columnstore/storagemanager.cnf.
Options: "LocalStorage" or "S3" [default: LocalStorage]
* `-i, --incremental TEXT`: Adds columnstore deltas to an existing full backup. Backup folder to apply increment could be a value or "auto_most_recent" - the incremental backup applies to last full backup.
* `-P, --parallel INTEGER`: Determines if columnstore data directories will have multiple rsync running at the same time for different subfolders to parallelize writes. Ignored if "-c/--compress" argument not set. [default: 4]
* `-ha, --highavilability / -no-ha, --no-highavilability`: Hint wether shared storage is attached @ below on all nodes to see all data
HA LocalStorage ( /var/lib/columnstore/dataX/ )
HA S3 ( /var/lib/columnstore/storagemanager/ ) [default: no-ha]
* `-f, --config-file TEXT`: Path to backup configuration file to load variables from.
* `-f, --config-file TEXT`: Path to backup configuration file to load variables from - relative or full path accepted.
* `-sbrm, --skip-save-brm / -no-sbrm, --no-skip-save-brm`: Skip saving brm prior to running a backup - ideal for dirty backups. [default: no-sbrm]
* `-spoll, --skip-polls / -no-spoll, --no-skip-polls`: Skip sql checks confirming no write/cpimports running. [default: no-spoll]
* `-slock, --skip-locks / -no-slock, --no-skip-locks`: Skip issuing write locks - ideal for dirty backups. [default: no-slock]
* `-smdb, --skip-mariadb-backup / -no-smdb, --no-skip-mariadb-backup`: Skip running a mariadb-backup for innodb data - ideal for incremental dirty backups. [default: no-smdb]
* `-sb, --skip-bucket-data / -no-sb, --no-skip-bucket-data`: Skip taking a copy of the columnstore data in the bucket. [default: no-sb]
* `-nb, --name-backup TEXT`: Define the name of the backup - default: $(date +%m-%d-%Y) [default: 03-20-2025]
* `-c, --compress TEXT`: Compress backup in X format - Options: [ pigz ].
* `-q, --quiet / -no-q, --no-quiet`: Silence verbose copy command outputs. [default: no-q]
* `-nv-ssl, --no-verify-ssl / -v-ssl, --verify-ssl`: Skips verifying ssl certs, useful for onpremise s3 storage. [default: v-ssl]
* `-pi, --poll-interval INTEGER`: Number of seconds between poll checks for active writes & cpimports. [default: 5]
* `-pmw, --poll-max-wait INTEGER`: Max number of minutes for polling checks for writes to wait before exiting as a failed backup attempt. [default: 60]
* `-q, --quiet / -no-q, --no-quiet`: Silence verbose copy command outputs. [default: no-q]
* `-c, --compress TEXT`: Compress backup in X format - Options: [ pigz ].
* `-P, --parallel INTEGER`: Determines if columnstore data directories will have multiple rsync running at the same time for different subfolders to parallelize writes. Ignored if "-c/--compress" argument not set. [default: 4]
* `-nb, --name-backup TEXT`: Define the name of the backup - default: $(date +%m-%d-%Y) [default: 03-06-2025]
* `-r, --retention-days INTEGER`: Retain backups created within the last X days, default 0 == keep all backups. [default: 0]
* `-aro, --apply-retention-only`: Only apply retention policy to existing backups, does not run a backup.
* `-li, --list`: List backups.
* `--help`: Show this message and exit.

## `mcs dbrm_backup`
Expand All @@ -87,13 +89,14 @@ $ mcs dbrm_backup [OPTIONS]

**Options**:

* `-m, --mode TEXT`: "loop" or "once" ; Determines if this script runs in a forever loop sleeping -i minutes or just once. [default: once]
* `-i, --interval INTEGER`: Number of minutes to sleep when --mode=loop. [default: 90]
* `-r, --retention-days INTEGER`: Retain dbrm backups created within the last X days, the rest are deleted [default: 7]
* `-p, --path TEXT`: Path of where to save the dbrm backups on disk. [default: /tmp/dbrm_backups]
* `-nb, --name-backup TEXT`: Custom name to prefex dbrm backups with. [default: dbrm_backup]
* `-q, --quiet / -no-q, --no-quiet`: Silence verbose copy command outputs. [default: no-q]
* `-bl, --backup-location TEXT`: Path of where to save the dbrm backups on disk. [default: /tmp/dbrm_backups]
* `-m, --mode TEXT`: "loop" or "once" ; Determines if this script runs in a forever loop sleeping -i minutes or just once. [default: once]
* `-nb, --name-backup TEXT`: Define the prefix of the backup - default: dbrm_backup+date +%Y%m%d_%H%M%S [default: dbrm_backup]
* `-ssm, --skip-storage-manager / -no-ssm, --no-skip-storage-manager`: Skip backing up storagemanager directory. [default: no-ssm]
* `-q, --quiet / -no-q, --no-quiet`: Silence verbose copy command outputs. [default: no-q]
* `-li, --list`: List backups.
* `--help`: Show this message and exit.

## `mcs restore`
Expand Down Expand Up @@ -127,15 +130,16 @@ Options: "LocalStorage" or "S3" [default: LocalStorage]
* `-nr, --new-region TEXT`: Defines the region of the new bucket to copy the s3 data to from the backup bucket.
* `-nk, --new-key TEXT`: Defines the aws key to connect to the new_bucket.
* `-ns, --new-secret TEXT`: Defines the aws secret of the aws key to connect to the new_bucket.
* `-P, --parallel INTEGER`: Determines number of decompression and mdbstream threads. Ignored if "-c/--compress" argument not set. [default: 4]
* `-ha, --highavilability / -no-ha, --no-highavilability`: Flag for high available systems (meaning shared storage exists supporting the topology so that each node sees all data) [default: no-ha]
* `-cont, --continue / -no-cont, --no-continue`: This acknowledges data in your --new_bucket is ok to delete when restoring S3. When set to true skips the enforcement that new_bucket should be empty prior to starting a restore. [default: no-cont]
* `-f, --config-file TEXT`: Path to backup configuration file to load variables from.
* `-f, --config-file TEXT`: Path to backup configuration file to load variables from - relative or full path accepted.
* `-smdb, --skip-mariadb-backup / -no-smdb, --no-skip-mariadb-backup`: Skip restoring mariadb server via mariadb-backup - ideal for only restoring columnstore. [default: no-smdb]
* `-sb, --skip-bucket-data / -no-sb, --no-skip-bucket-data`: Skip restoring columnstore data in the bucket - ideal if looking to only restore mariadb server. [default: no-sb]
* `-c, --compress TEXT`: Hint that the backup is compressed in X format. Options: [ pigz ].
* `-P, --parallel INTEGER`: Determines number of decompression and mdbstream threads. Ignored if "-c/--compress" argument not set. [default: 4]
* `-q, --quiet / -no-q, --no-quiet`: Silence verbose copy command outputs. [default: no-q]
* `-nv-ssl, --no-verify-ssl / -v-ssl, --verify-ssl`: Skips verifying ssl certs, useful for onpremise s3 storage. [default: v-ssl]
* `-li, --list`: List backups.
* `--help`: Show this message and exit.

## `mcs dbrm_restore`
Expand All @@ -150,11 +154,12 @@ $ mcs dbrm_restore [OPTIONS]

**Options**:

* `-p, --path TEXT`: Path of where dbrm backups stored on disk. [default: /tmp/dbrm_backups]
* `-d, --directory TEXT`: Date or directory chose to restore from.
* `-bl, --backup-location TEXT`: Path of where dbrm backups exist on disk. [default: /tmp/dbrm_backups]
* `-l, --load TEXT`: Name of the directory to restore from -bl
* `-ns, --no-start`: Do not attempt columnstore startup post dbrm_restore.
* `-sdbk, --skip-dbrm-backup / -no-sdbk, --no-skip-dbrm-backup`: Skip backing up dbrms before restoring. [default: sdbk]
* `-ssm, --skip-storage-manager / -no-ssm, --no-skip-storage-manager`: Skip backing up storagemanager directory. [default: ssm]
* `-li, --list`: List backups.
* `--help`: Show this message and exit.

## `mcs help-all`
Expand Down
151 changes: 84 additions & 67 deletions cmapi/mcs_cluster_tool/backup_commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,13 +76,6 @@ def backup(
)
)
] = '',
nv_ssl: Annotated[
bool,
typer.Option(
'-nv-ssl/-v-ssl','--no-verify-ssl/--verify-ssl',
help='Skips verifying ssl certs, useful for onpremise s3 storage.'
)
] = False,
s: Annotated[
str,
typer.Option(
Expand All @@ -107,6 +100,18 @@ def backup(
show_default=False
)
] = '',
P: Annotated[
int,
typer.Option(
'-P', '--parallel',
help=(
'Determines if columnstore data directories will have '
'multiple rsync running at the same time for different '
'subfolders to parallelize writes. '
'Ignored if "-c/--compress" argument not set.'
)
)
] = 4,
ha: Annotated[
bool,
typer.Option(
Expand Down Expand Up @@ -171,73 +176,68 @@ def backup(
help='Skip taking a copy of the columnstore data in the bucket.'
)
] = False,
pi: Annotated[
int,
nb: Annotated[
str,
typer.Option(
'-pi', '--poll-interval',
help=(
'Number of seconds between poll checks for active writes & '
'cpimports.'
)
'-nb', '--name-backup',
help='Define the name of the backup - default: $(date +%m-%d-%Y)'
)
] = 5,
pmw: Annotated[
int,
] = datetime.now().strftime('%m-%d-%Y'),
m: Annotated[
str,
typer.Option(
'-pmw', '--poll-max-wait',
'-m', '--mode',
help=(
'Max number of minutes for polling checks for writes to wait '
'before exiting as a failed backup attempt.'
)
'Modes ["direct","indirect"] - direct backups run on the '
'columnstore nodes themselves. indirect run on another '
'machine that has read-only mounts associated with '
'columnstore/mariadb\n'
),
hidden=True
)
] = 60,
] = 'direct',
c: Annotated[
str,
typer.Option(
'-c', '--compress',
help='Compress backup in X format - Options: [ pigz ].',
show_default=False
)
] = '',
q: Annotated[
bool,
typer.Option(
'-q/-no-q', '--quiet/--no-quiet',
help='Silence verbose copy command outputs.'
)
] = False,
c: Annotated[
str,
nv_ssl: Annotated[
bool,
typer.Option(
'-c', '--compress',
help='Compress backup in X format - Options: [ pigz ].',
show_default=False
'-nv-ssl/-v-ssl','--no-verify-ssl/--verify-ssl',
help='Skips verifying ssl certs, useful for onpremise s3 storage.'
)
] = '',
P: Annotated[
] = False,
pi: Annotated[
int,
typer.Option(
'-P', '--parallel',
'-pi', '--poll-interval',
help=(
'Determines if columnstore data directories will have '
'multiple rsync running at the same time for different '
'subfolders to parallelize writes. '
'Ignored if "-c/--compress" argument not set.'
'Number of seconds between poll checks for active writes & '
'cpimports.'
)
)
] = 4,
nb: Annotated[
str,
typer.Option(
'-nb', '--name-backup',
help='Define the name of the backup - default: $(date +%m-%d-%Y)'
)
] = datetime.now().strftime('%m-%d-%Y'),
m: Annotated[
str,
] = 5,
pmw: Annotated[
int,
typer.Option(
'-m', '--mode',
'-pmw', '--poll-max-wait',
help=(
'Modes ["direct","indirect"] - direct backups run on the '
'columnstore nodes themselves. indirect run on another '
'machine that has read-only mounts associated with '
'columnstore/mariadb\n'
),
hidden=True
'Max number of minutes for polling checks for writes to wait '
'before exiting as a failed backup attempt.'
)
)
] = 'direct',
] = 60,
r: Annotated[
int,
typer.Option(
Expand All @@ -248,13 +248,23 @@ def backup(
)
)
] = 0,
aro: Annotated[
bool,
typer.Option(
'-aro', '--apply-retention-only',
help=(
'Only apply retention policy to existing backups, '
'does not run a backup.'
)
)
] = False,
list: Annotated[
bool,
typer.Option(
'list',
'-li', '--list',
help='List backups.'
)
] = False
] = False,
):
"""Backup Columnstore and/or MariDB data."""

Expand Down Expand Up @@ -286,16 +296,6 @@ def backup(

@handle_output
def dbrm_backup(
m: Annotated[
str,
typer.Option(
'-m', '--mode',
help=(
'"loop" or "once" ; Determines if this script runs in a '
'forever loop sleeping -i minutes or just once.'
),
)
] = 'once',
i: Annotated[
int,
typer.Option(
Expand All @@ -320,6 +320,16 @@ def dbrm_backup(
help='Path of where to save the dbrm backups on disk.'
)
] = '/tmp/dbrm_backups',
m: Annotated[
str,
typer.Option(
'-m', '--mode',
help=(
'"loop" or "once" ; Determines if this script runs in a '
'forever loop sleeping -i minutes or just once.'
),
)
] = 'once',
nb: Annotated[
str,
typer.Option(
Expand All @@ -330,18 +340,25 @@ def dbrm_backup(
)
)
] = 'dbrm_backup',
ssm: Annotated[
bool,
typer.Option(
'-ssm/-no-ssm', '--skip-storage-manager/--no-skip-storage-manager',
help='Skip backing up storagemanager directory.'
)
] = False,
q: Annotated[
bool,
typer.Option(
'-q/-no-q', '--quiet/--no-quiet',
help='Silence verbose copy command outputs.'
)
] = False,
ssm: Annotated[
list: Annotated[
bool,
typer.Option(
'-ssm/-no-ssm', '--skip-storage-manager/--no-skip-storage-manager',
help='Skip backing up storagemanager directory.'
'-li', '--list',
help='List backups.'
)
] = False,
):
Expand Down
Loading