Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

For prod deployments use snapshots instead of rebuilding the db on prod #1295

Closed
robinbb opened this issue Jul 15, 2024 · 5 comments
Closed
Assignees
Labels
DevOps DevOps, Nix, CI/CD, etc.

Comments

@robinbb
Copy link
Member

robinbb commented Jul 15, 2024

After separating Prod Indexer deployments to their own branch (#1756), use the database snapshots to start the Indexer.

This will help avoid the problem where prod goes down because of out-of-memory condition caused by a database rebuild.

@robinbb robinbb converted this from a draft issue Jul 15, 2024
@robinbb robinbb self-assigned this Jul 15, 2024
@jhult
Copy link
Member

jhult commented Jul 15, 2024

Loosely related to Granola-Team/infra#23

Perhaps the same upload mechanism will be used for both.

@robinbb robinbb moved this from In Progress to Next in MinaSearch.com Jul 18, 2024
@robinbb robinbb moved this from Next to In Progress in MinaSearch.com Jul 22, 2024
@Isaac-DeFrain Isaac-DeFrain added the DevOps DevOps, Nix, CI/CD, etc. label Aug 15, 2024
@Isaac-DeFrain
Copy link
Member

What is the status of this?

@jhult jhult changed the title Automated snapshot upload/download to object storage For prod deployments, use prod branch and snapshot Feb 4, 2025
@jhult
Copy link
Member

jhult commented Feb 5, 2025

For branching, this is the idea: https://trunkbaseddevelopment.com/branch-for-release/

Tags may be an interesting possibility:

Late creation of release branches
Some teams release from a tag on the trunk and do not create a branch at that time. That in itself is an alternate practice to this one, “branch for release”.

Those teams wait for a bug that needs fixing for a released, before creating a branch from the release tag (if they are not going to just issue another release from the trunk).

Brad Appleton points out that many do not realize that branches can be created retroactively. That is taken advantage of here in the case of bugs after “release from a tag”, or even changes for point releases.

@robinbb robinbb added this to the Berkeley Support milestone Feb 5, 2025
@robinbb robinbb removed this from MinaSearch.com Feb 5, 2025
@robinbb robinbb modified the milestones: Berkeley Support, Maintenance Feb 5, 2025
@robinbb robinbb moved this from Backlog to In Progress in MinaSearch.com Feb 5, 2025
@robinbb robinbb moved this from In Progress to Blocked in MinaSearch.com Feb 5, 2025
@robinbb robinbb moved this from Blocked to Next in MinaSearch.com Feb 6, 2025
@robinbb robinbb moved this from Next to Blocked in MinaSearch.com Feb 6, 2025
@robinbb robinbb moved this from Blocked to Next in MinaSearch.com Feb 6, 2025
@robinbb robinbb changed the title For prod deployments, use prod branch and snapshot For prod deployments use snapshots instead of rebuilding the db on prod Feb 6, 2025
@robinbb robinbb moved this from Next to In Progress in MinaSearch.com Feb 15, 2025
@robinbb robinbb moved this from In Progress to Next in MinaSearch.com Feb 21, 2025
@robinbb robinbb moved this from Next to In Progress in MinaSearch.com Feb 24, 2025
@robinbb robinbb moved this from In Progress to Next in MinaSearch.com Feb 25, 2025
@robinbb robinbb moved this from Next to In Progress in MinaSearch.com Feb 25, 2025
@jhult
Copy link
Member

jhult commented Feb 26, 2025

If we want to use a CNAME such as db-snapshots.minasearch.com, the Linode bucket name will need to be that exact CNAME.

@robinbb
Copy link
Member Author

robinbb commented Feb 27, 2025

Blocked on #1781 .

@robinbb robinbb moved this from In Progress to Blocked in MinaSearch.com Feb 27, 2025
@robinbb robinbb modified the milestones: Maintenance, Berkeley Support Feb 27, 2025
@jhult jhult moved this from Blocked to In Progress in MinaSearch.com Feb 28, 2025
@robinbb robinbb closed this as completed Mar 1, 2025
@github-project-automation github-project-automation bot moved this from In Progress to Done in MinaSearch.com Mar 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DevOps DevOps, Nix, CI/CD, etc.
Projects
Archived in project
Development

No branches or pull requests

3 participants