Skip to content

Latest commit

 

History

History
76 lines (61 loc) · 4.72 KB

storage-servers.md

File metadata and controls

76 lines (61 loc) · 4.72 KB
Raw

Git Storage Servers

Currently, we run 24 storage servers for git repository data, with 4 of them being used for new projects. Once the 4 servers for new projects reach 60% capacity, we build 4 new ones and set them to be the default.

here's a recording from one time when we had to add new nodes

Building VMs

In order to build a new storage server, you will add the necessary definitions to the GitLab terraform repository. With the move to GCP, this is quite simple. All you must do is bump the number of multizone-store servers in the variables.tf. DO NOT increase the number of stor servers. We should only build multi-zone servers from now on.

Initial configuration

Thanks to our bootstrap script, no manual configuration is needed. However, it does NOT automatically create /var/opt/gitlab/git-data/repositories. You will need to manually create the directory:

$ sudo mkdir -p /var/opt/gitlab/git-data/repositories
$ sudo chown git:git /var/opt/gitlab/git-data/
$ sudo chown git:git /var/opt/gitlab/git-data/repositories/

make sure chef-client runs without any errors! check that the gitaly service is running: gitlab-ctl status gitaly and there are no errors in logs in /var/log/gitlab/gitaly/current

Gitlab application configuration

Once the servers have been configured as above, you will need to tell the GitLab application about them. This is done via updates to the gprd-base and gprd-base-stor-gitaly roles json files. The gprd-base role file will need to be updated with the Gitaly storage targets as seen in this MR. Please note, you no longer need to add mount data, only the Gitaly storage targets. You will also need to add the new servers to the gprd-base-stor-gitaly role file otherwise Gitaly will not know about the new servers which will cause strange errors.

In order to roll out new config:

  1. prepare an MR in the chef-repo with the relevant changes
  2. check status of chef on prod machines that had their roles edited in the chef MR
$ knife ssh -C 5 "roles:gprd-base-stor-gitaly OR roles:gprd-base NOT name:bastion-01-inf-gprd.c.gitlab-production.internal NOT name:bastion-02-inf-gprd.c.gitlab-production.internal NOT name:bastion-03-inf-gprd.c.gitlab-production.internal" "sudo systemctl is-active chef-client.service"
  1. now stop chef-client on those nodes:
$ knife ssh -C 5 "roles:gprd-base-stor-gitaly OR roles:gprd-base NOT name:bastion-01-inf-gprd.c.gitlab-production.internal NOT name:bastion-02-inf-gprd.c.gitlab-production.internal NOT name:bastion-03-inf-gprd.c.gitlab-production.internal" "sudo systemctl stop chef-client.service"
  1. merge the MR in the chef-repo that you prepared earlier
  2. do a dry run on one old gitaly machine, one new gitaly machine, one web machine and confirm the changes are as desired, for example:
$ knife ssh 'name:web-cny-01-sv-gprd.c.gitlab-production.internal' 'sudo chef-client --why-run'
$ knife ssh 'name:file-01-stor-gprd.c.gitlab-production.internal' 'sudo chef-client --why-run'
$ knife ssh 'name:file-33-stor-gprd.c.gitlab-production.internal' 'sudo chef-client --why-run'
  1. force a chef-client run on gitaly nodes (if you run chef on web/api nodes at this point they would be trying to connect to gitaly nodes before they were ready):
$ knife ssh -C 2 "roles:gprd-base-stor-gitaly" "sudo chef-client"
  1. check gitaly logs to confirm they are fine
  2. run chef-client on remaining machines, gradually:
$ knife ssh -C 2 "roles:gprd-base" "sudo chef-client"

Testing new nodes

To confirm new storage nodes are operational:

  1. Create a NEW project, do not push any data to it
  2. Use the API to move it to a new storage server before pushing any data to it
  3. Now that the project is moved, push some data to it and ensure that everything works. Namely, be sure that the web interface updates with the data you've pushed.

Configuring Gitlab to use new storage nodes

If all of the above works, use your admin account to change where new projects are stored. In Admin Panel -> Settings -> Repository -> Repository storage (https://staging.gitlab.com/admin/application_settings/repository) , you'll see a list of storage nodes. The ones that are highglighted are the ones that will receive new projects. For more information see gitlab docs.