Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

M1-12 Maintenance and DevOps #675

Closed
28 of 29 tasks
Tracked by #878
elisabettai opened this issue Sep 7, 2022 · 20 comments · Fixed by ITISFoundation/osparc-dask-gateway#45, ITISFoundation/osparc-dask-gateway#46, ITISFoundation/osparc-dask-gateway#48, ITISFoundation/osparc-dask-gateway#49 or ITISFoundation/osparc-simcore#3949
Closed
28 of 29 tasks
Tracked by #878

M1-12 Maintenance and DevOps #675

elisabettai opened this issue Sep 7, 2022 · 20 comments · Fixed by ITISFoundation/osparc-dask-gateway#45, ITISFoundation/osparc-dask-gateway#46, ITISFoundation/osparc-dask-gateway#48, ITISFoundation/osparc-dask-gateway#49 or ITISFoundation/osparc-simcore#3949
Assignees
@Surfict @GitHK @mrnicegyu11 @pcrespov @sanderegg @YuryHrytsuk @matusdrobuliak66
Labels
Epic Zenhub label (Pleas do not modify) PO issue Created by Product owners y6 Milestones/Tasks for NIH year 6, "M" in the title means "month" and M1 is September 2022
Milestone
Baklava

Comments

@elisabettai
Copy link
Collaborator

elisabettai commented Sep 7, 2022 •
edited by mrnicegyu11

(This is a task, not a milestone)
Continuously maintain o²S²PARC platform, harden, and increase stability and scalability to accommodate increased use.

For example:

  • enable scaling of selected core services through recently introduced modularization and plugin concept
  • file-system performance boosting, especially for large files

Deliverable: monthly update slides as part of the NIH SIM-Core update meeting
Deadline: Month 1–12 (continuous task)

DETAILS IN https://github.com/orgs/ITISFoundation/projects/9/views/34

Watermelon
Beta Give feedback
  1. Remove invitations machine and expose invitations.osparc.io osparc-ops-environments#172
    t:bug
    mrnicegyu11
  2. Investigate: S3 Storage Size prometheus exporter per project/node osparc-ops-environments#173
    observability
    mrnicegyu11
  3. Make progress on terraform (EPIC) osparc-ops-environments#176
    3 of 3
    EPIC t:enhancement
    mrnicegyu11

Sundae
Beta Give feedback
  1. Make progress on terraform (EPIC) osparc-ops-environments#176
    3 of 3
    EPIC t:enhancement
    mrnicegyu11
  2. Clean Up the ZMT AWS account osparc-ops-environments#200
    FAST p:high-prio
    mrnicegyu11
  3. Re-Provision /docker filesystem as xfs osparc-ops-environments#190
    10 of 24
    blocked / paused during maintenance p:high-prio t:infra-ops
  4. Upgrade all machines to latest docker engine version osparc-ops-environments#215
    5 of 6
    blocked / paused during maintenance p:high-prio
    mrnicegyu11
  5. Grafana import for tip deploy is failing osparc-ops-environments#248
    FAST p:mid-prio t:bug
    mrnicegyu11
  6. Deploy script for aws production doesn't run osparc-ops-environments#250
    FAST p:mid-prio t:bug
    mrnicegyu11
  7. Create IAM AWS accounts for YH osparc-ops-environments#258
    p:high-prio t:infra-ops
    mrnicegyu11
  8. pgsql-query-exporter fails to log into pgsql on dalco-staging osparc-ops-environments#262
    FAST p:high-prio t:bug
    YuryHrytsuk
  9. Fix Dollar Sign Escape in Env-Var in Invitation service on aws-production osparc-ops-environments#274
    5 of 5
    p:high-prio t:bug
    YuryHrytsuk
  10. Audit and prune list of administrators in github Itisfoundation organization osparc-ops-environments#276
    SECURITY p:low-prio
    mrnicegyu11
  11. stack services have been deployed in nodes not assigned to the simcore- stack osparc-ops-environments#278
    p:high-prio t:bug
    YuryHrytsuk
  12. Fix INVITATIONS_URL for aws-production osparc-ops-environments#282
    FAST p:high-prio t:bug
    YuryHrytsuk
  13. Make create-certificates fails due to invalid extfile.cnf osparc-ops-environments#287
    FAST p:mid-prio t:bug

Baklava (ops)
Beta Give feedback
  1. 🐛 Update traefik extractor pattern osparc-ops-environments#310
    FAST p:low-prio t:bug
    YuryHrytsuk
  2. ✨ Add deployment labels for clusters_keeper microservices osparc-ops-environments#312
    p:high-prio
    mrnicegyu11
  3. Disable vcs check on aws/dalco/public deployment osparc-ops-environments#313
    FAST t:bug
    mrnicegyu11
  4. ♻️ Add pyenv to CI docker container osparc-ops-environments#314
    t:enhancement
    mrnicegyu11
  5. ♻️ Fix email credentials in staging osparc-ops-environments#319
    p:high-prio t:bug
    YuryHrytsuk
  6. 🐛 Ceph scraping and simcore container labels osparc-ops-environments#322
    observability p:high-prio t:bug t:infra-ops
    mrnicegyu11
  7. 🐛 node labels inconsistencies: s4l-lite services are not guaranteed anymore to run alone on autoscaled nodes osparc-ops-environments#323
    p:high-prio t:bug
    YuryHrytsuk mrnicegyu11
    sanderegg
  8. ✨ Add docker resource limits to 20% services osparc-ops-environments#329
    p:mid-prio t:enhancement t:infra-ops
    YuryHrytsuk
  9. ✨ Setup new s4l FQDNs osparc-ops-environments#332
    0 of 2
    FAST p:high-prio
    mrnicegyu11
  10. 🐛 Fix itis aws account terraform errors osparc-ops-environments#333
    FAST p:high-prio t:bug
    mrnicegyu11
  11. ✨ Add capture of director-v2 loglevel osparc-ops-environments#335
    FAST observability
    mrnicegyu11
  12. ✨ Add pipeline to standardize traefik log level (uppercase) osparc-ops-environments#338
    FAST p:low-prio
    YuryHrytsuk
  13. ♻️ Increase resouce-usage-tracker service replicas osparc-ops-environments#340
    FAST p:high-prio
    YuryHrytsuk
@elisabettai elisabettai added the PO issue Created by Product owners label Sep 7, 2022
@elisabettai
Copy link
Collaborator Author

This is a continuation of ITISFoundation/osparc-simcore#428, fresh start into Y6. Could also be used to write updates on DevOps side.

This was referenced Sep 9, 2022
Closed
Closed
@sanderegg sanderegg added the Epic Zenhub label (Pleas do not modify) label Sep 9, 2022
@elisabettai elisabettai added the y6 Milestones/Tasks for NIH year 6, "M" in the title means "month" and M1 is September 2022 label Sep 12, 2022
@pcrespov
Copy link
Member

pcrespov commented Oct 5, 2022 •
edited by GitHK

Update on sprint Vaporwave

Done

Ongoing

Open

@sanderegg
Copy link
Member

sanderegg commented Oct 17, 2022 •
edited

Goal for sprint Katherine Switzer

@mrnicegyu11
Copy link
Member

mrnicegyu11 commented Nov 3, 2022 •
edited by sanderegg

Update on sprint Katherine Switzer

User support

Security

Bugfixes / Technical debt

CI/CD

e2e testing

Pre/releases during this sprint

Dependencies

DevOps

  • Traefik Dashboard /api handling, which leads to not showing weird errors if a jupyter-service container dies (thx @elisabettai )
  • Add log-rotation to docker daemon
  • Dont run docker daemon on loglevel DEBUG
  • Harmonize /docker mounts
  • Added 2 grafana visualisations
  • Add e2e test for oSparc products,APIs, and OpenAPI Swagger Doc Pages
  • Allow for CPU/RAM limits in simcore & ops services, mitigating config output strips quote marks and loses version number docker/compose#7771
  • Add pre-commit hooks and linting to opsarc-ops repo (thx @pcrespov)
  • Add graylog alert for dy-volume-removal finding zombies

@pcrespov
Copy link
Member

pcrespov commented Nov 14, 2022 •
edited

Goal for sprint Athena

@sanderegg
Copy link
Member

sanderegg commented Dec 1, 2022 •
edited by mrnicegyu11

Update on sprint Athena

User support

Security

Bugfixes / Technical debt

CI/CD

e2e testing

Pre/releases during this sprint

Dependencies

DevOps

  • Bugfix release for deployment-agent: v.0.10.4 : 🐛 Fix bug: Fix pulling only files with tags osparc-deployment-agent#121
  • Remove burstable aws instances
  • PG Migration v10 --> v14
  • Refactor ops-repo
  • Licensing server adjustments
  • Add pyenv to all machines
  • Add secure sudo password to all machines
  • Increase machine memory on osparc.io managers
  • Terraform: Prototype for EC2, VPC, Subnets, Elastic IPs, SSH Keys, LoadBalancer
  • Admin-Panel: Add docker socket access, script by @GitHK

@pcrespov pcrespov mentioned this issue Dec 2, 2022
Closed
4 tasks
@sanderegg
Copy link
Member

sanderegg commented Dec 12, 2022 •
edited

Goal for sprint Zefram Cochrane

This was referenced Dec 14, 2022
Merged
Merged
Closed
@sanderegg
Copy link
Member

sanderegg commented Jan 10, 2023 •
edited by mrnicegyu11

Update on sprint Zefram Cochrane

User support

Security

Bugfixes / Technical debt

CI/CD

e2e testing

Pre/releases during this sprint

Dependencies

DevOps

  • Re-Establish Postgres database backups
  • Create AWS Machine Image (AMI) with S4L-lite already on it
  • Enable dynamic services in local deployment
  • Switch staging.osparc.io to new github repository
  • Proof of Concept for limiting maximum filesize in a docker container
  • Add s4l-lite domains and products everywhere
  • Enable E2E tests when maintenance page is still up
  • Fix: graylog dashboard provisioning
  • Service deprecation on osparc.io (Date: 30th of Jan 2023)

@sanderegg
Copy link
Member

sanderegg commented Feb 16, 2023 •
edited by mrnicegyu11

Update on sprint Resistance is Futile

User support

Security

Bugfixes / Technical debt

CI/CD

e2e testing

Pre/releases during this sprint

DevOps

@sanderegg sanderegg added this to the Mithril milestone Feb 28, 2023
@sanderegg
Copy link
Member

sanderegg commented Mar 2, 2023 •
edited by pcrespov

Goal for sprint Mithril

Waiting to decide which issues of the Maintenance panel get here.

--

@mrnicegyu11
Copy link
Member

mrnicegyu11 commented Mar 6, 2023 •
edited by sanderegg

Goal for sprint Mithril

Taken from https://git.speag.com/oSparc/osparc-ops-environments/-/issues?milestone_title=Mithril

This was referenced Mar 7, 2023
Merged
Merged
@pcrespov
Copy link
Member

pcrespov commented Jul 7, 2023 •
edited

Update Watermelon

Highlights

  • 🔨 Tooling:
    • Almost finished with full-repo type-checking , i.e. mypy
    • Integrated ruff: an extremly fast python linter
  • 🚀 Releases
    • 4x releases to staging
    • 2x releases to production + 1 hotfix
  • ⬆️ Dependency upgrades (requirements.txt)
  • 🏷️ New convention: Redefined services labels names to follow inversed url notation
  • ✅ json-schemas for all service configurations as artifacts of CI -> can validate stack configuration before deploy!
  • ♻️ Towards sqlalchemy 2.0

In detail

@pcrespov pcrespov mentioned this issue Jul 10, 2023
Open
@pcrespov pcrespov mentioned this issue Jul 20, 2023
Closed
1 task
@pcrespov pcrespov modified the milestones: Watermelon, Sundae Jul 25, 2023
@odeimaiz odeimaiz removed their assignment Jul 25, 2023
@YuryHrytsuk YuryHrytsuk self-assigned this Jul 25, 2023
This was referenced Aug 7, 2023
Merged
Merged
@sanderegg
Copy link
Member

sanderegg commented Aug 10, 2023 •
edited by matusdrobuliak66

Update Sundae

Highlights:

  • completed mypy type-checking on osparc-simcore repository, CI steps checks for all python-based packages
  • improved operations tooling for releases

In details:

@pcrespov pcrespov modified the milestones: Sundae, Baklava Aug 21, 2023
@pcrespov pcrespov mentioned this issue Aug 25, 2023
Open
@sanderegg sanderegg mentioned this issue Aug 28, 2023
Open
1 task
@sanderegg
Copy link
Member

sanderegg commented Sep 7, 2023 •
edited

Update Baklava

In details:

@elisabettai
Copy link
Collaborator Author

elisabettai commented Sep 11, 2023 •
edited

@sanderegg, @pcrespov, @matusdrobuliak66. I am closing this one. For Y7 (simcore)-related maintenance, please use this new one #1108

@mrnicegyu11, @YuryHrytsuk: for operations, we have a new issue in Y7 as well: #1109. I guess we can move the tasks still open here in that new one. I can do that, if that's useful.

@sanderegg, @pcrespov, @matusdrobuliak66, @mrnicegyu11, @YuryHrytsuk. In Y7, we also have a series of new cases, which are a bit of mixture of comp. backend, operations, monitoring, etc... Those should cover the work we're doing for the release. Those new cases are:
#1100
#1101
#1102
Maybe I will try to link the new ones with the work packages we already have. The goal is not to add more work/useless issues, only making reporting on those new milestones easier. 😉

@mrnicegyu11 mrnicegyu11 mentioned this issue Nov 21, 2023
Open
1 task
@matusdrobuliak66 matusdrobuliak66 mentioned this issue Feb 6, 2024
Open
7 tasks
@bisgaard-itis bisgaard-itis mentioned this issue Feb 19, 2024
Closed
1 task
@bisgaard-itis bisgaard-itis mentioned this issue Mar 18, 2024
Closed
1 task
This was referenced Apr 2, 2024
Closed
Open
Closed
Closed
@bisgaard-itis bisgaard-itis mentioned this issue Apr 10, 2024
Closed
1 task
This was referenced Apr 23, 2024
Open
Closed
@bisgaard-itis bisgaard-itis mentioned this issue May 2, 2024
Closed
1 task
@matusdrobuliak66 matusdrobuliak66 mentioned this issue May 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Surfict Surfict

@GitHK GitHK

@mrnicegyu11 mrnicegyu11

@pcrespov pcrespov

@sanderegg sanderegg

@YuryHrytsuk YuryHrytsuk

@matusdrobuliak66 matusdrobuliak66

Labels
Epic Zenhub label (Pleas do not modify) PO issue Created by Product owners y6 Milestones/Tasks for NIH year 6, "M" in the title means "month" and M1 is September 2022
Projects
None yet
Milestone
Baklava
Development

Successfully merging a pull request may close this issue.

⬆️ Mainteance: update requirements + pre-commit sanderegg/osparc-dask-gateway
⬆️Maintenance: docker compose v2 sanderegg/osparc-dask-gateway
♻️Maintenance: fixes and refactoring sanderegg/osparc-dask-gateway
⬆️ Removed bokeh dependency + upgrade reqs sanderegg/osparc-dask-gateway
⬆️ Maintenance: upgrade dask libraries sanderegg/osparc-simcore
⬆️ CI: upgrade ubuntu/buildx sanderegg/osparc-simcore
⬆️ Autoscaling: requirements update sanderegg/osparc-simcore
♻️ Maintenance: cleanup catalog sanderegg/osparc-simcore
🐛Docker buildx 0.10.2 does not resolve correctly ENVs in docker-compose labels section sanderegg/osparc-simcore
🐛CATALOG_SERVICES_DEFAULT_RESOURCES missing a S in docker-compose sanderegg/osparc-simcore
9 participants
@Surfict @GitHK @mrnicegyu11 @elisabettai @pcrespov @odeimaiz @sanderegg @YuryHrytsuk @matusdrobuliak66