Staging Server Environment (DevOps)

If in a team of 3, one member of the team should establish a staging environment and the three of you should use pair programming techniques to move through these steps. If in a team of 4, two members of the team should establish a staging environment and the other two members should pair up to follow these steps.

Based on the above, choose the members’ CloudApps namespace to setup as the staging environment. All other team members will be added to this namespace as collaborators so everyone has access.

In establishing the staging environment on our cloud infrastructure, we will work from the bottom up. We will start with the database server, then add secrets needed to build and run the application, then add the application, and finally add the route to expose the application to the internet.

Cleaning Up from EX04

1. Login to OpenShift (remember: off-campus access requires connecting to the VPN)
2. Open a Terminal on your Host Machine (not the VSCode Dev Env!)
   1. Login to the OpenShift Terminal on your Host Machine (not the VSCode DevEnv!)
   2. Navigate to your project’s directory in the terminal
   3. Login to OpenShift’s oc program you installed for EX03. You will need a new token from OpenShift: Click your name after logging in, select Copy Login Command, Display Token, copy the oc login ... line, paste into Terminal.
3. Go ahead and delete your EX03 deployment now that everything is graded:
   • Run: oc delete all --selector app=comp423-ex04

Giving Team Members Access to Your Project

1. Add team members to your course workspace

• Navigate to the Administrator sidebar (Default is Developer)
• Select: User Management > Role Bindings
• For each team member, with their onyen:
  1. Create binding
  2. Name: admin-ONYEN (replace ONYEN with teammate’s ONYEN)
  3. Be sure the project you are in in CloudApps is comp590-140-23fa-ONYEN * ONYEN should be your UNC ONYEN
  4. Role name: admin
  5. Subject:
     • User
     • Name: your teammate’s ONYEN

Creating the Database Server

1. Add a PostgreSQL database to your project:
   1. Developer View Sidebar
   2. Add (from the Sidebar)
   3. Database
   4. PostgreSQL Provided by Red Hat
   5. Instantiate Template
   6. Change the following settings:
      1. Database Service Name: db
      2. PostgreSQL Database Name: csxl
      3. Version of PostgreSQL Image: latest
   7. Create

Once the database is created, you can go to the Secrets page and view the generated credentials for the database under db (the name you gave it). If you select “Reveal Values” you can see the name, username, and password for the database. These secrets will be used as environment variables in your application in the next step.

Creating Secrets for your Application

Let’s create a secret for your application to use. This will be used to store the database credentials, and will ultimately be mounted as environment variables in your application.

Make note of the username and password for the database, from above. Additionally, you will need to generate a random string for the JWT_SECRET environment variable. This will be used to sign the JWT tokens that your application will use to authenticate users. You can generate a random string using the following command in your Dev Container:

openssl rand -hex 32

From your host machine’s terminal, in a shell prompt (avoiding PowerShell on windows is encourage!) run the following command to create the secret:

oc create secret generic final-project-environment \
    --from-literal=POSTGRES_HOST=db \
    --from-literal=POSTGRES_PORT=5432 \
    --from-literal=POSTGRES_DATABASE=csxl \
    --from-literal=POSTGRES_USER=<from-secret-above> \
    --from-literal=POSTGRES_PASSWORD=<from-secret-above> \
    --from-literal=JWT_SECRET=<generate-random-string>

From the OpenShift web console, you can verify that the secret was created by navigating to the Secrets page and selecting the final-project-environment secret.

Repository Deploy Key (Secret)

As with in EX04, you will need to add a deploy key to your repository so that OpenShift can pull your code from GitHub. This is the same process as EX03, but you will need to add the deploy key to your final project repository’s settings.

In your Dev Container’s terminal, run the following command to generate a new deploy key:

Important: Do NOT set a passphrase! When prompted for a passphrase, just press enter.

ssh-keygen -t ed25519 -C "Deploy Key for Final Project" -f ./deploy_key

Important: Do NOT set a passphrase! When prompted for a passphrase, just press enter.

This will generate two files: deploy_key and deploy_key.pub. The public key (deploy_key.pub) will need to be added to your repository’s settings, and the private key (deploy_key) will need to be added to OpenShift as a secret.

Add the public key to your final project repository’s settings on GitHub:

1. Navigate to your repository’s settings
2. Select Deploy Keys
3. Add Deploy Key
4. Title: CloudApps Deploy Key
5. Key: Copy the contents of deploy_key.pub into the key field
6. Check the box to allow write access
7. Click Add Key

Create the secret in OpenShift:

1. Back in your host machine’s terminal, not the Dev Container’s terminal, navigate to your project directory and run the following command to create the secret in OpenShift:

oc create secret generic final-project-deploy-key \
    --from-file=ssh-privatekey=./deploy_key \
    --type=kubernetes.io/ssh-auth

To verify the secret was correctly created, run the following command:

oc get secret final-project-deploy-key

Finally, you need to link the secret to the “builder” process of OpenShift. This will allow OpenShift to use the secret when it pulls your code from GitHub and builds your project.

oc secrets link builder final-project-deploy-key

This command will succeed silently.

Create the OpenShift Application

Back in your host’s terminal, since it has the oc command installed, run the following commands to create the application in OpenShift.

IMPORTANT: Be sure you substitute your team’s information in TWO places below! First: the repository URL, second the HOST variable should not be team-z9, but should instead be your zone letter and table number.

IMPORTANT #2: The second team member to deploy will need to add a suffix to the host to avoid conflicting hosts, such as team-z9-backup-comp423-23f.apps.cloudapps.unc.edu

oc new-app python:3.11~git@github.com:comp423-23f/<your-final_repo_name>.git#stage \
  --source-secret=final-project-deploy-key \
  --name=final-project \
  --strategy=docker \
  --env=MODE=development \
  --env=HOST=team-z9-comp423-23f.apps.cloudapps.unc.edu

Notice the #stage at the end of the repository URL. This is the branch name that OpenShift will pull from. When setting up the final project, you created a branch named stage and established it as the primary branch for your repository. This notion of a staging branch is a common practice in DevOps, and is a good way to keep your production code separate (live at csxl.unc.edu) from your development code (which you are establishing right now).

While the project is building, add secrets to the environment variables of the deployment and verify their existence with list:

oc set env deployment/final-project --from=secret/final-project-environment
oc set env deployment/final-project --list

Exposing the Application

Once your application builds, it will be running on a pod that is not exposed to the internet. To establish a public route, first we need to expose it as a service, run the following command:

oc expose deployment final-project \
  --port=80 \
  --target-port=8080

Next, we can create a route to the service with a specifically chosen hostname. Please replace z9 with your team’s zone (lowercase) and table number.

IMPORTANT: The second team member to deploy will need to add a suffix to the host to avoid conflicting hosts, such as team-z9-backup-comp423-23f.apps.cloudapps.unc.edu

oc create route edge \
  --service=final-project \
  --hostname=team-z9-comp423-23f.apps.cloudapps.unc.edu

You can now visit the hostname for your team and access it in the browser. If you see a message from OpenShift that says “Application is not available”, it means that the application is still building. Once your build completes, you should see the application running, but there is still one more important step: resetting the database.

Resetting the Database

The database that you created in the previous step is empty. You will need to reset the database to the state that it was in when you submitted your final project. To do this, you will need to run the reset_demo.py script that is included in your final project repository.

This script needs to be run from within your pod, so in this section you will learn how to connect to your pod and run commands from within it.

First, you will need to find the name of your pod. Run the following command to get a list of pods running in your project:

oc get pods --selector deployment=final-project

You should see a single pod with a name like final-project-648fdff8d5-rr4fs. The letters and numbers at the end of the name are a unique identifier for the pod. This identifier changes every time a new build of your pod is deployed, environment variables change, the pod gets restarted, and in other instances. Copy the name of your running pod and run the following command to connect to it:

oc rsh final-project-YOUR-POD-IDENTIFIER

The rsh stands for “remote shell”. You are now connected to your pod running in the cloud via a secure shell (ssh)! The commands you run are not running on your host machine, but on the CloudApps infrastructure. If you ls you will see you are in your project’s built directory. Not everything is there, importantly not the frontend because it was compiled into the static directory as part of the build process.

To confirm you are logged into your pod, you can assure yourself with the following command:

hostname

You can now run the reset_demo script to reset the database. Run the following command to do so:

python3 -m backend.script.reset_demo

You should see the SQLAlchemy log messages creating tables, inserting dev data, etc. Your staging database is now reset!

Important: As you deploy new versions, add new entities, add new dev data, etc., this process of resetting the database in staging is one you and your team members will both need to be comfortable doing and remember to do.

If you run into issues with resetting the database, you are encouraged to delete and recreate the database between resets:

python3 -m backend.script.delete_database
python3 -m backend.script.create_database
python3 -m backend.script.reset_demo

Setting up Push-to-Deploy Webhooks

GitHub repositories can be configured with webhooks, which are URLs that get called when events occur in order to notify another service of the event. In our case, we want to set up a webhook so that when we merge pull requests to the stage branch, OpenShift’s build configuration for our project will receive a webhook notification and kick off a new build and deploy pipeline.

To find the URL for the web hook, open up your project in OpenShift and navigate to the Admin sidebar, followed by Builds > BuildConfigs. Select final-project and look for the webhooks section at the bottom. Click the Copy URL with Secrets button for the GitHub webhook. This copies the URL to your clipboard.

Next, open your project’s settings in GitHub and navigate to Webhooks. Click Add Webhook and paste the URL into the Payload URL field. Be sure to set the content type to application/json and leave the secret field empty. Click Add Webhook.

From the “Webhooks” page you’re brought back to, click your webhook. Then go to the Recent Deliveries tab. You should see a successful delivery. Congratulations, your project is now set up to automatically build and deploy every time your team merges PRs into the stage branch! As a reminder, if your data entities change, you will need to reset the database in staging after the build and deploy completes.

You’re in Staging/Production!

This setup mirrors our production setup of csxl.unc.edu and is running the same code base. Congratulations on setting up what is, in essence, a production cloud environment for a small, modern web application!

We call this “Staging” in a nod to how many organizations think of “Staging” environments. It’s an area where your team can work on a feature, see it deployed publicly on the internet, and test it without having any impact on our production deployment.