Development

Before running any individual scripts, make sure to run the shell configuration script:

source scripts/docker_swarm/.env.sh

In ‘.env.sh’, the default server is fixed by ‘IP_ADRESS=127.0.0.1’, which means that Singa-Auto will be using the ‘local’ machine as the server. HOST_WORKDIR_PATH by default is the current directory, and ‘SINGA_AUTO_VERSION’ is set to ‘dev’ for development mode, otherwise, a specific version should be given.

Refer to SINGA-Auto’s Architecture and Folder Structure for a developer’s overview of SINGA-Auto.

Testing Latest Code Changes

To test the lastet code changes e.g. in the dev branch, you’ll need to do the following:

1. Build SINGA-Auto’s images on each participating node (the quickstart instructions pull pre-built SINGA-Auto’s images from Docker Hub):

bash scripts/docker_swarm/build_images.sh

2. Purge all of SINGA-Auto’s data (since there might be database schema changes):

bash scripts/clean.sh

Making a Release to master

In general, before making a release to master from dev, ensure that the code at dev is stable & well-tested:

1. Consider running all of SINGA-Auto’s tests (see Running SINGA-Auto’s Tests). Remember to re-build the Docker images to ensure the latest code changes are reflected (see Testing Latest Code Changes)
2. Consider running all of SINGA-Auto’s example models in ./examples/models/
3. Consider running all of SINGA-Auto’s example usage scripts in ./examples/scripts/
4. Consider running all of SINGA-Auto’s example dataset-preparation scripts in ./examples/datasets/
5. Consider visiting SINGA-Auto Web Admin and manually testing it
6. Consider building SINGA-Auto’s documentation site and checking if the documentation matches the codebase (see Building SINGA-Auto’s Documentation)

After merging dev into master, do the following:

1. Build & push SINGA-Auto’s new Docker images to SINGA-Auto’s own Docker Hub account:

   bash scripts/docker_swarm/build_images.sh
   bash scripts/push_images.sh
   

   Get Docker Hub credentials from @nginyc.

2. Build & deploy SINGA-Auto’s new documentation to SINGA-Auto's microsite powered by Github Pages. Run the following:

   bash scripts/docker_swarm/build_docs.sh latest
   

   Finally, commit all resultant generated documentation changes and push them to dev branch. The latest documentation should be reflected at https://singa-auto.readthedocs.io/en/latest/.

   Refer to documentation on Github Pages <https://guides.github.com/features/pages/> to understand more on how this works.

3. Draft a new Singa-Auto Github release. Make sure to include the list of changes relative to the previous release.

Subsequently, you’ll need to increase SINGA_AUTO_VERSION in .env.sh to reflect a new release.

Managing SINGA-Auto’s DB

By default, you can connect to the PostgreSQL DB using a PostgreSQL client (e.g Postico) with these credentials:

SINGA_AUTO_ADDR=127.0.0.1
POSTGRES_EXT_PORT=5433
POSTGRES_USER=singa_auto
POSTGRES_DB=singa_auto
POSTGRES_PASSWORD=singa_auto

Connecting to SINGA-Auto’s Redis

You can connect to Redis DB with rebrow:

bash scripts/docker_swarm/test/start_rebrow.sh

…with these credentials by default:

SINGA_AUTO_ADDR=127.0.0.1
REDIS_EXT_PORT=6380

Pushing Images to Docker Hub

To push the SINGA-Auto’s latest images to Docker Hub (e.g. to reflect the latest code changes):

bash scripts/push_images.sh

Building SINGA-Auto’s Documentation

SINGA-Auto uses Sphinx documentation and hosts the documentation with Github Pages on the dev branch. Build & view SINGA-Auto’s Sphinx documentation on your machine with the following commands:

bash scripts/docker_swarm/build_docs.sh latest
open docs/index.html

Running SINGA-Auto’s Tests

SINGA-Auto uses pytest.

First, start SINGA-Auto.

Then, run all integration tests with:

pip install -r singa_auto/requirements.txt
pip install -r singa_auto/advisor/requirements.txt
bash scripts/docker_swarm/test/test.sh

Troubleshooting

While building SINGA-Auto’s images locally, if you encounter errors like “No space left on device”, you might be running out of space allocated for Docker. Try one of the following:

# Prunes dangling images
docker system prune --all

# Delete all containers
docker rm $(docker ps -a -q)
# Delete all images
docker rmi $(docker images -q)

From Mac Mojave onwards, due to Mac’s new privacy protection feature, you might need to explicitly give Docker Full Disk Access, restart Docker, or even do a factory reset of Docker.

Using SINGA-Auto Admin’s HTTP interface

To make calls to the HTTP endpoints of SINGA-Auto Admin, you’ll need first authenticate with email & password against the POST /tokens endpoint to obtain an authentication token token, and subsequently add the Authorization header for every other call:

Authorization: Bearer {{token}}