Development¶
Before running any individual scripts, make sure to run the shell configuration script:
source scripts/docker_swarm/.env.sh
In ‘.env.sh’, the default server is fixed by ‘IP_ADRESS=127.0.0.1’, which means that Singa-Auto will be using the ‘local’ machine as the server. HOST_WORKDIR_PATH by default is the current directory, and ‘SINGA_AUTO_VERSION’ is set to ‘dev’ for development mode, otherwise, a specific version should be given.
Refer to SINGA-Auto’s Architecture and Folder Structure for a developer’s overview of SINGA-Auto.
Testing Latest Code Changes¶
To test the lastet code changes e.g. in the dev
branch, you’ll need to do the following:
- Build SINGA-Auto’s images on each participating node (the quickstart instructions pull pre-built SINGA-Auto’s images from Docker Hub):
bash scripts/docker_swarm/build_images.sh
- Purge all of SINGA-Auto’s data (since there might be database schema changes):
bash scripts/clean.sh
Making a Release to master
¶
In general, before making a release to master
from dev
, ensure that the code at dev
is stable & well-tested:
- Consider running all of SINGA-Auto’s tests (see Running SINGA-Auto’s Tests). Remember to re-build the Docker images to ensure the latest code changes are reflected (see Testing Latest Code Changes)
- Consider running all of SINGA-Auto’s example models in ./examples/models/
- Consider running all of SINGA-Auto’s example usage scripts in ./examples/scripts/
- Consider running all of SINGA-Auto’s example dataset-preparation scripts in ./examples/datasets/
- Consider visiting SINGA-Auto Web Admin and manually testing it
- Consider building SINGA-Auto’s documentation site and checking if the documentation matches the codebase (see Building SINGA-Auto’s Documentation)
After merging dev
into master
, do the following:
Build & push SINGA-Auto’s new Docker images to SINGA-Auto’s own Docker Hub account:
bash scripts/docker_swarm/build_images.sh bash scripts/push_images.shGet Docker Hub credentials from @nginyc.
Build & deploy SINGA-Auto’s new documentation to
SINGA-Auto's microsite powered by Github Pages
. Run the following:bash scripts/docker_swarm/build_docs.sh latestFinally, commit all resultant generated documentation changes and push them to dev branch. The latest documentation should be reflected at https://singa-auto.readthedocs.io/en/latest/.
Refer to documentation on Github Pages <https://guides.github.com/features/pages/> to understand more on how this works.
Draft a new Singa-Auto Github release. Make sure to include the list of changes relative to the previous release.
Subsequently, you’ll need to increase SINGA_AUTO_VERSION
in .env.sh
to reflect a new release.
Managing SINGA-Auto’s DB¶
By default, you can connect to the PostgreSQL DB using a PostgreSQL client (e.g Postico) with these credentials:
SINGA_AUTO_ADDR=127.0.0.1 POSTGRES_EXT_PORT=5433 POSTGRES_USER=singa_auto POSTGRES_DB=singa_auto POSTGRES_PASSWORD=singa_auto
Connecting to SINGA-Auto’s Redis¶
You can connect to Redis DB with rebrow:
bash scripts/docker_swarm/test/start_rebrow.sh
…with these credentials by default:
SINGA_AUTO_ADDR=127.0.0.1 REDIS_EXT_PORT=6380
Pushing Images to Docker Hub¶
To push the SINGA-Auto’s latest images to Docker Hub (e.g. to reflect the latest code changes):
bash scripts/push_images.sh
Building SINGA-Auto’s Documentation¶
SINGA-Auto uses Sphinx documentation and hosts the documentation with Github Pages on the dev branch. Build & view SINGA-Auto’s Sphinx documentation on your machine with the following commands:
bash scripts/docker_swarm/build_docs.sh latest open docs/index.html
Running SINGA-Auto’s Tests¶
SINGA-Auto uses pytest.
First, start SINGA-Auto.
Then, run all integration tests with:
pip install -r singa_auto/requirements.txt pip install -r singa_auto/advisor/requirements.txt bash scripts/docker_swarm/test/test.sh
Troubleshooting¶
While building SINGA-Auto’s images locally, if you encounter errors like “No space left on device”, you might be running out of space allocated for Docker. Try one of the following:
# Prunes dangling images docker system prune --all# Delete all containers docker rm $(docker ps -a -q) # Delete all images docker rmi $(docker images -q)
From Mac Mojave onwards, due to Mac’s new privacy protection feature, you might need to explicitly give Docker Full Disk Access, restart Docker, or even do a factory reset of Docker.
Using SINGA-Auto Admin’s HTTP interface¶
To make calls to the HTTP endpoints of SINGA-Auto Admin, you’ll need first authenticate with email & password against the POST /tokens endpoint to obtain an authentication token token, and subsequently add the Authorization header for every other call:
Authorization: Bearer {{token}}