Developer install¶
Installation requirements¶
Just as for a regular installation, you will need docker and docker-compose (see Installation requirements above).
During development, you will typically also want to run FlowMachine, FlowAPI, FlowAuth and/or AutoFlow outside docker containers. This requires additional prerequisites to be available.
- Pipenv (to manage separate pipenv environment for each FlowKit component)
- FlowMachine server:
Python >= 3.7 - FlowAuth:
npm(we recommend installing it via nvm); Cypress for testing
Setting up FlowKit for development¶
After cloning the GitHub repository, the FlowKit system can be started by running set -a && . development_environment && set +a (this will set the required environment variables) followed by make up in the root directory. This requires Docker and Docker Compose to be installed, and starts the FlowKit docker containers using the docker-compose.yml file. The docker containers to start can be selected by running make up DOCKER_SERVICES="<service 1> <service 2> ...", where the services can be any of flowmachine, flowmachine_query_locker, flowapi, flowauth, flowdb, worked_examples, flowdb_testdata, flowdb_synthetic_data, flowetl or flowetl_db (at most one of the flowdb containers). The default is to start flowdb, flowapi, flowmachine, flowauth, flowmachine_query_locker, flowetl, flowetl_db and worked_examples. Alternatively, containers can be built, started or stopped individually by running make <service>-build, make <service>-up or make <service>-down, respectively.
FlowKit uses pipenv to manage Python environments. To start a Python session in which you can use FlowClient:
cd flowclient
pipenv install
pipenv run python
>>> import flowclient
To run the tests in the flowapi, flowclient, flowdb, flowmachine, flowetl or integration_tests directory:
cd <directory>
pipenv install --dev
pipenv run pytest
Getting set up to contribute¶
Prerequisites¶
Pre-commit hook (for Python code formatting with black)¶
FlowKit's Python code is formatted with black (which is included in the FlowKit pipenv environments). There is also a pre-commit hook which runs black on all Python files before each commit.
To install the pre-commit hook run:
pre-commit install
pipenv run pre-commit uninstall.
The above command requires pre-commit to be installed on your system. For convenience, it is also
included in the flowmachine pipenv environment, so if you don't want to install it system-wide you
can run it as follows (it will still be installed and available for all FlowKit components, not just
flowmachine).
cd flowmachine/
pipenv run pre-commit install
Note that if you run git commit and any files are modified by the re-formatting, the pre-commit hook will abort
the commit (but leave the files re-formatted). Simply repeat the git commit command in order to complete the commit.
Example:
# 1) First attempt to commit; this is aborted because one of the files
# is reformatted by the pre-commit hook.
$ git commit -a -m "Some changes"
black...................................................Failed
hookid: black
Files were modified by this hook. Additional output:
reformatted flowmachine/__init__.py
All done! ✨ 🍰 ✨
1 file reformatted.
# 2) Complete the commit by running the same 'git commit' command again.
$ git commit -a -m "Some changes"
black...................................................Passed
[black-pre-commit-hook 8ebaace] Some changes
1 file changed, 2 insertions(+), 1 deletion(-)
Diff tool (for verifying changes in ApprovalTests-based tests)¶
Some of the tests use ApprovalTests to verify large output against
a known "approved" version (stored in files called *approved.txt). For example, the API specification is
verified in this way. If you make code changes that alter the results of these tests, the content of the
relevant *.approved.txt file needs to be updated. ApprovalTests will do this automatically for you, but it may be useful to have a
diff tool installed. (See here for some recommendations for diff tools on Mac
and Windows.) The reporters supported out of the box by ApprovalTests can be found here.
To use a specific diff reporter, you should create files named reporters.json in flowmachine/tests and integration_tests following the format shown in the ApprovalTests README.
Warning
The project root contains an env file (development_environment), in which are default values for all of the environment variables used to control and configure the FlowDB, FlowAPI, FlowMachine and FlowAuth. You will need to source this file (set -a && . ./development_environment && set +a), or otherwise set the environment variables before running any other command.
Option 1: Starting up all FlowKit components inside a dockerised development environment¶
For convenience, FlowKit comes with a dockerised development environment. You can start up a development version of all components by running:
make up
This will use the specifications in docker-compose.yml to spin up development versions of flowdb_testdata,
flowmachine, flowapi and flowauth (as well as a redis container, which is used internally by flowmachine).
The first time you run this command it will build the docker images locally, which will take ~15 minutes. Subsequent runs will be much faster because it will typically only need to (re-)start the containers, or re-build small parts of the images.
Here is an example of the running docker services after a successful make up:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
54857c5f5da5 flowminder/flowdb-testdata:latest "docker-entrypoint.s…" 5 minutes ago Up 5 minutes 8000/tcp, 0.0.0.0:9000->5432/tcp flowdb_testdata
e57c8eeb7d38 flowminder/flowmachine:latest "/bin/sh -c 'pipenv …" 5 minutes ago Up 5 minutes 0.0.0.0:5555->5555/tcp flowmachine
3d54ce0c4484 flowminder/flowapi:latest "/bin/sh -c 'pipenv …" 5 minutes ago Up 5 minutes 0.0.0.0:9090->9090/tcp flowapi
89e9575c8d42 flowminder/flowauth:latest "/entrypoint.sh /sta…" 5 minutes ago Up 5 minutes 443/tcp, 0.0.0.0:8080->80/tcp flowauth
384355e94ae6 bitnami/redis "/entrypoint.sh /run…" 5 minutes ago Up 5 minutes 0.0.0.0:6379->6379/tcp flowmachine_query_locker
Note: Currently there is a glitch which means that flowmachine may not start up correctly if flowdb_testdata
is still in the start-up phase while the flowmachine container is already up and running. This will be fixed soon but for the
time being you can check if there was an error by checking the flowmachine logs (via docker logs -f flowmachine),
and if necessary running make flowmachine-down followed by make flowmachine-up).
To make changes to the containers, you should run make down && make up.
Option 2: Starting Flowmachine and FlowAPI manually, outside the dockerised setup¶
While the fully dockerised setup described above is convenient, it has the disadvantage that interactive debugging is difficult or impossible if Flowmachine and FlowAPI are running inside a docker container. If you want to set breakpoints and use for example pdb, ipdb or similar debuggers to step through the code then it is easier to start the relevant components manually.
To do this, first start the components which you want to run inside the dockerised environment, e.g. via:
make flowmachine_testdata-up flowauth-up
Next, run one or more of the following commands (in separate terminals) to start up the remaining components.
Flowmachine¶
cd flowmachine/
pipenv run watchmedo auto-restart --recursive --patterns="*.py" --directory="." pipenv run flowmachine
FlowAPI¶
cd flowapi/
pipenv run hypercorn --debug --reload --bind 0.0.0.0:9090 "app.main:create_app()"
FlowAuth¶
cd flowauth/
pipenv run start-all
Warning
If you have started FlowMachine and FlowAPI directly, you will need to set the FLOWKIT_INTEGRATION_TESTS_DISABLE_AUTOSTART_SERVERS=TRUE before running the integration tests.
Without this setting, both will be started automatically to allow test coverage to be collected.
Verifying the setup¶
This section provides example commands which you can run to verify that flowdb, flowmachine and flowapi started up
successfully and are wired up correctly so that they can talk to each other.
Running the integration tests¶
You can run the integration tests as follows. If these pass, this is a good indication that everything is set up correctly.
cd integration_tests
pipenv install # only needed once to install dependencies
pipenv run pytest
FlowDB¶
psql "postgresql://flowdb:flowflow@localhost:9000/flowdb" -c "SELECT flowdb_version()"
flowdb_version
-------------------------------
(0.2.2-8-g0558488,2019-01-21)
(1 row)
Flowmachine¶
From within the flowmachine/ folder, run the following command to send an example message to the Flowmachine server via ZeroMQ.
pipenv run python -c "from flowmachine.core.server.utils import FM_EXAMPLE_MESSAGE, send_message_and_receive_reply; print(send_message_and_receive_reply(FM_EXAMPLE_MESSAGE))"
Sending message: {'action': 'run_query', 'query_kind': 'daily_location', 'request_id': 'DUMMY_ID', 'params': {'date': '2016-01-01', 'daily_location_method': 'last', 'aggregation_unit': 'admin3', 'subscriber_subset': 'all'}}
{'status': 'accepted', 'id': 'ddc61a04f608dee16fff0655f91c2057'}
FlowAPI¶
First navigate to http://localhost:8080/ (where Flowauth should be running), create a valid token (TODO: add
instructions how to do this or link to the appropriate section in the docs) and store it in the environment
variable TOKEN.
Then run the following command to submit a daily location calculation to FlowKit via the API:
curl -v -X POST http://localhost:9090/api/0/run \
-H "Authorization: Bearer ${TOKEN}" \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{"params": {"date":"2016-01-01", "level":"admin3", "daily_location_method":"last", "aggregation_unit":"admin3", "subscriber_subset":"all"}, "query_kind":"daily_location"}'
The output should contain a location header that similar to this:
[...]
< location: /api/0/poll/ddc61a04f608dee16fff0655f91c2057
[...]