[Feature] Documentation for development setup

## Summary

This PR improves the local development setup experience by adding Docker Compose and Makefile for streamlined setup.

## Changes

- **Added `docker-compose.yml`**: for one-command PostgreSQL database setup
- **Added `Makefile`**: Convenient shortcuts for common dev tasks (\`make setup\`, \`make dev\`, etc.)
- **Updated `README.md`**: Quick development setup instructions using Make
-
- **Added**: RDkit installation pain point documentation
- **Fixed**: Made Java feature properly dependent

## Why these changes?

The application uses PostgreSQL-specific features (\`ArrayField\`) and requires an anonymous user created by the bootstrap command. This PR makes the setup process trivial for new developers:

```bash
cp .env.local.example .env
make setup  # Starts DB, runs migrations, bootstraps data
make dev    # Starts development server
```

Java fix:
Moved global Java import to inline to avoid everyone having to configure the Java path.

Numerous changes to view and settings.
- Applied ruff-formatting

## Testing

Verified complete setup from scratch works with:
- PostgreSQL running in Docker
- All migrations applied
- Bootstrap data loaded successfully
- Anonymous user created
- The development server starts correctly.

Co-authored-by: Tobias O <tobias.olenyi@tum.de>
Co-authored-by: Tobias O <tobias.olenyi@envipath.com>
Co-authored-by: Liam <62733830+limmooo@users.noreply.github.com>
Reviewed-on: enviPath/enviPy#143
Reviewed-by: jebus <lorsbach@envipath.com>
Reviewed-by: liambrydon <lbry121@aucklanduni.ac.nz>
Co-authored-by: t03i <mail+envipath@t03i.net>
Co-committed-by: t03i <mail+envipath@t03i.net>
This commit is contained in:
2025-10-08 18:51:50 +13:00
committed by jebus
parent c2c46fbfa7
commit 36879c266b
11 changed files with 1570 additions and 836 deletions

22
.env.local.example Normal file
View File

@ -0,0 +1,22 @@
# Django settings
SECRET_KEY='a-secure-secret-key-for-development'
DEBUG=True
ALLOWED_HOSTS=*
# Database settings (using PostgreSQL for local development)
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=envipath
POSTGRES_SERVICE_NAME=localhost
POSTGRES_PORT=5432
# Celery settings
CELERY_BROKER_URL='redis://localhost:6379/0'
CELERY_RESULT_BACKEND='redis://localhost:6379/0'
FLAG_CELERY_PRESENT=False
# Other settings
LOG_LEVEL='INFO'
SERVER_URL='http://localhost:8000'
PLUGINS_ENABLED=True
EP_DATA_DIR='data'

View File

@ -16,4 +16,3 @@ POSTGRES_PORT=
# MAIL
EMAIL_HOST_USER=
EMAIL_HOST_PASSWORD=

6
.gitignore vendored
View File

@ -5,4 +5,8 @@ static/admin/
static/django_extensions/
.env
debug.log
scratches/
scratches/
data/
.DS_Store

30
.pre-commit-config.yaml Normal file
View File

@ -0,0 +1,30 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.2.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.13.3
hooks:
# Run the linter.
- id: ruff-check
types_or: [python, pyi]
args: [--fix]
# Run the formatter.
- id: ruff-format
types_or: [python, pyi]
# - repo: local
# hooks:
# - id: django-check
# name: Run Django Check
# entry: uv run python manage.py check
# language: system
# pass_filenames: false
# types: [python]

View File

@ -1,2 +1,91 @@
# enviPy
## Local Development Setup
These instructions will guide you through setting up the project for local development.
### Prerequisites
- Python 3.11 or later
- [uv](https://github.com/astral-sh/uv) - A fast Python package installer and resolver.
- **Docker and Docker Compose** - Required for running the PostgreSQL database.
- Git
> **Note:** This application requires PostgreSQL, which uses `ArrayField`. Docker is the recommended way to run PostgreSQL locally.
### 1. Install Dependencies
This project uses `uv` to manage dependencies and `poe-the-poet` for task running. First, [install `uv` if you don't have it yet](https://docs.astral.sh/uv/guides/install-python/).
Then, sync the project dependencies. This will create a virtual environment in `.venv` and install all necessary packages, including `poe-the-poet`.
```bash
uv sync --dev
```
> **Note on RDkit:** If you have a different version of rdkit installed globally, the dependency installation may fail. If this happens, please uninstall the global version and run `uv sync` again.
### 2. Set Up Environment File
Copy the example environment file for local setup:
```bash
cp .env.local.example .env
```
This file contains the necessary environment variables for local development.
### 3. Quick Setup with Poe
The easiest way to set up the development environment is by using the `poe` task runner, which is executed via `uv run`.
```bash
uv run poe setup
```
This single command will:
1. Start the PostgreSQL database using Docker Compose.
2. Run database migrations.
3. Bootstrap initial data (anonymous user, default packages, models).
After setup, start the development server:
```bash
uv run poe dev
```
The application will be available at `http://localhost:8000`.
#### Other useful Poe commands:
You can list all available commands by running `uv run poe --help`.
```bash
uv run poe db-up # Start PostgreSQL only
uv run poe db-down # Stop PostgreSQL
uv run poe migrate # Run migrations only
uv run poe bootstrap # Bootstrap data only
uv run poe shell # Open the Django shell
uv run poe clean # Remove database volumes (WARNING: destroys all data)
```
### Troubleshooting
* **Docker Connection Error:** If you see an error like `open //./pipe/dockerDesktopLinuxEngine: The system cannot find the file specified` (on Windows), it likely means your Docker Desktop application is not running. Please start Docker Desktop and try the command again.
* **SSH Keys for Git Dependencies:** Some dependencies are installed from private git repositories and require SSH authentication. Ensure your SSH keys are configured correctly for Git.
* For a general guide, see [GitHub's official documentation](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent).
* **Windows Users:** If `uv sync` hangs while fetching git dependencies, you may need to explicitly configure Git to use the Windows OpenSSH client and use the `ssh-agent` to manage your key's passphrase.
1. **Point Git to the correct SSH executable:**
```powershell
git config --global core.sshCommand "C:/Windows/System32/OpenSSH/ssh.exe"
```
2. **Enable and use the SSH agent:**
```powershell
# Run these commands in an administrator PowerShell
Get-Service ssh-agent | Set-Service -StartupType Automatic -PassThru | Start-Service
# Add your key to the agent. It will prompt for the passphrase once.
ssh-add
```

20
docker-compose.dev.yml Normal file
View File

@ -0,0 +1,20 @@
services:
db:
image: postgres:15
container_name: envipath-postgres
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: envipath
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
timeout: 5s
retries: 5
volumes:
postgres_data:

View File

@ -9,6 +9,7 @@ https://docs.djangoproject.com/en/4.2/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/4.2/ref/settings/
"""
import os
from pathlib import Path
@ -20,33 +21,35 @@ from sklearn.tree import DecisionTreeClassifier
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
load_dotenv(BASE_DIR / '.env', override=False)
load_dotenv(BASE_DIR / ".env", override=False)
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/4.2/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '7!VTW`aZqg/UBLsM.P=m)2]lWqg>{+:xUgG1"WO@bCyaHR2Up8XW&g<*3.F4l2gi9c.E3}dHyA0D`&z?u#U%^7HYbj],eP"g_MS|3BNMD[mI>s#<i/%2ngZ~Oy+/w&@]'
SECRET_KEY = os.environ.get("SECRET_KEY", "secret-key")
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = os.environ.get('DEBUG', 'False') == 'True'
ALLOWED_HOSTS = os.environ['ALLOWED_HOSTS'].split(',')
DEBUG = os.environ.get("DEBUG", "False") == "True"
ALLOWED_HOSTS = os.environ["ALLOWED_HOSTS"].split(",")
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
"django.contrib.admin",
"django.contrib.auth",
"django.contrib.contenttypes",
"django.contrib.sessions",
"django.contrib.messages",
"django.contrib.staticfiles",
# 3rd party
'django_extensions',
'oauth2_provider',
"django_extensions",
"oauth2_provider",
# Custom
'epdb',
'migration',
"epdb",
"migration",
]
AUTHENTICATION_BACKENDS = [
@ -54,42 +57,42 @@ AUTHENTICATION_BACKENDS = [
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'oauth2_provider.middleware.OAuth2TokenMiddleware',
"django.middleware.security.SecurityMiddleware",
"django.contrib.sessions.middleware.SessionMiddleware",
"django.middleware.common.CommonMiddleware",
"django.middleware.csrf.CsrfViewMiddleware",
"django.contrib.auth.middleware.AuthenticationMiddleware",
"django.contrib.messages.middleware.MessageMiddleware",
"django.middleware.clickjacking.XFrameOptionsMiddleware",
"oauth2_provider.middleware.OAuth2TokenMiddleware",
]
OAUTH2_PROVIDER = {
"PKCE_REQUIRED": False, # Accept PKCE requests but dont require them
}
if os.environ.get('REGISTRATION_MANDATORY', False) == 'True':
MIDDLEWARE.append('epdb.middleware.login_required_middleware.LoginRequiredMiddleware')
if os.environ.get("REGISTRATION_MANDATORY", False) == "True":
MIDDLEWARE.append("epdb.middleware.login_required_middleware.LoginRequiredMiddleware")
ROOT_URLCONF = 'envipath.urls'
ROOT_URLCONF = "envipath.urls"
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': (os.path.join(BASE_DIR, 'templates'),),
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
"BACKEND": "django.template.backends.django.DjangoTemplates",
"DIRS": (os.path.join(BASE_DIR, "templates"),),
"APP_DIRS": True,
"OPTIONS": {
"context_processors": [
"django.template.context_processors.debug",
"django.template.context_processors.request",
"django.contrib.auth.context_processors.auth",
"django.contrib.messages.context_processors.messages",
],
},
},
]
WSGI_APPLICATION = 'envipath.wsgi.application'
WSGI_APPLICATION = "envipath.wsgi.application"
# Database
# https://docs.djangoproject.com/en/4.2/ref/settings/#databases
@ -97,11 +100,11 @@ WSGI_APPLICATION = 'envipath.wsgi.application'
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"USER": os.environ['POSTGRES_USER'],
"NAME": os.environ['POSTGRES_DB'],
"PASSWORD": os.environ['POSTGRES_PASSWORD'],
"HOST": os.environ['POSTGRES_SERVICE_NAME'],
"PORT": os.environ['POSTGRES_PORT']
"USER": os.environ["POSTGRES_USER"],
"NAME": os.environ["POSTGRES_DB"],
"PASSWORD": os.environ["POSTGRES_PASSWORD"],
"HOST": os.environ["POSTGRES_SERVICE_NAME"],
"PORT": os.environ["POSTGRES_PORT"],
}
}
@ -109,96 +112,84 @@ DATABASES = {
# https://docs.djangoproject.com/en/4.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
{"NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator"},
{"NAME": "django.contrib.auth.password_validation.MinimumLengthValidator"},
{"NAME": "django.contrib.auth.password_validation.CommonPasswordValidator"},
{"NAME": "django.contrib.auth.password_validation.NumericPasswordValidator"},
]
# Internationalization
# https://docs.djangoproject.com/en/4.2/topics/i18n/
LANGUAGE_CODE = 'en-us'
LANGUAGE_CODE = "en-us"
TIME_ZONE = 'UTC'
TIME_ZONE = "UTC"
USE_I18N = True
USE_TZ = True
# Default primary key field type
# https://docs.djangoproject.com/en/4.2/ref/settings/#default-auto-field
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
if DEBUG:
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
else:
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_USE_TLS = True
EMAIL_HOST = 'mail.gandi.net'
EMAIL_HOST_USER = os.environ['EMAIL_HOST_USER']
EMAIL_HOST_PASSWORD = os.environ['EMAIL_HOST_PASSWORD']
EMAIL_HOST = "mail.gandi.net"
EMAIL_HOST_USER = os.environ["EMAIL_HOST_USER"]
EMAIL_HOST_PASSWORD = os.environ["EMAIL_HOST_PASSWORD"]
EMAIL_PORT = 587
AUTH_USER_MODEL = "epdb.User"
ADMIN_APPROVAL_REQUIRED = os.environ.get('ADMIN_APPROVAL_REQUIRED', 'False') == 'True'
ADMIN_APPROVAL_REQUIRED = os.environ.get("ADMIN_APPROVAL_REQUIRED", "False") == "True"
# # SESAME
# SESAME_MAX_AGE = 300
# # TODO set to "home"
# LOGIN_REDIRECT_URL = "/"
LOGIN_URL = '/login/'
LOGIN_URL = "/login/"
SERVER_URL = os.environ.get('SERVER_URL', 'http://localhost:8000')
SERVER_URL = os.environ.get("SERVER_URL", "http://localhost:8000")
CSRF_TRUSTED_ORIGINS = [SERVER_URL]
AMBIT_URL = 'http://localhost:9001'
DEFAULT_VALUES = {
'description': 'no description'
}
AMBIT_URL = "http://localhost:9001"
DEFAULT_VALUES = {"description": "no description"}
EP_DATA_DIR = os.environ['EP_DATA_DIR']
MODEL_DIR = os.path.join(EP_DATA_DIR, 'models')
EP_DATA_DIR = os.environ["EP_DATA_DIR"]
if not os.path.exists(EP_DATA_DIR):
os.mkdir(EP_DATA_DIR)
MODEL_DIR = os.path.join(EP_DATA_DIR, "models")
if not os.path.exists(MODEL_DIR):
os.mkdir(MODEL_DIR)
STATIC_DIR = os.path.join(EP_DATA_DIR, 'static')
STATIC_DIR = os.path.join(EP_DATA_DIR, "static")
if not os.path.exists(STATIC_DIR):
os.mkdir(STATIC_DIR)
LOG_DIR = os.path.join(EP_DATA_DIR, 'log')
LOG_DIR = os.path.join(EP_DATA_DIR, "log")
if not os.path.exists(LOG_DIR):
os.mkdir(LOG_DIR)
PLUGIN_DIR = os.path.join(EP_DATA_DIR, 'plugins')
PLUGIN_DIR = os.path.join(EP_DATA_DIR, "plugins")
if not os.path.exists(PLUGIN_DIR):
os.mkdir(PLUGIN_DIR)
# Set this as our static root dir
STATIC_ROOT = STATIC_DIR
STATIC_URL = '/static/'
STATIC_URL = "/static/"
# Where the sources are stored...
STATICFILES_DIRS = (
BASE_DIR / 'static',
)
STATICFILES_DIRS = (BASE_DIR / "static",)
FIXTURE_DIRS = (
BASE_DIR / 'fixtures',
)
FIXTURE_DIRS = (BASE_DIR / "fixtures",)
# Logging
LOGGING = {
@ -206,8 +197,8 @@ LOGGING = {
"disable_existing_loggers": True,
"formatters": {
"simple": {
'format': '[%(asctime)s] %(levelname)s %(module)s - %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S',
"format": "[%(asctime)s] %(levelname)s %(module)s - %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S",
},
},
"handlers": {
@ -220,7 +211,7 @@ LOGGING = {
"level": "DEBUG", # Or higher
"class": "logging.FileHandler",
"filename": os.path.join(LOG_DIR, "debug.log"),
"formatter": "simple"
"formatter": "simple",
},
},
"loggers": {
@ -228,66 +219,66 @@ LOGGING = {
"epdb": {
"handlers": ["file"], # "console",
"propagate": True,
"level": os.environ.get('LOG_LEVEL', 'INFO')
"level": os.environ.get("LOG_LEVEL", "INFO"),
},
# For everything under envipath/ loaded via getlogger(__name__)
'envipath': {
'handlers': ['file', 'console'],
'propagate': True,
'level': os.environ.get('LOG_LEVEL', 'INFO')
"envipath": {
"handlers": ["file", "console"],
"propagate": True,
"level": os.environ.get("LOG_LEVEL", "INFO"),
},
# For everything under utilities/ loaded via getlogger(__name__)
'utilities': {
'handlers': ['file', 'console'],
'propagate': True,
'level': os.environ.get('LOG_LEVEL', 'INFO')
"utilities": {
"handlers": ["file", "console"],
"propagate": True,
"level": os.environ.get("LOG_LEVEL", "INFO"),
},
},
}
# Flags
ENVIFORMER_PRESENT = os.environ.get('ENVIFORMER_PRESENT', 'False') == 'True'
ENVIFORMER_DEVICE = os.environ.get('ENVIFORMER_DEVICE', 'cpu')
ENVIFORMER_PRESENT = os.environ.get("ENVIFORMER_PRESENT", "False") == "True"
ENVIFORMER_DEVICE = os.environ.get("ENVIFORMER_DEVICE", "cpu")
# If celery is not present set always eager to true which will cause delayed tasks to block until finished
FLAG_CELERY_PRESENT = os.environ.get('FLAG_CELERY_PRESENT', 'False') == 'True'
FLAG_CELERY_PRESENT = os.environ.get("FLAG_CELERY_PRESENT", "False") == "True"
if not FLAG_CELERY_PRESENT:
CELERY_TASK_ALWAYS_EAGER = True
# Celery Configuration Options
CELERY_TIMEZONE = "Europe/Berlin"
# Celery Configuration
CELERY_BROKER_URL = 'redis://localhost:6379/0' # Use Redis as message broker
CELERY_RESULT_BACKEND = 'redis://localhost:6379/1'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_BROKER_URL = "redis://localhost:6379/0" # Use Redis as message broker
CELERY_RESULT_BACKEND = "redis://localhost:6379/1"
CELERY_ACCEPT_CONTENT = ["json"]
CELERY_TASK_SERIALIZER = "json"
MODEL_BUILDING_ENABLED = os.environ.get('MODEL_BUILDING_ENABLED', 'False') == 'True'
APPLICABILITY_DOMAIN_ENABLED = os.environ.get('APPLICABILITY_DOMAIN_ENABLED', 'False') == 'True'
MODEL_BUILDING_ENABLED = os.environ.get("MODEL_BUILDING_ENABLED", "False") == "True"
APPLICABILITY_DOMAIN_ENABLED = os.environ.get("APPLICABILITY_DOMAIN_ENABLED", "False") == "True"
DEFAULT_RF_MODEL_PARAMS = {
'base_clf': RandomForestClassifier(
"base_clf": RandomForestClassifier(
n_estimators=100,
max_features='log2',
max_features="log2",
random_state=42,
criterion='entropy',
criterion="entropy",
ccp_alpha=0.0,
max_depth=3,
min_samples_leaf=1
min_samples_leaf=1,
),
'num_chains': 10,
"num_chains": 10,
}
DEFAULT_MODEL_PARAMS = {
'base_clf': DecisionTreeClassifier(
criterion='entropy',
"base_clf": DecisionTreeClassifier(
criterion="entropy",
max_depth=3,
min_samples_split=5,
# min_samples_leaf=5,
max_features='sqrt',
max_features="sqrt",
# class_weight='balanced',
random_state=42
random_state=42,
),
'num_chains': 10,
"num_chains": 10,
}
DEFAULT_MAX_NUMBER_OF_NODES = 30
@ -295,9 +286,10 @@ DEFAULT_MAX_DEPTH = 5
DEFAULT_MODEL_THRESHOLD = 0.25
# Loading Plugins
PLUGINS_ENABLED = os.environ.get('PLUGINS_ENABLED', 'False') == 'True'
PLUGINS_ENABLED = os.environ.get("PLUGINS_ENABLED", "False") == "True"
if PLUGINS_ENABLED:
from utilities.plugin import discover_plugins
CLASSIFIER_PLUGINS = discover_plugins(_cls=Classifier)
PROPERTY_PLUGINS = discover_plugins(_cls=Property)
DESCRIPTOR_PLUGINS = discover_plugins(_cls=Descriptor)
@ -306,59 +298,59 @@ else:
PROPERTY_PLUGINS = {}
DESCRIPTOR_PLUGINS = {}
SENTRY_ENABLED = os.environ.get('SENTRY_ENABLED', 'False') == 'True'
SENTRY_ENABLED = os.environ.get("SENTRY_ENABLED", "False") == "True"
if SENTRY_ENABLED:
import sentry_sdk
def before_send(event, hint):
# Check if was a handled exception by one of our loggers
if event.get('logger'):
for log_path in LOGGING.get('loggers').keys():
if event['logger'].startswith(log_path):
if event.get("logger"):
for log_path in LOGGING.get("loggers").keys():
if event["logger"].startswith(log_path):
return None
return event
sentry_sdk.init(
dsn=os.environ.get('SENTRY_DSN'),
dsn=os.environ.get("SENTRY_DSN"),
# Add data like request headers and IP for users,
# see https://docs.sentry.io/platforms/python/data-management/data-collected/ for more info
send_default_pii=True,
environment=os.environ.get('SENTRY_ENVIRONMENT', 'development'),
environment=os.environ.get("SENTRY_ENVIRONMENT", "development"),
before_send=before_send,
)
# compile into digestible flags
FLAGS = {
'MODEL_BUILDING': MODEL_BUILDING_ENABLED,
'CELERY': FLAG_CELERY_PRESENT,
'PLUGINS': PLUGINS_ENABLED,
'SENTRY': SENTRY_ENABLED,
'ENVIFORMER': ENVIFORMER_PRESENT,
'APPLICABILITY_DOMAIN': APPLICABILITY_DOMAIN_ENABLED,
"MODEL_BUILDING": MODEL_BUILDING_ENABLED,
"CELERY": FLAG_CELERY_PRESENT,
"PLUGINS": PLUGINS_ENABLED,
"SENTRY": SENTRY_ENABLED,
"ENVIFORMER": ENVIFORMER_PRESENT,
"APPLICABILITY_DOMAIN": APPLICABILITY_DOMAIN_ENABLED,
}
# path of the URL are checked via "startswith"
# -> /password_reset/done is covered as well
LOGIN_EXEMPT_URLS = [
'/register',
'/api/legacy/',
'/o/token/',
'/o/userinfo/',
'/password_reset/',
'/reset/',
'/microsoft/',
"/register",
"/api/legacy/",
"/o/token/",
"/o/userinfo/",
"/password_reset/",
"/reset/",
"/microsoft/",
]
# MS AD/Entra
MS_ENTRA_ENABLED = os.environ.get('MS_ENTRA_ENABLED', 'False') == 'True'
MS_ENTRA_ENABLED = os.environ.get("MS_ENTRA_ENABLED", "False") == "True"
if MS_ENTRA_ENABLED:
# Add app to installed apps
INSTALLED_APPS.append('epauth')
INSTALLED_APPS.append("epauth")
# Set vars required by app
MS_ENTRA_CLIENT_ID = os.environ['MS_CLIENT_ID']
MS_ENTRA_CLIENT_SECRET = os.environ['MS_CLIENT_SECRET']
MS_ENTRA_TENANT_ID = os.environ['MS_TENANT_ID']
MS_ENTRA_CLIENT_ID = os.environ["MS_CLIENT_ID"]
MS_ENTRA_CLIENT_SECRET = os.environ["MS_CLIENT_SECRET"]
MS_ENTRA_TENANT_ID = os.environ["MS_TENANT_ID"]
MS_ENTRA_AUTHORITY = f"https://login.microsoftonline.com/{MS_ENTRA_TENANT_ID}"
MS_ENTRA_REDIRECT_URI = os.environ['MS_REDIRECT_URI']
MS_ENTRA_SCOPES = os.environ.get('MS_SCOPES', '').split(',')
MS_ENTRA_REDIRECT_URI = os.environ["MS_REDIRECT_URI"]
MS_ENTRA_SCOPES = os.environ.get("MS_SCOPES", "").split(",")

View File

@ -151,7 +151,7 @@ class Command(BaseCommand):
mapping = {}
for p in packages:
print(f"Importing {p}...")
package_data = json.loads(open(s.BASE_DIR / 'fixtures' / 'packages' / '2025-07-18' / p).read())
package_data = json.loads(open(s.BASE_DIR / 'fixtures' / 'packages' / '2025-07-18' / p, encoding='utf-8').read())
imported_package = self.import_package(package_data, admin)
mapping[p.replace('.json', '')] = imported_package

View File

@ -12,7 +12,8 @@ from epdb.logic import PackageManager
from epdb.models import Rule, SimpleAmbitRule, Package, CompoundStructure
from epdb.views import get_base_context, _anonymous_or_real
from utilities.chem import FormatConverter
from envipy_ambit import apply
from rdkit import Chem
from rdkit.Chem.MolStandardize import rdMolStandardize
@ -31,11 +32,19 @@ def normalize_smiles(smiles):
def run_both_engines(SMILES, SMIRKS):
from envipy_ambit import apply
ambit_res = apply(SMIRKS, SMILES)
# ambit_res, ambit_errors = FormatConverter.sanitize_smiles([str(s) for s in ambit_res])
ambit_res = list(set([normalize_smiles(str(x)) for x in
FormatConverter.sanitize_smiles([str(s) for s in ambit_res])[0]]))
ambit_res = list(
set(
[
normalize_smiles(str(x))
for x in FormatConverter.sanitize_smiles([str(s) for s in ambit_res])[0]
]
)
)
products = FormatConverter.apply(SMILES, SMIRKS)
@ -46,22 +55,39 @@ def run_both_engines(SMILES, SMIRKS):
all_rdkit_prods = list(set(all_rdkit_prods))
# all_rdkit_res, rdkit_errors = FormatConverter.sanitize_smiles(all_rdkit_prods)
all_rdkit_res = list(set([normalize_smiles(str(x)) for x in
FormatConverter.sanitize_smiles([str(s) for s in all_rdkit_prods])[0]]))
all_rdkit_res = list(
set(
[
normalize_smiles(str(x))
for x in FormatConverter.sanitize_smiles(
[str(s) for s in all_rdkit_prods]
)[0]
]
)
)
# return ambit_res, ambit_errors, all_rdkit_res, rdkit_errors
return ambit_res, 0, all_rdkit_res, 0
def migration(request):
if request.method == 'GET':
if request.method == "GET":
context = get_base_context(request)
if os.path.exists(s.BASE_DIR / 'fixtures' / 'migration_status_per_rule.json') and request.GET.get(
"force") is None:
migration_status = json.load(open(s.BASE_DIR / 'fixtures' / 'migration_status_per_rule.json'))
if (
os.path.exists(s.BASE_DIR / "fixtures" / "migration_status_per_rule.json")
and request.GET.get("force") is None
):
migration_status = json.load(
open(s.BASE_DIR / "fixtures" / "migration_status_per_rule.json")
)
else:
BBD = Package.objects.get(url='http://localhost:8000/package/32de3cf4-e3e6-4168-956e-32fa5ddb0ce1')
ALL_SMILES = [cs.smiles for cs in CompoundStructure.objects.filter(compound__package=BBD)]
BBD = Package.objects.get(
url="http://localhost:8000/package/32de3cf4-e3e6-4168-956e-32fa5ddb0ce1"
)
ALL_SMILES = [
cs.smiles
for cs in CompoundStructure.objects.filter(compound__package=BBD)
]
RULES = SimpleAmbitRule.objects.filter(package=BBD)
results = list()
@ -71,7 +97,7 @@ def migration(request):
total = 0
for i, r in enumerate(RULES):
logger.debug(f'\r{i + 1:03d}/{num_rules}')
logger.debug(f"\r{i + 1:03d}/{num_rules}")
res = True
for smiles in ALL_SMILES:
try:
@ -81,13 +107,19 @@ def migration(request):
except Exception as e:
logger.error(e)
results.append({
'name': r.name,
'detail_url': s.SERVER_URL + '/migration/' + r.url.replace('https://envipath.org/', '').replace('http://localhost:8000/', ''),
'id': str(r.uuid),
'url': r.url,
'status': res,
})
results.append(
{
"name": r.name,
"detail_url": s.SERVER_URL
+ "/migration/"
+ r.url.replace("https://envipath.org/", "").replace(
"http://localhost:8000/", ""
),
"id": str(r.uuid),
"url": r.url,
"status": res,
}
)
if res:
success += 1
@ -95,32 +127,37 @@ def migration(request):
error += 1
total += 1
results = sorted(results, key=lambda x: (x['status'], x['name']))
results = sorted(results, key=lambda x: (x["status"], x["name"]))
migration_status = {
'results': results,
'success': success,
'error': error,
'total': total
"results": results,
"success": success,
"error": error,
"total": total,
}
json.dump(migration_status, open(s.BASE_DIR / 'fixtures' / 'migration_status_per_rule.json', 'w'))
json.dump(
migration_status,
open(s.BASE_DIR / "fixtures" / "migration_status_per_rule.json", "w"),
)
for r in migration_status['results']:
r['detail_url'] = r['detail_url'].replace('http://localhost:8000', s.SERVER_URL)
for r in migration_status["results"]:
r["detail_url"] = r["detail_url"].replace(
"http://localhost:8000", s.SERVER_URL
)
context.update(**migration_status)
return render(request, 'migration.html', context)
return render(request, "migration.html", context)
def migration_detail(request, package_uuid, rule_uuid):
current_user = _anonymous_or_real(request)
if request.method == 'GET':
if request.method == "GET":
context = get_base_context(request)
BBD = Package.objects.get(name='EAWAG-BBD')
BBD = Package.objects.get(name="EAWAG-BBD")
STRUCTURES = CompoundStructure.objects.filter(compound__package=BBD)
rule = Rule.objects.get(package=BBD, uuid=rule_uuid)
@ -132,8 +169,9 @@ def migration_detail(request, package_uuid, rule_uuid):
all_prods = set()
for structure in STRUCTURES:
ambit_smiles, ambit_errors, rdkit_smiles, rdkit_errors = run_both_engines(structure.smiles, smirks)
ambit_smiles, ambit_errors, rdkit_smiles, rdkit_errors = run_both_engines(
structure.smiles, smirks
)
for x in ambit_smiles:
all_prods.add(x)
@ -152,13 +190,13 @@ def migration_detail(request, package_uuid, rule_uuid):
if len(ambit_smiles) or len(rdkit_smiles):
temp = {
'url': structure.url,
'id': str(structure.uuid),
'name': structure.name,
'initial_smiles': structure.smiles,
'ambit_smiles': sorted(list(ambit_smiles)),
'rdkit_smiles': sorted(list(rdkit_smiles)),
'status': set(ambit_smiles) == set(rdkit_smiles),
"url": structure.url,
"id": str(structure.uuid),
"name": structure.name,
"initial_smiles": structure.smiles,
"ambit_smiles": sorted(list(ambit_smiles)),
"rdkit_smiles": sorted(list(rdkit_smiles)),
"status": set(ambit_smiles) == set(rdkit_smiles),
}
detail = f"""
BT: {bt_rule_name}
@ -177,29 +215,32 @@ def migration_detail(request, package_uuid, rule_uuid):
rdkit_errors: {rdkit_errors}
"""
temp['detail'] = '\n'.join([x.strip() for x in detail.split('\n')])
temp["detail"] = "\n".join([x.strip() for x in detail.split("\n")])
results.append(temp)
res &= partial_res
results = sorted(results, key=lambda x: x['status'])
context['results'] = results
context['res'] = res
context['bt_rule_name'] = bt_rule_name
return render(request, 'migration_detail.html', context)
results = sorted(results, key=lambda x: x["status"])
context["results"] = results
context["res"] = res
context["bt_rule_name"] = bt_rule_name
return render(request, "migration_detail.html", context)
def compare(request):
context = get_base_context(request)
if request.method == 'GET':
context[
"smirks"] = "[#1,#6:6][#7;X3;!$(NC1CC1)!$([N][C]=O)!$([!#8]CNC=O):1]([#1,#6:7])[#6;A;X4:2][H:3]>>[#1,#6:6][#7;X3:1]([#1,#6:7])[H:3].[#6;A:2]=O"
context["smiles"] = "C(CC(=O)N[C@@H](CS[Se-])C(=O)NCC(=O)[O-])[C@@H](C(=O)[O-])N"
return render(request, 'compare.html', context)
if request.method == "GET":
context["smirks"] = (
"[#1,#6:6][#7;X3;!$(NC1CC1)!$([N][C]=O)!$([!#8]CNC=O):1]([#1,#6:7])[#6;A;X4:2][H:3]>>[#1,#6:6][#7;X3:1]([#1,#6:7])[H:3].[#6;A:2]=O"
)
context["smiles"] = (
"C(CC(=O)N[C@@H](CS[Se-])C(=O)NCC(=O)[O-])[C@@H](C(=O)[O-])N"
)
return render(request, "compare.html", context)
elif request.method == 'POST':
elif request.method == "POST":
smiles = request.POST.get("smiles")
smirks = request.POST.get("smirks")
@ -219,9 +260,9 @@ def compare(request):
rdkit_res, _ = FormatConverter.sanitize_smiles(all_rdkit_prods)
context["result"] = True
context['ambit_res'] = sorted(set(ambit_res))
context['rdkit_res'] = sorted(set(rdkit_res))
context['diff'] = sorted(set(ambit_res).difference(set(rdkit_res)))
context["ambit_res"] = sorted(set(ambit_res))
context["rdkit_res"] = sorted(set(rdkit_res))
context["diff"] = sorted(set(ambit_res).difference(set(rdkit_res)))
context["smirks"] = smirks
context["smiles"] = smiles
@ -230,7 +271,7 @@ def compare(request):
if r.exists():
context["rule"] = r.first()
return render(request, 'compare.html', context)
return render(request, "compare.html", context)
else:
return HttpResponseNotAllowed(['GET', 'POST'])
return HttpResponseNotAllowed(["GET", "POST"])

View File

@ -38,3 +38,53 @@ envipy-ambit = { git = "ssh://git@git.envipath.com/enviPath/enviPy-ambit.git" }
[project.optional-dependencies]
ms-login = ["msal>=1.33.0"]
dev = [
"poethepoet>=0.37.0",
"pre-commit>=4.3.0",
"ruff>=0.13.3",
]
[tool.ruff]
line-length = 100
[tool.ruff.lint]
# Allow fix for all enabled rules (when `--fix`) is provided.
fixable = ["ALL"]
# Allow unused variables when underscore-prefixed.
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
[tool.ruff.format]
docstring-code-format = true
# 4. Ignore `E402` (import violations) in all `__init__.py` files, and in selected subdirectories.
[tool.ruff.lint.per-file-ignores]
"__init__.py" = ["E402"]
"**/{tests,docs,tools}/*" = ["E402"]
[tool.poe.tasks]
# Main tasks
setup = { sequence = ["db-up", "migrate", "bootstrap"], help = "Complete setup: start database, run migrations, and bootstrap data" }
dev = { cmd = "python manage.py runserver", help = "Start the development server", deps = ["db-up"] }
# Database tasks
db-up = { cmd = "docker compose -f docker-compose.dev.yml up -d", help = "Start PostgreSQL database using Docker Compose" }
db-down = { cmd = "docker compose -f docker-compose.dev.yml down", help = "Stop PostgreSQL database" }
# Full cleanup tasks
clean = { sequence = ["clean-db"], help = "Remove model files and database volumes (WARNING: destroys all data!)" }
clean-db = { cmd = "docker compose -f docker-compose.dev.yml down -v", help = "Removes the database container and volume." }
# Django tasks
migrate = { cmd = "python manage.py migrate", help = "Run database migrations" }
bootstrap = { shell = """
echo "Bootstrapping initial data..."
echo "This will take a bit . Get yourself some coffee..."
python manage.py bootstrap
echo " Bootstrap complete"
echo ""
echo "Default admin credentials:"
echo " Username: admin"
echo " Email: admin@envipath.com"
echo " Password: SuperSafe"
""", help = "Bootstrap initial data (anonymous user, packages, models)" }
shell = { cmd = "python manage.py shell", help = "Open Django shell" }

1763
uv.lock generated

File diff suppressed because it is too large Load Diff