jebus/enviPy-bayer
1
0
Fork 0
forked from enviPath/enviPy
Code Issues Pull Requests Actions Releases Activity

[Feature] Documentation for development setup

Browse Source 
## Summary

This PR improves the local development setup experience by adding Docker Compose and Makefile for streamlined setup.

## Changes

- **Added `docker-compose.yml`**: for one-command PostgreSQL database setup
- **Added `Makefile`**: Convenient shortcuts for common dev tasks (\`make setup\`, \`make dev\`, etc.)
- **Updated `README.md`**: Quick development setup instructions using Make
-
- **Added**: RDkit installation pain point documentation
- **Fixed**: Made Java feature properly dependent

## Why these changes?

The application uses PostgreSQL-specific features (\`ArrayField\`) and requires an anonymous user created by the bootstrap command. This PR makes the setup process trivial for new developers:

```bash
cp .env.local.example .env
make setup  # Starts DB, runs migrations, bootstraps data
make dev    # Starts development server
```

Java fix:
Moved global Java import to inline to avoid everyone having to configure the Java path.

Numerous changes to view and settings.
- Applied ruff-formatting

## Testing

Verified complete setup from scratch works with:
- PostgreSQL running in Docker
- All migrations applied
- Bootstrap data loaded successfully
- Anonymous user created
- The development server starts correctly.

Co-authored-by: Tobias O <tobias.olenyi@tum.de>
Co-authored-by: Tobias O <tobias.olenyi@envipath.com>
Co-authored-by: Liam <62733830+limmooo@users.noreply.github.com>
Reviewed-on: enviPath/enviPy#143
Reviewed-by: jebus <lorsbach@envipath.com>
Reviewed-by: liambrydon <lbry121@aucklanduni.ac.nz>
Co-authored-by: t03i <mail+envipath@t03i.net>
Co-committed-by: t03i <mail+envipath@t03i.net>
This commit is contained in:
Tobias O
2025-10-08 18:51:50 +13:00
committed by jebus
parent c2c46fbfa7
commit 36879c266b
11 changed files with 1570 additions and 836 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
.env.local.example Normal file
View File

@ -0,0 +1,22 @@
# Django settings
SECRET_KEY='a-secure-secret-key-for-development'
DEBUG=True
ALLOWED_HOSTS=*
# Database settings (using PostgreSQL for local development)
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=envipath
POSTGRES_SERVICE_NAME=localhost
POSTGRES_PORT=5432
# Celery settings
CELERY_BROKER_URL='redis://localhost:6379/0'
CELERY_RESULT_BACKEND='redis://localhost:6379/0'
FLAG_CELERY_PRESENT=False
# Other settings
LOG_LEVEL='INFO'
SERVER_URL='http://localhost:8000'
PLUGINS_ENABLED=True
EP_DATA_DIR='data'

1
.env-example → .env.prod.example
View File

@ -16,4 +16,3 @@ POSTGRES_PORT=
# MAIL # MAIL
EMAIL_HOST_USER= EMAIL_HOST_USER=
EMAIL_HOST_PASSWORD= EMAIL_HOST_PASSWORD=

6
.gitignore vendored
View File

@ -5,4 +5,8 @@ static/admin/
static/django_extensions/ static/django_extensions/
.env .env
debug.log debug.log
scratches/ scratches/
data/
.DS_Store

30
.pre-commit-config.yaml Normal file
View File

@ -0,0 +1,30 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.2.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.13.3
hooks:
# Run the linter.
- id: ruff-check
types_or: [python, pyi]
args: [--fix]
# Run the formatter.
- id: ruff-format
types_or: [python, pyi]
# - repo: local
# hooks:
# - id: django-check
# name: Run Django Check
# entry: uv run python manage.py check
# language: system
# pass_filenames: false
# types: [python]

89
README.md
View File

@ -1,2 +1,91 @@
# enviPy # enviPy
## Local Development Setup
These instructions will guide you through setting up the project for local development.
### Prerequisites
- Python 3.11 or later
- [uv](https://github.com/astral-sh/uv) - A fast Python package installer and resolver.
- **Docker and Docker Compose** - Required for running the PostgreSQL database.
- Git
> **Note:** This application requires PostgreSQL, which uses `ArrayField`. Docker is the recommended way to run PostgreSQL locally.
### 1. Install Dependencies
This project uses `uv` to manage dependencies and `poe-the-poet` for task running. First, [install `uv` if you don't have it yet](https://docs.astral.sh/uv/guides/install-python/).
Then, sync the project dependencies. This will create a virtual environment in `.venv` and install all necessary packages, including `poe-the-poet`.
```bash
uv sync --dev
```
> **Note on RDkit:** If you have a different version of rdkit installed globally, the dependency installation may fail. If this happens, please uninstall the global version and run `uv sync` again.
### 2. Set Up Environment File
Copy the example environment file for local setup:
```bash
cp .env.local.example .env
```
This file contains the necessary environment variables for local development.
### 3. Quick Setup with Poe
The easiest way to set up the development environment is by using the `poe` task runner, which is executed via `uv run`.
```bash
uv run poe setup
```
This single command will:
1. Start the PostgreSQL database using Docker Compose.
2. Run database migrations.
3. Bootstrap initial data (anonymous user, default packages, models).
After setup, start the development server:
```bash
uv run poe dev
```
The application will be available at `http://localhost:8000`.
#### Other useful Poe commands:
You can list all available commands by running `uv run poe --help`.
```bash
uv run poe db-up # Start PostgreSQL only
uv run poe db-down # Stop PostgreSQL
uv run poe migrate # Run migrations only
uv run poe bootstrap # Bootstrap data only
uv run poe shell # Open the Django shell
uv run poe clean # Remove database volumes (WARNING: destroys all data)
```
### Troubleshooting
* **Docker Connection Error:** If you see an error like `open //./pipe/dockerDesktopLinuxEngine: The system cannot find the file specified` (on Windows), it likely means your Docker Desktop application is not running. Please start Docker Desktop and try the command again.
* **SSH Keys for Git Dependencies:** Some dependencies are installed from private git repositories and require SSH authentication. Ensure your SSH keys are configured correctly for Git.
* For a general guide, see [GitHub's official documentation](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent).
* **Windows Users:** If `uv sync` hangs while fetching git dependencies, you may need to explicitly configure Git to use the Windows OpenSSH client and use the `ssh-agent` to manage your key's passphrase.
1. **Point Git to the correct SSH executable:**
```powershell
git config --global core.sshCommand "C:/Windows/System32/OpenSSH/ssh.exe"
```
2. **Enable and use the SSH agent:**
```powershell
# Run these commands in an administrator PowerShell
Get-Service ssh-agent | Set-Service -StartupType Automatic -PassThru | Start-Service
# Add your key to the agent. It will prompt for the passphrase once.
ssh-add
```

20
docker-compose.dev.yml Normal file
View File

@ -0,0 +1,20 @@
services:
db:
image: postgres:15
container_name: envipath-postgres
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: envipath
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
timeout: 5s
retries: 5
volumes:
postgres_data:

268
envipath/settings.py
View File

@ -9,6 +9,7 @@ https://docs.djangoproject.com/en/4.2/topics/settings/
For the full list of settings and their values, see For the full list of settings and their values, see
https://docs.djangoproject.com/en/4.2/ref/settings/ https://docs.djangoproject.com/en/4.2/ref/settings/
""" """
import os import os
from pathlib import Path from pathlib import Path
@ -20,33 +21,35 @@ from sklearn.tree import DecisionTreeClassifier
# Build paths inside the project like this: BASE_DIR / 'subdir'. # Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent BASE_DIR = Path(__file__).resolve().parent.parent
load_dotenv(BASE_DIR / '.env', override=False) load_dotenv(BASE_DIR / ".env", override=False)
# Quick-start development settings - unsuitable for production # Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/4.2/howto/deployment/checklist/ # See https://docs.djangoproject.com/en/4.2/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret! # SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '7!VTW`aZqg/UBLsM.P=m)2]lWqg>{+:xUgG1"WO@bCyaHR2Up8XW&g<*3.F4l2gi9c.E3}dHyA0D`&z?u#U%^7HYbj],eP"g_MS|3BNMD[mI>s#<i/%2ngZ~Oy+/w&@]' SECRET_KEY = os.environ.get("SECRET_KEY", "secret-key")
# SECURITY WARNING: don't run with debug turned on in production! # SECURITY WARNING: don't run with debug turned on in production!
DEBUG = os.environ.get('DEBUG', 'False') == 'True' DEBUG = os.environ.get("DEBUG", "False") == "True"
ALLOWED_HOSTS = os.environ['ALLOWED_HOSTS'].split(',')
ALLOWED_HOSTS = os.environ["ALLOWED_HOSTS"].split(",")
# Application definition # Application definition
INSTALLED_APPS = [ INSTALLED_APPS = [
'django.contrib.admin', "django.contrib.admin",
'django.contrib.auth', "django.contrib.auth",
'django.contrib.contenttypes', "django.contrib.contenttypes",
'django.contrib.sessions', "django.contrib.sessions",
'django.contrib.messages', "django.contrib.messages",
'django.contrib.staticfiles', "django.contrib.staticfiles",
# 3rd party # 3rd party
'django_extensions', "django_extensions",
'oauth2_provider', "oauth2_provider",
# Custom # Custom
'epdb', "epdb",
'migration', "migration",
] ]
AUTHENTICATION_BACKENDS = [ AUTHENTICATION_BACKENDS = [
@ -54,42 +57,42 @@ AUTHENTICATION_BACKENDS = [
] ]
MIDDLEWARE = [ MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware', "django.middleware.security.SecurityMiddleware",
'django.contrib.sessions.middleware.SessionMiddleware', "django.contrib.sessions.middleware.SessionMiddleware",
'django.middleware.common.CommonMiddleware', "django.middleware.common.CommonMiddleware",
'django.middleware.csrf.CsrfViewMiddleware', "django.middleware.csrf.CsrfViewMiddleware",
'django.contrib.auth.middleware.AuthenticationMiddleware', "django.contrib.auth.middleware.AuthenticationMiddleware",
'django.contrib.messages.middleware.MessageMiddleware', "django.contrib.messages.middleware.MessageMiddleware",
'django.middleware.clickjacking.XFrameOptionsMiddleware', "django.middleware.clickjacking.XFrameOptionsMiddleware",
'oauth2_provider.middleware.OAuth2TokenMiddleware', "oauth2_provider.middleware.OAuth2TokenMiddleware",
] ]
OAUTH2_PROVIDER = { OAUTH2_PROVIDER = {
"PKCE_REQUIRED": False, # Accept PKCE requests but dont require them "PKCE_REQUIRED": False, # Accept PKCE requests but dont require them
} }
if os.environ.get('REGISTRATION_MANDATORY', False) == 'True': if os.environ.get("REGISTRATION_MANDATORY", False) == "True":
MIDDLEWARE.append('epdb.middleware.login_required_middleware.LoginRequiredMiddleware') MIDDLEWARE.append("epdb.middleware.login_required_middleware.LoginRequiredMiddleware")
ROOT_URLCONF = 'envipath.urls' ROOT_URLCONF = "envipath.urls"
TEMPLATES = [ TEMPLATES = [
{ {
'BACKEND': 'django.template.backends.django.DjangoTemplates', "BACKEND": "django.template.backends.django.DjangoTemplates",
'DIRS': (os.path.join(BASE_DIR, 'templates'),), "DIRS": (os.path.join(BASE_DIR, "templates"),),
'APP_DIRS': True, "APP_DIRS": True,
'OPTIONS': { "OPTIONS": {
'context_processors': [ "context_processors": [
'django.template.context_processors.debug', "django.template.context_processors.debug",
'django.template.context_processors.request', "django.template.context_processors.request",
'django.contrib.auth.context_processors.auth', "django.contrib.auth.context_processors.auth",
'django.contrib.messages.context_processors.messages', "django.contrib.messages.context_processors.messages",
], ],
}, },
}, },
] ]
WSGI_APPLICATION = 'envipath.wsgi.application' WSGI_APPLICATION = "envipath.wsgi.application"
# Database # Database
# https://docs.djangoproject.com/en/4.2/ref/settings/#databases # https://docs.djangoproject.com/en/4.2/ref/settings/#databases
@ -97,11 +100,11 @@ WSGI_APPLICATION = 'envipath.wsgi.application'
DATABASES = { DATABASES = {
"default": { "default": {
"ENGINE": "django.db.backends.postgresql", "ENGINE": "django.db.backends.postgresql",
"USER": os.environ['POSTGRES_USER'], "USER": os.environ["POSTGRES_USER"],
"NAME": os.environ['POSTGRES_DB'], "NAME": os.environ["POSTGRES_DB"],
"PASSWORD": os.environ['POSTGRES_PASSWORD'], "PASSWORD": os.environ["POSTGRES_PASSWORD"],
"HOST": os.environ['POSTGRES_SERVICE_NAME'], "HOST": os.environ["POSTGRES_SERVICE_NAME"],
"PORT": os.environ['POSTGRES_PORT'] "PORT": os.environ["POSTGRES_PORT"],
} }
} }
@ -109,96 +112,84 @@ DATABASES = {
# https://docs.djangoproject.com/en/4.2/ref/settings/#auth-password-validators # https://docs.djangoproject.com/en/4.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [ AUTH_PASSWORD_VALIDATORS = [
{ {"NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator"},
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', {"NAME": "django.contrib.auth.password_validation.MinimumLengthValidator"},
}, {"NAME": "django.contrib.auth.password_validation.CommonPasswordValidator"},
{ {"NAME": "django.contrib.auth.password_validation.NumericPasswordValidator"},
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
] ]
# Internationalization # Internationalization
# https://docs.djangoproject.com/en/4.2/topics/i18n/ # https://docs.djangoproject.com/en/4.2/topics/i18n/
LANGUAGE_CODE = 'en-us' LANGUAGE_CODE = "en-us"
TIME_ZONE = 'UTC' TIME_ZONE = "UTC"
USE_I18N = True USE_I18N = True
USE_TZ = True USE_TZ = True
# Default primary key field type # Default primary key field type
# https://docs.djangoproject.com/en/4.2/ref/settings/#default-auto-field # https://docs.djangoproject.com/en/4.2/ref/settings/#default-auto-field
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField' DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
if DEBUG: if DEBUG:
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend' EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
else: else:
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_USE_TLS = True EMAIL_USE_TLS = True
EMAIL_HOST = 'mail.gandi.net' EMAIL_HOST = "mail.gandi.net"
EMAIL_HOST_USER = os.environ['EMAIL_HOST_USER'] EMAIL_HOST_USER = os.environ["EMAIL_HOST_USER"]
EMAIL_HOST_PASSWORD = os.environ['EMAIL_HOST_PASSWORD'] EMAIL_HOST_PASSWORD = os.environ["EMAIL_HOST_PASSWORD"]
EMAIL_PORT = 587 EMAIL_PORT = 587
AUTH_USER_MODEL = "epdb.User" AUTH_USER_MODEL = "epdb.User"
ADMIN_APPROVAL_REQUIRED = os.environ.get('ADMIN_APPROVAL_REQUIRED', 'False') == 'True' ADMIN_APPROVAL_REQUIRED = os.environ.get("ADMIN_APPROVAL_REQUIRED", "False") == "True"
# # SESAME # # SESAME
# SESAME_MAX_AGE = 300 # SESAME_MAX_AGE = 300
# # TODO set to "home" # # TODO set to "home"
# LOGIN_REDIRECT_URL = "/" # LOGIN_REDIRECT_URL = "/"
LOGIN_URL = '/login/' LOGIN_URL = "/login/"
SERVER_URL = os.environ.get('SERVER_URL', 'http://localhost:8000') SERVER_URL = os.environ.get("SERVER_URL", "http://localhost:8000")
CSRF_TRUSTED_ORIGINS = [SERVER_URL] CSRF_TRUSTED_ORIGINS = [SERVER_URL]
AMBIT_URL = 'http://localhost:9001' AMBIT_URL = "http://localhost:9001"
DEFAULT_VALUES = { DEFAULT_VALUES = {"description": "no description"}
'description': 'no description'
}
EP_DATA_DIR = os.environ['EP_DATA_DIR'] EP_DATA_DIR = os.environ["EP_DATA_DIR"]
MODEL_DIR = os.path.join(EP_DATA_DIR, 'models') if not os.path.exists(EP_DATA_DIR):
os.mkdir(EP_DATA_DIR)
MODEL_DIR = os.path.join(EP_DATA_DIR, "models")
if not os.path.exists(MODEL_DIR): if not os.path.exists(MODEL_DIR):
os.mkdir(MODEL_DIR) os.mkdir(MODEL_DIR)
STATIC_DIR = os.path.join(EP_DATA_DIR, 'static') STATIC_DIR = os.path.join(EP_DATA_DIR, "static")
if not os.path.exists(STATIC_DIR): if not os.path.exists(STATIC_DIR):
os.mkdir(STATIC_DIR) os.mkdir(STATIC_DIR)
LOG_DIR = os.path.join(EP_DATA_DIR, 'log') LOG_DIR = os.path.join(EP_DATA_DIR, "log")
if not os.path.exists(LOG_DIR): if not os.path.exists(LOG_DIR):
os.mkdir(LOG_DIR) os.mkdir(LOG_DIR)
PLUGIN_DIR = os.path.join(EP_DATA_DIR, 'plugins') PLUGIN_DIR = os.path.join(EP_DATA_DIR, "plugins")
if not os.path.exists(PLUGIN_DIR): if not os.path.exists(PLUGIN_DIR):
os.mkdir(PLUGIN_DIR) os.mkdir(PLUGIN_DIR)
# Set this as our static root dir # Set this as our static root dir
STATIC_ROOT = STATIC_DIR STATIC_ROOT = STATIC_DIR
STATIC_URL = '/static/' STATIC_URL = "/static/"
# Where the sources are stored... # Where the sources are stored...
STATICFILES_DIRS = ( STATICFILES_DIRS = (BASE_DIR / "static",)
BASE_DIR / 'static',
)
FIXTURE_DIRS = ( FIXTURE_DIRS = (BASE_DIR / "fixtures",)
BASE_DIR / 'fixtures',
)
# Logging # Logging
LOGGING = { LOGGING = {
@ -206,8 +197,8 @@ LOGGING = {
"disable_existing_loggers": True, "disable_existing_loggers": True,
"formatters": { "formatters": {
"simple": { "simple": {
'format': '[%(asctime)s] %(levelname)s %(module)s - %(message)s', "format": "[%(asctime)s] %(levelname)s %(module)s - %(message)s",
'datefmt': '%Y-%m-%d %H:%M:%S', "datefmt": "%Y-%m-%d %H:%M:%S",
}, },
}, },
"handlers": { "handlers": {
@ -220,7 +211,7 @@ LOGGING = {
"level": "DEBUG", # Or higher "level": "DEBUG", # Or higher
"class": "logging.FileHandler", "class": "logging.FileHandler",
"filename": os.path.join(LOG_DIR, "debug.log"), "filename": os.path.join(LOG_DIR, "debug.log"),
"formatter": "simple" "formatter": "simple",
}, },
}, },
"loggers": { "loggers": {
@ -228,66 +219,66 @@ LOGGING = {
"epdb": { "epdb": {
"handlers": ["file"], # "console", "handlers": ["file"], # "console",
"propagate": True, "propagate": True,
"level": os.environ.get('LOG_LEVEL', 'INFO') "level": os.environ.get("LOG_LEVEL", "INFO"),
}, },
# For everything under envipath/ loaded via getlogger(__name__) # For everything under envipath/ loaded via getlogger(__name__)
'envipath': { "envipath": {
'handlers': ['file', 'console'], "handlers": ["file", "console"],
'propagate': True, "propagate": True,
'level': os.environ.get('LOG_LEVEL', 'INFO') "level": os.environ.get("LOG_LEVEL", "INFO"),
}, },
# For everything under utilities/ loaded via getlogger(__name__) # For everything under utilities/ loaded via getlogger(__name__)
'utilities': { "utilities": {
'handlers': ['file', 'console'], "handlers": ["file", "console"],
'propagate': True, "propagate": True,
'level': os.environ.get('LOG_LEVEL', 'INFO') "level": os.environ.get("LOG_LEVEL", "INFO"),
}, },
}, },
} }
# Flags # Flags
ENVIFORMER_PRESENT = os.environ.get('ENVIFORMER_PRESENT', 'False') == 'True' ENVIFORMER_PRESENT = os.environ.get("ENVIFORMER_PRESENT", "False") == "True"
ENVIFORMER_DEVICE = os.environ.get('ENVIFORMER_DEVICE', 'cpu') ENVIFORMER_DEVICE = os.environ.get("ENVIFORMER_DEVICE", "cpu")
# If celery is not present set always eager to true which will cause delayed tasks to block until finished # If celery is not present set always eager to true which will cause delayed tasks to block until finished
FLAG_CELERY_PRESENT = os.environ.get('FLAG_CELERY_PRESENT', 'False') == 'True' FLAG_CELERY_PRESENT = os.environ.get("FLAG_CELERY_PRESENT", "False") == "True"
if not FLAG_CELERY_PRESENT: if not FLAG_CELERY_PRESENT:
CELERY_TASK_ALWAYS_EAGER = True CELERY_TASK_ALWAYS_EAGER = True
# Celery Configuration Options # Celery Configuration Options
CELERY_TIMEZONE = "Europe/Berlin" CELERY_TIMEZONE = "Europe/Berlin"
# Celery Configuration # Celery Configuration
CELERY_BROKER_URL = 'redis://localhost:6379/0' # Use Redis as message broker CELERY_BROKER_URL = "redis://localhost:6379/0" # Use Redis as message broker
CELERY_RESULT_BACKEND = 'redis://localhost:6379/1' CELERY_RESULT_BACKEND = "redis://localhost:6379/1"
CELERY_ACCEPT_CONTENT = ['json'] CELERY_ACCEPT_CONTENT = ["json"]
CELERY_TASK_SERIALIZER = 'json' CELERY_TASK_SERIALIZER = "json"
MODEL_BUILDING_ENABLED = os.environ.get('MODEL_BUILDING_ENABLED', 'False') == 'True' MODEL_BUILDING_ENABLED = os.environ.get("MODEL_BUILDING_ENABLED", "False") == "True"
APPLICABILITY_DOMAIN_ENABLED = os.environ.get('APPLICABILITY_DOMAIN_ENABLED', 'False') == 'True' APPLICABILITY_DOMAIN_ENABLED = os.environ.get("APPLICABILITY_DOMAIN_ENABLED", "False") == "True"
DEFAULT_RF_MODEL_PARAMS = { DEFAULT_RF_MODEL_PARAMS = {
'base_clf': RandomForestClassifier( "base_clf": RandomForestClassifier(
n_estimators=100, n_estimators=100,
max_features='log2', max_features="log2",
random_state=42, random_state=42,
criterion='entropy', criterion="entropy",
ccp_alpha=0.0, ccp_alpha=0.0,
max_depth=3, max_depth=3,
min_samples_leaf=1 min_samples_leaf=1,
), ),
'num_chains': 10, "num_chains": 10,
} }
DEFAULT_MODEL_PARAMS = { DEFAULT_MODEL_PARAMS = {
'base_clf': DecisionTreeClassifier( "base_clf": DecisionTreeClassifier(
criterion='entropy', criterion="entropy",
max_depth=3, max_depth=3,
min_samples_split=5, min_samples_split=5,
# min_samples_leaf=5, # min_samples_leaf=5,
max_features='sqrt', max_features="sqrt",
# class_weight='balanced', # class_weight='balanced',
random_state=42 random_state=42,
), ),
'num_chains': 10, "num_chains": 10,
} }
DEFAULT_MAX_NUMBER_OF_NODES = 30 DEFAULT_MAX_NUMBER_OF_NODES = 30
@ -295,9 +286,10 @@ DEFAULT_MAX_DEPTH = 5
DEFAULT_MODEL_THRESHOLD = 0.25 DEFAULT_MODEL_THRESHOLD = 0.25
# Loading Plugins # Loading Plugins
PLUGINS_ENABLED = os.environ.get('PLUGINS_ENABLED', 'False') == 'True' PLUGINS_ENABLED = os.environ.get("PLUGINS_ENABLED", "False") == "True"
if PLUGINS_ENABLED: if PLUGINS_ENABLED:
from utilities.plugin import discover_plugins from utilities.plugin import discover_plugins
CLASSIFIER_PLUGINS = discover_plugins(_cls=Classifier) CLASSIFIER_PLUGINS = discover_plugins(_cls=Classifier)
PROPERTY_PLUGINS = discover_plugins(_cls=Property) PROPERTY_PLUGINS = discover_plugins(_cls=Property)
DESCRIPTOR_PLUGINS = discover_plugins(_cls=Descriptor) DESCRIPTOR_PLUGINS = discover_plugins(_cls=Descriptor)
@ -306,59 +298,59 @@ else:
PROPERTY_PLUGINS = {} PROPERTY_PLUGINS = {}
DESCRIPTOR_PLUGINS = {} DESCRIPTOR_PLUGINS = {}
SENTRY_ENABLED = os.environ.get('SENTRY_ENABLED', 'False') == 'True' SENTRY_ENABLED = os.environ.get("SENTRY_ENABLED", "False") == "True"
if SENTRY_ENABLED: if SENTRY_ENABLED:
import sentry_sdk import sentry_sdk
def before_send(event, hint): def before_send(event, hint):
# Check if was a handled exception by one of our loggers # Check if was a handled exception by one of our loggers
if event.get('logger'): if event.get("logger"):
for log_path in LOGGING.get('loggers').keys(): for log_path in LOGGING.get("loggers").keys():
if event['logger'].startswith(log_path): if event["logger"].startswith(log_path):
return None return None
return event return event
sentry_sdk.init( sentry_sdk.init(
dsn=os.environ.get('SENTRY_DSN'), dsn=os.environ.get("SENTRY_DSN"),
# Add data like request headers and IP for users, # Add data like request headers and IP for users,
# see https://docs.sentry.io/platforms/python/data-management/data-collected/ for more info # see https://docs.sentry.io/platforms/python/data-management/data-collected/ for more info
send_default_pii=True, send_default_pii=True,
environment=os.environ.get('SENTRY_ENVIRONMENT', 'development'), environment=os.environ.get("SENTRY_ENVIRONMENT", "development"),
before_send=before_send, before_send=before_send,
) )
# compile into digestible flags # compile into digestible flags
FLAGS = { FLAGS = {
'MODEL_BUILDING': MODEL_BUILDING_ENABLED, "MODEL_BUILDING": MODEL_BUILDING_ENABLED,
'CELERY': FLAG_CELERY_PRESENT, "CELERY": FLAG_CELERY_PRESENT,
'PLUGINS': PLUGINS_ENABLED, "PLUGINS": PLUGINS_ENABLED,
'SENTRY': SENTRY_ENABLED, "SENTRY": SENTRY_ENABLED,
'ENVIFORMER': ENVIFORMER_PRESENT, "ENVIFORMER": ENVIFORMER_PRESENT,
'APPLICABILITY_DOMAIN': APPLICABILITY_DOMAIN_ENABLED, "APPLICABILITY_DOMAIN": APPLICABILITY_DOMAIN_ENABLED,
} }
# path of the URL are checked via "startswith" # path of the URL are checked via "startswith"
# -> /password_reset/done is covered as well # -> /password_reset/done is covered as well
LOGIN_EXEMPT_URLS = [ LOGIN_EXEMPT_URLS = [
'/register', "/register",
'/api/legacy/', "/api/legacy/",
'/o/token/', "/o/token/",
'/o/userinfo/', "/o/userinfo/",
'/password_reset/', "/password_reset/",
'/reset/', "/reset/",
'/microsoft/', "/microsoft/",
] ]
# MS AD/Entra # MS AD/Entra
MS_ENTRA_ENABLED = os.environ.get('MS_ENTRA_ENABLED', 'False') == 'True' MS_ENTRA_ENABLED = os.environ.get("MS_ENTRA_ENABLED", "False") == "True"
if MS_ENTRA_ENABLED: if MS_ENTRA_ENABLED:
# Add app to installed apps # Add app to installed apps
INSTALLED_APPS.append('epauth') INSTALLED_APPS.append("epauth")
# Set vars required by app # Set vars required by app
MS_ENTRA_CLIENT_ID = os.environ['MS_CLIENT_ID'] MS_ENTRA_CLIENT_ID = os.environ["MS_CLIENT_ID"]
MS_ENTRA_CLIENT_SECRET = os.environ['MS_CLIENT_SECRET'] MS_ENTRA_CLIENT_SECRET = os.environ["MS_CLIENT_SECRET"]
MS_ENTRA_TENANT_ID = os.environ['MS_TENANT_ID'] MS_ENTRA_TENANT_ID = os.environ["MS_TENANT_ID"]
MS_ENTRA_AUTHORITY = f"https://login.microsoftonline.com/{MS_ENTRA_TENANT_ID}" MS_ENTRA_AUTHORITY = f"https://login.microsoftonline.com/{MS_ENTRA_TENANT_ID}"
MS_ENTRA_REDIRECT_URI = os.environ['MS_REDIRECT_URI'] MS_ENTRA_REDIRECT_URI = os.environ["MS_REDIRECT_URI"]
MS_ENTRA_SCOPES = os.environ.get('MS_SCOPES', '').split(',') MS_ENTRA_SCOPES = os.environ.get("MS_SCOPES", "").split(",")

2
epdb/management/commands/bootstrap.py
View File

@ -151,7 +151,7 @@ class Command(BaseCommand):
mapping = {} mapping = {}
for p in packages: for p in packages:
print(f"Importing {p}...") print(f"Importing {p}...")
package_data = json.loads(open(s.BASE_DIR / 'fixtures' / 'packages' / '2025-07-18' / p).read()) package_data = json.loads(open(s.BASE_DIR / 'fixtures' / 'packages' / '2025-07-18' / p, encoding='utf-8').read())
imported_package = self.import_package(package_data, admin) imported_package = self.import_package(package_data, admin)
mapping[p.replace('.json', '')] = imported_package mapping[p.replace('.json', '')] = imported_package

155
migration/views.py
View File

@ -12,7 +12,8 @@ from epdb.logic import PackageManager
from epdb.models import Rule, SimpleAmbitRule, Package, CompoundStructure from epdb.models import Rule, SimpleAmbitRule, Package, CompoundStructure
from epdb.views import get_base_context, _anonymous_or_real from epdb.views import get_base_context, _anonymous_or_real
from utilities.chem import FormatConverter from utilities.chem import FormatConverter
from envipy_ambit import apply
from rdkit import Chem from rdkit import Chem
from rdkit.Chem.MolStandardize import rdMolStandardize from rdkit.Chem.MolStandardize import rdMolStandardize
@ -31,11 +32,19 @@ def normalize_smiles(smiles):
def run_both_engines(SMILES, SMIRKS): def run_both_engines(SMILES, SMIRKS):
from envipy_ambit import apply
ambit_res = apply(SMIRKS, SMILES) ambit_res = apply(SMIRKS, SMILES)
# ambit_res, ambit_errors = FormatConverter.sanitize_smiles([str(s) for s in ambit_res]) # ambit_res, ambit_errors = FormatConverter.sanitize_smiles([str(s) for s in ambit_res])
ambit_res = list(set([normalize_smiles(str(x)) for x in ambit_res = list(
FormatConverter.sanitize_smiles([str(s) for s in ambit_res])[0]])) set(
[
normalize_smiles(str(x))
for x in FormatConverter.sanitize_smiles([str(s) for s in ambit_res])[0]
]
)
)
products = FormatConverter.apply(SMILES, SMIRKS) products = FormatConverter.apply(SMILES, SMIRKS)
@ -46,22 +55,39 @@ def run_both_engines(SMILES, SMIRKS):
all_rdkit_prods = list(set(all_rdkit_prods)) all_rdkit_prods = list(set(all_rdkit_prods))
# all_rdkit_res, rdkit_errors = FormatConverter.sanitize_smiles(all_rdkit_prods) # all_rdkit_res, rdkit_errors = FormatConverter.sanitize_smiles(all_rdkit_prods)
all_rdkit_res = list(set([normalize_smiles(str(x)) for x in all_rdkit_res = list(
FormatConverter.sanitize_smiles([str(s) for s in all_rdkit_prods])[0]])) set(
[
normalize_smiles(str(x))
for x in FormatConverter.sanitize_smiles(
[str(s) for s in all_rdkit_prods]
)[0]
]
)
)
# return ambit_res, ambit_errors, all_rdkit_res, rdkit_errors # return ambit_res, ambit_errors, all_rdkit_res, rdkit_errors
return ambit_res, 0, all_rdkit_res, 0 return ambit_res, 0, all_rdkit_res, 0
def migration(request): def migration(request):
if request.method == 'GET': if request.method == "GET":
context = get_base_context(request) context = get_base_context(request)
if os.path.exists(s.BASE_DIR / 'fixtures' / 'migration_status_per_rule.json') and request.GET.get( if (
"force") is None: os.path.exists(s.BASE_DIR / "fixtures" / "migration_status_per_rule.json")
migration_status = json.load(open(s.BASE_DIR / 'fixtures' / 'migration_status_per_rule.json')) and request.GET.get("force") is None
):
migration_status = json.load(
open(s.BASE_DIR / "fixtures" / "migration_status_per_rule.json")
)
else: else:
BBD = Package.objects.get(
BBD = Package.objects.get(url='http://localhost:8000/package/32de3cf4-e3e6-4168-956e-32fa5ddb0ce1') url="http://localhost:8000/package/32de3cf4-e3e6-4168-956e-32fa5ddb0ce1"
ALL_SMILES = [cs.smiles for cs in CompoundStructure.objects.filter(compound__package=BBD)] )
ALL_SMILES = [
cs.smiles
for cs in CompoundStructure.objects.filter(compound__package=BBD)
]
RULES = SimpleAmbitRule.objects.filter(package=BBD) RULES = SimpleAmbitRule.objects.filter(package=BBD)
results = list() results = list()
@ -71,7 +97,7 @@ def migration(request):
total = 0 total = 0
for i, r in enumerate(RULES): for i, r in enumerate(RULES):
logger.debug(f'\r{i + 1:03d}/{num_rules}') logger.debug(f"\r{i + 1:03d}/{num_rules}")
res = True res = True
for smiles in ALL_SMILES: for smiles in ALL_SMILES:
try: try:
@ -81,13 +107,19 @@ def migration(request):
except Exception as e: except Exception as e:
logger.error(e) logger.error(e)
results.append({ results.append(
'name': r.name, {
'detail_url': s.SERVER_URL + '/migration/' + r.url.replace('https://envipath.org/', '').replace('http://localhost:8000/', ''), "name": r.name,
'id': str(r.uuid), "detail_url": s.SERVER_URL
'url': r.url, + "/migration/"
'status': res, + r.url.replace("https://envipath.org/", "").replace(
}) "http://localhost:8000/", ""
),
"id": str(r.uuid),
"url": r.url,
"status": res,
}
)
if res: if res:
success += 1 success += 1
@ -95,32 +127,37 @@ def migration(request):
error += 1 error += 1
total += 1 total += 1
results = sorted(results, key=lambda x: (x['status'], x['name'])) results = sorted(results, key=lambda x: (x["status"], x["name"]))
migration_status = { migration_status = {
'results': results, "results": results,
'success': success, "success": success,
'error': error, "error": error,
'total': total "total": total,
} }
json.dump(migration_status, open(s.BASE_DIR / 'fixtures' / 'migration_status_per_rule.json', 'w')) json.dump(
migration_status,
open(s.BASE_DIR / "fixtures" / "migration_status_per_rule.json", "w"),
)
for r in migration_status['results']: for r in migration_status["results"]:
r['detail_url'] = r['detail_url'].replace('http://localhost:8000', s.SERVER_URL) r["detail_url"] = r["detail_url"].replace(
"http://localhost:8000", s.SERVER_URL
)
context.update(**migration_status) context.update(**migration_status)
return render(request, 'migration.html', context) return render(request, "migration.html", context)
def migration_detail(request, package_uuid, rule_uuid): def migration_detail(request, package_uuid, rule_uuid):
current_user = _anonymous_or_real(request) current_user = _anonymous_or_real(request)
if request.method == 'GET': if request.method == "GET":
context = get_base_context(request) context = get_base_context(request)
BBD = Package.objects.get(name='EAWAG-BBD') BBD = Package.objects.get(name="EAWAG-BBD")
STRUCTURES = CompoundStructure.objects.filter(compound__package=BBD) STRUCTURES = CompoundStructure.objects.filter(compound__package=BBD)
rule = Rule.objects.get(package=BBD, uuid=rule_uuid) rule = Rule.objects.get(package=BBD, uuid=rule_uuid)
@ -132,8 +169,9 @@ def migration_detail(request, package_uuid, rule_uuid):
all_prods = set() all_prods = set()
for structure in STRUCTURES: for structure in STRUCTURES:
ambit_smiles, ambit_errors, rdkit_smiles, rdkit_errors = run_both_engines(
ambit_smiles, ambit_errors, rdkit_smiles, rdkit_errors = run_both_engines(structure.smiles, smirks) structure.smiles, smirks
)
for x in ambit_smiles: for x in ambit_smiles:
all_prods.add(x) all_prods.add(x)
@ -152,13 +190,13 @@ def migration_detail(request, package_uuid, rule_uuid):
if len(ambit_smiles) or len(rdkit_smiles): if len(ambit_smiles) or len(rdkit_smiles):
temp = { temp = {
'url': structure.url, "url": structure.url,
'id': str(structure.uuid), "id": str(structure.uuid),
'name': structure.name, "name": structure.name,
'initial_smiles': structure.smiles, "initial_smiles": structure.smiles,
'ambit_smiles': sorted(list(ambit_smiles)), "ambit_smiles": sorted(list(ambit_smiles)),
'rdkit_smiles': sorted(list(rdkit_smiles)), "rdkit_smiles": sorted(list(rdkit_smiles)),
'status': set(ambit_smiles) == set(rdkit_smiles), "status": set(ambit_smiles) == set(rdkit_smiles),
} }
detail = f""" detail = f"""
BT: {bt_rule_name} BT: {bt_rule_name}
@ -177,29 +215,32 @@ def migration_detail(request, package_uuid, rule_uuid):
rdkit_errors: {rdkit_errors} rdkit_errors: {rdkit_errors}
""" """
temp['detail'] = '\n'.join([x.strip() for x in detail.split('\n')]) temp["detail"] = "\n".join([x.strip() for x in detail.split("\n")])
results.append(temp) results.append(temp)
res &= partial_res res &= partial_res
results = sorted(results, key=lambda x: x['status']) results = sorted(results, key=lambda x: x["status"])
context['results'] = results context["results"] = results
context['res'] = res context["res"] = res
context['bt_rule_name'] = bt_rule_name context["bt_rule_name"] = bt_rule_name
return render(request, 'migration_detail.html', context) return render(request, "migration_detail.html", context)
def compare(request): def compare(request):
context = get_base_context(request) context = get_base_context(request)
if request.method == 'GET': if request.method == "GET":
context[ context["smirks"] = (
"smirks"] = "[#1,#6:6][#7;X3;!$(NC1CC1)!$([N][C]=O)!$([!#8]CNC=O):1]([#1,#6:7])[#6;A;X4:2][H:3]>>[#1,#6:6][#7;X3:1]([#1,#6:7])[H:3].[#6;A:2]=O" "[#1,#6:6][#7;X3;!$(NC1CC1)!$([N][C]=O)!$([!#8]CNC=O):1]([#1,#6:7])[#6;A;X4:2][H:3]>>[#1,#6:6][#7;X3:1]([#1,#6:7])[H:3].[#6;A:2]=O"
context["smiles"] = "C(CC(=O)N[C@@H](CS[Se-])C(=O)NCC(=O)[O-])[C@@H](C(=O)[O-])N" )
return render(request, 'compare.html', context) context["smiles"] = (
"C(CC(=O)N[C@@H](CS[Se-])C(=O)NCC(=O)[O-])[C@@H](C(=O)[O-])N"
)
return render(request, "compare.html", context)
elif request.method == 'POST': elif request.method == "POST":
smiles = request.POST.get("smiles") smiles = request.POST.get("smiles")
smirks = request.POST.get("smirks") smirks = request.POST.get("smirks")
@ -219,9 +260,9 @@ def compare(request):
rdkit_res, _ = FormatConverter.sanitize_smiles(all_rdkit_prods) rdkit_res, _ = FormatConverter.sanitize_smiles(all_rdkit_prods)
context["result"] = True context["result"] = True
context['ambit_res'] = sorted(set(ambit_res)) context["ambit_res"] = sorted(set(ambit_res))
context['rdkit_res'] = sorted(set(rdkit_res)) context["rdkit_res"] = sorted(set(rdkit_res))
context['diff'] = sorted(set(ambit_res).difference(set(rdkit_res))) context["diff"] = sorted(set(ambit_res).difference(set(rdkit_res)))
context["smirks"] = smirks context["smirks"] = smirks
context["smiles"] = smiles context["smiles"] = smiles
@ -230,7 +271,7 @@ def compare(request):
if r.exists(): if r.exists():
context["rule"] = r.first() context["rule"] = r.first()
return render(request, 'compare.html', context) return render(request, "compare.html", context)
else: else:
return HttpResponseNotAllowed(['GET', 'POST']) return HttpResponseNotAllowed(["GET", "POST"])

50
pyproject.toml
View File

@ -38,3 +38,53 @@ envipy-ambit = { git = "ssh://git@git.envipath.com/enviPath/enviPy-ambit.git" }
[project.optional-dependencies] [project.optional-dependencies]
ms-login = ["msal>=1.33.0"] ms-login = ["msal>=1.33.0"]
dev = [
"poethepoet>=0.37.0",
"pre-commit>=4.3.0",
"ruff>=0.13.3",
]
[tool.ruff]
line-length = 100
[tool.ruff.lint]
# Allow fix for all enabled rules (when `--fix`) is provided.
fixable = ["ALL"]
# Allow unused variables when underscore-prefixed.
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
[tool.ruff.format]
docstring-code-format = true
# 4. Ignore `E402` (import violations) in all `__init__.py` files, and in selected subdirectories.
[tool.ruff.lint.per-file-ignores]
"__init__.py" = ["E402"]
"**/{tests,docs,tools}/*" = ["E402"]
[tool.poe.tasks]
# Main tasks
setup = { sequence = ["db-up", "migrate", "bootstrap"], help = "Complete setup: start database, run migrations, and bootstrap data" }
dev = { cmd = "python manage.py runserver", help = "Start the development server", deps = ["db-up"] }
# Database tasks
db-up = { cmd = "docker compose -f docker-compose.dev.yml up -d", help = "Start PostgreSQL database using Docker Compose" }
db-down = { cmd = "docker compose -f docker-compose.dev.yml down", help = "Stop PostgreSQL database" }
# Full cleanup tasks
clean = { sequence = ["clean-db"], help = "Remove model files and database volumes (WARNING: destroys all data!)" }
clean-db = { cmd = "docker compose -f docker-compose.dev.yml down -v", help = "Removes the database container and volume." }
# Django tasks
migrate = { cmd = "python manage.py migrate", help = "Run database migrations" }
bootstrap = { shell = """
echo "Bootstrapping initial data..."
echo "This will take a bit ⏱️. Get yourself some coffee..."
python manage.py bootstrap
echo "✓ Bootstrap complete"
echo ""
echo "Default admin credentials:"
echo " Username: admin"
echo " Email: admin@envipath.com"
echo " Password: SuperSafe"
""", help = "Bootstrap initial data (anonymous user, packages, models)" }
shell = { cmd = "python manage.py shell", help = "Open Django shell" }

1763
uv.lock generated
View File

File diff suppressed because it is too large Load Diff