feat(email): add Google OAuth2 for Google Workspace / .edu IMAP & SMTP

FruitPnchSamuraiG commented

2026-06-01 05:31:27 +02:00

(Migrated from github.com)

Problem

Google deprecated basic-auth (username + password) IMAP/SMTP access for all Google Workspace accounts in May 2025. This silently broke email for anyone using a .edu, company Google, or any org-managed Google account — the error is [AUTHENTICATIONFAILED] Invalid credentials with no clear explanation.

Personal @gmail.com accounts still work via App Passwords, but Workspace accounts (which includes most university emails) cannot use App Passwords either — they require OAuth2.

This affects a significant slice of users: students, researchers, and professionals whose primary email is org-managed Google.

Solution

Full Google OAuth2 (XOAUTH2) support for IMAP and SMTP. Users connect their account via a standard Google consent flow — no password ever stored.

Changes

Backend

core/database.py: oauth_provider, oauth_access_token, oauth_refresh_token, oauth_token_expiry, display_name columns on EmailAccount + idempotent SQLite migration
routes/email_helpers.py: XOAUTH2 auth in _imap_connect() and _send_smtp_message(), automatic token refresh via _refresh_google_token(), OAuth fields in _get_email_config()
routes/email_routes.py:
- GET /api/email/oauth/google/authorize — builds and redirects to Google consent URL
- GET /api/email/oauth/google/callback — exchanges code for tokens, auto-fills IMAP/SMTP settings + display name from Google userinfo
- _smtp_ready() recognises OAuth accounts as send-capable
- OAuth fields flow through the _deliver() background closure
- From: header uses display_name via email.utils.formataddr()

Frontend

static/js/settings.js: "Google Workspace / .edu" provider preset, "Connect with Google" button, success/error banner on OAuth redirect back, "Display Name" field for all accounts
static/js/document.js: _accountCanSend() treats OAuth accounts as SMTP-capable (previously fell back silently to another account)

Setup (self-hosted)

Users need to create their own Google Cloud OAuth app (one-time, ~5 min):

console.cloud.google.com → new project
APIs & Services → enable Gmail API
OAuth consent screen → External → fill app name + email → add yourself as test user
Credentials → Create → OAuth 2.0 Client ID → Web application
Authorized redirect URI: http://localhost:7000/api/email/oauth/google/callback
Add to .env:

GOOGLE_OAUTH_CLIENT_ID=your-client-id.apps.googleusercontent.com
GOOGLE_OAUTH_CLIENT_SECRET=your-secret
GOOGLE_OAUTH_REDIRECT_URI=http://localhost:7000/api/email/oauth/google/callback

docker compose up -d --build
Settings → Integrations → Add Account → Provider: Google Workspace / .edu → Connect with Google

IMAP host, SMTP host, username, and display name are auto-filled after authorization. Tokens refresh automatically.

Note for maintainers: A future improvement would be a single published Odysseus OAuth app (after Google verification) so users skip steps 1–6 entirely. The callback endpoint and DB schema are already designed for this.

Test plan

NYU .edu (Google Workspace) inbox loads via XOAUTH2 IMAP
Sending from NYU account works via XOAUTH2 SMTP
Display name auto-filled from Google userinfo appears correctly in From: header
Re-connecting (Reconnect with Google) updates tokens without creating a duplicate account
Personal Gmail with App Password still works unchanged
Token refresh path tested via _refresh_google_token() direct call

## Problem Google deprecated basic-auth (username + password) IMAP/SMTP access for all **Google Workspace** accounts in May 2025. This silently broke email for anyone using a `.edu`, company Google, or any org-managed Google account — the error is `[AUTHENTICATIONFAILED] Invalid credentials` with no clear explanation. Personal `@gmail.com` accounts still work via App Passwords, but Workspace accounts (which includes most university emails) cannot use App Passwords either — they require OAuth2. This affects a significant slice of users: students, researchers, and professionals whose primary email is org-managed Google. ## Solution Full Google OAuth2 (XOAUTH2) support for IMAP and SMTP. Users connect their account via a standard Google consent flow — no password ever stored. ## Changes ### Backend - **`core/database.py`**: `oauth_provider`, `oauth_access_token`, `oauth_refresh_token`, `oauth_token_expiry`, `display_name` columns on `EmailAccount` + idempotent SQLite migration - **`routes/email_helpers.py`**: XOAUTH2 auth in `_imap_connect()` and `_send_smtp_message()`, automatic token refresh via `_refresh_google_token()`, OAuth fields in `_get_email_config()` - **`routes/email_routes.py`**: - `GET /api/email/oauth/google/authorize` — builds and redirects to Google consent URL - `GET /api/email/oauth/google/callback` — exchanges code for tokens, auto-fills IMAP/SMTP settings + display name from Google userinfo - `_smtp_ready()` recognises OAuth accounts as send-capable - OAuth fields flow through the `_deliver()` background closure - `From:` header uses `display_name` via `email.utils.formataddr()` ### Frontend - **`static/js/settings.js`**: "Google Workspace / .edu" provider preset, "Connect with Google" button, success/error banner on OAuth redirect back, "Display Name" field for all accounts - **`static/js/document.js`**: `_accountCanSend()` treats OAuth accounts as SMTP-capable (previously fell back silently to another account) ## Setup (self-hosted) Users need to create their own Google Cloud OAuth app (one-time, ~5 min): 1. [console.cloud.google.com](https://console.cloud.google.com) → new project 2. **APIs & Services** → enable **Gmail API** 3. **OAuth consent screen** → External → fill app name + email → add yourself as test user 4. **Credentials** → Create → **OAuth 2.0 Client ID** → Web application 5. Authorized redirect URI: `http://localhost:7000/api/email/oauth/google/callback` 6. Add to `.env`: ```env GOOGLE_OAUTH_CLIENT_ID=your-client-id.apps.googleusercontent.com GOOGLE_OAUTH_CLIENT_SECRET=your-secret GOOGLE_OAUTH_REDIRECT_URI=http://localhost:7000/api/email/oauth/google/callback ``` 7. `docker compose up -d --build` 8. Settings → Integrations → Add Account → Provider: **Google Workspace / .edu** → Connect with Google IMAP host, SMTP host, username, and display name are auto-filled after authorization. Tokens refresh automatically. > **Note for maintainers:** A future improvement would be a single published Odysseus OAuth app (after Google verification) so users skip steps 1–6 entirely. The callback endpoint and DB schema are already designed for this. ## Test plan - [x] NYU `.edu` (Google Workspace) inbox loads via XOAUTH2 IMAP - [x] Sending from NYU account works via XOAUTH2 SMTP - [x] Display name auto-filled from Google userinfo appears correctly in `From:` header - [x] Re-connecting (Reconnect with Google) updates tokens without creating a duplicate account - [x] Personal Gmail with App Password still works unchanged - [x] Token refresh path tested via `_refresh_google_token()` direct call

pewdiepie-archdaemon commented

2026-06-01 06:16:45 +02:00

(Migrated from github.com)

This is a useful direction for Workspace/.edu email, but I don't think it is merge-ready yet. Main blockers I see:\n\n- OAuth state is just account_id, and the callback route does not require the logged-in user or re-check ownership before writing tokens. Please use a nonce/signed state tied to the initiating user/account and validate it in the callback before updating the row.\n- The callback currently writes OAuth tokens to db.get(EmailAccount, account_id) without owner scoping. In multi-user mode that bypasses the account ownership protections used elsewhere.\n- static/js/settings.js has smart quotes in HTML attributes around the new Name/Email/Display Name inputs (for example class=”settings-row”, id=”eaf-name”). That will break DOM lookup/event handling in browsers; these need normal ASCII quotes.\n- The added send-config/error logs include account name, SMTP host/user, owner, and raw error text. I would keep OAuth/send logs less identifying because email addresses and provider errors are sensitive.\n\nThe XOAUTH2 helper path itself looks plausible, so I would keep this PR alive, but fix the OAuth state/ownership flow and the frontend quote issue before merge.

This is a useful direction for Workspace/.edu email, but I don't think it is merge-ready yet. Main blockers I see:\n\n- OAuth `state` is just `account_id`, and the callback route does not require the logged-in user or re-check ownership before writing tokens. Please use a nonce/signed state tied to the initiating user/account and validate it in the callback before updating the row.\n- The callback currently writes OAuth tokens to `db.get(EmailAccount, account_id)` without owner scoping. In multi-user mode that bypasses the account ownership protections used elsewhere.\n- `static/js/settings.js` has smart quotes in HTML attributes around the new Name/Email/Display Name inputs (for example `class=”settings-row”`, `id=”eaf-name”`). That will break DOM lookup/event handling in browsers; these need normal ASCII quotes.\n- The added send-config/error logs include account name, SMTP host/user, owner, and raw error text. I would keep OAuth/send logs less identifying because email addresses and provider errors are sensitive.\n\nThe XOAUTH2 helper path itself looks plausible, so I would keep this PR alive, but fix the OAuth state/ownership flow and the frontend quote issue before merge.

FruitPnchSamuraiG commented

2026-06-01 07:32:02 +02:00

(Migrated from github.com)

Thank you for the thorough review; pushed a follow-up commit (d2e436d) addressing every point:

1. OAuth state — nonce/signed + validated in callback
State is now an HMAC-SHA256 token (keyed with the app secret from secret_storage) encoding account_id + owner + random nonce, verified with hmac.compare_digest in the callback before any token write. The bare account_id state is gone. Unit-tested: valid states round-trip, while tampered/forged/garbage states are all rejected.

2. Callback ownership scoping
The callback now pulls owner from the verified state and re-checks it against EmailAccount.owner before writing tokens, matching the ownership guards used elsewhere. Single-user mode (owner == "") still accepts any account, consistent with _assert_owns_account.

3. Smart quotes in settings.js
Fixed — the Name/Email/Display Name input rows now use plain ASCII quotes. Confirmed in-browser that the fields edit/save correctly (the getElementById lookups resolve now).

4. Sensitive data in logs
Stripped account name, SMTP host/user, owner, and raw provider error text from the send-config and OAuth logs. Failures now surface as generic error codes in the redirect rather than raw exception strings.

Re-verified end to end after the changes: IMAP receive and SMTP send both still work over XOAUTH2 (tested on a real Google Workspace / .edu account), and reconnecting updates tokens without creating a duplicate account.

On a personal note: it's been a journey, from watching LWIAY to the life-in-Japan transition. Thank you for all of it. I'm greatly inspired by the way you go about life, and grateful for the feedback and for keeping the PR alive. Happy to make any further changes you'd like before merge.

Thank you for the thorough review; pushed a follow-up commit (d2e436d) addressing every point: **1. OAuth state — nonce/signed + validated in callback** State is now an HMAC-SHA256 token (keyed with the app secret from `secret_storage`) encoding `account_id + owner + random nonce`, verified with `hmac.compare_digest` in the callback *before* any token write. The bare `account_id` state is gone. Unit-tested: valid states round-trip, while tampered/forged/garbage states are all rejected. **2. Callback ownership scoping** The callback now pulls `owner` from the *verified* state and re-checks it against `EmailAccount.owner` before writing tokens, matching the ownership guards used elsewhere. Single-user mode (`owner == ""`) still accepts any account, consistent with `_assert_owns_account`. **3. Smart quotes in settings.js** Fixed — the Name/Email/Display Name input rows now use plain ASCII quotes. Confirmed in-browser that the fields edit/save correctly (the `getElementById` lookups resolve now). **4. Sensitive data in logs** Stripped account name, SMTP host/user, owner, and raw provider error text from the send-config and OAuth logs. Failures now surface as generic error codes in the redirect rather than raw exception strings. Re-verified end to end after the changes: IMAP receive and SMTP send both still work over XOAUTH2 (tested on a real Google Workspace / `.edu` account), and reconnecting updates tokens without creating a duplicate account. On a personal note: it's been a journey, from watching LWIAY to the life-in-Japan transition. Thank you for all of it. I'm greatly inspired by the way you go about life, and grateful for the feedback and for keeping the PR alive. Happy to make any further changes you'd like before merge.

FruitPnchSamuraiG commented

2026-06-01 07:54:17 +02:00

(Migrated from github.com)

Pushed a couple more commits: added tests/test_email_oauth.py (14 pure-function tests covering signed-state round-trip + tamper/forgery rejection, _smtp_ready for OAuth accounts, and the XOAUTH2 SASL framing), and did a small self-review pass — consolidated the XOAUTH2 frame to a single helper (dropped an unused base64 variant + an inlined duplicate) and switched a couple of raise Exception calls to RuntimeError to match the convention in src/. All 14 new tests + the existing 28 security regressions pass, and IMAP/SMTP are re-verified live against a real Workspace account. Still happy to adjust anything further.

Pushed a couple more commits: added `tests/test_email_oauth.py` (14 pure-function tests covering signed-state round-trip + tamper/forgery rejection, `_smtp_ready` for OAuth accounts, and the XOAUTH2 SASL framing), and did a small self-review pass — consolidated the XOAUTH2 frame to a single helper (dropped an unused base64 variant + an inlined duplicate) and switched a couple of `raise Exception` calls to `RuntimeError` to match the convention in `src/`. All 14 new tests + the existing 28 security regressions pass, and IMAP/SMTP are re-verified live against a real Workspace account. Still happy to adjust anything further.

codemonkey76 commented

2026-06-01 09:26:48 +02:00

(Migrated from github.com)

FYI — I've built Microsoft 365 / Outlook support (via Microsoft Graph) on top of this branch in #334 (draft). It reuses the OAuth scaffolding added here, so #334 is stacked behind this PR. No action needed; just flagging that #237 unblocks a second email-OAuth feature. 🙂