# NextIT MeetAI

## Hướng Dẫn Cài Đặt Nhanh

Hệ thống chạy tối thiểu với 3 service: `backend`, `server`, và `frontend`.
Trước khi chạy `backend`, cần có MongoDB đang chạy và `MONGO_URI` trong
`backend/.env` trỏ đúng tới database, ví dụ `mongodb://localhost:27017/NextIT-MeetAI`.

### Backend

```bash
cd backend

conda create -n meetai-backend python=3.11 -y
conda activate meetai-backend

cp .env.example .env
```

Cập nhật các biến môi trường cần thiết trong `backend/.env`:

```env
MONGO_URI=mongodb://localhost:27017/NextIT-MeetAI
MODEL_SERVER_BASE_URL=http://localhost:8002
MODEL_SERVER_API_KEY=change-me-model-server-key

LLM_BASE_URL=https://openrouter.ai/api/v1
LLM_API_KEY=your-openrouter-api-key

GOOGLE_STITCH_API_KEY=your-google-stitch-api-key
GOOGLE_STITCH_ACCESS_TOKEN=
GOOGLE_STITCH_CLOUD_PROJECT=
```

Sau đó cài package và chạy backend:

```bash
pip install -r requirements.txt

uvicorn main:app --reload --host 0.0.0.0 --port 8000
```

Backend API docs: <http://localhost:8000/docs>

### Server

```bash
cd server

conda create -n meetai-server python=3.11 -y
conda activate meetai-server

cp .env.example .env
```

Cập nhật `BACKEND_BASE_URL` và API key trong `server/.env`:

```env
BACKEND_BASE_URL=http://localhost:8000
MODEL_SERVER_API_KEY=change-me-model-server-key
```

Nếu dùng embedding model local hoặc ASR local bằng GPU, cài PyTorch CUDA phù hợp
trước khi cài `requirements.txt`. Ví dụ CUDA 12.1:

```bash
pip install torch==2.3.1+cu121 torchaudio==2.3.1+cu121 \
  --index-url https://download.pytorch.org/whl/cu121
```

Sau đó cài package và chạy server:

```bash
pip install -r requirements.txt

uvicorn main:app --reload --host 0.0.0.0 --port 8002
```

Server API docs: <http://localhost:8002/docs>

### Frontend

```bash
cd frontend

cp .env.local.example .env.local
npm install
npm run dev
```

Frontend: <http://localhost:3005>

Trong `frontend/.env.local`, cấu hình mặc định cho 3 service local là:

```env
NEXT_PUBLIC_API_URL=http://localhost:8000
NEXT_PUBLIC_WS_URL=ws://localhost:8002
NEXT_PUBLIC_APP_URL=http://localhost:3005
```

---

Project Manager AI workspace with a split service architecture: FastAPI REST
for auth/database, a separate live ASR model server, and Google Stitch powered
project design generation. Projects own meetings, documents, contributions,
summaries, designs, and project knowledge.

> Important developer note
>
> The repo is transitioning away from several older assumptions.
> Treat these as deprecated even if some implementation work is still in flight:
> - Do not store or read transcript data from `recordings.transcript`.
>   Transcript source of truth is `recording_transcripts`.
> - Do not design new flows around `Continue` + reconnect grace window.
>   Meetings should be resumed explicitly from project/meeting context, not by
>   browser refresh recovery.
> - Designs are generated from project context through the Studio flow, not from
>   a single meeting transcript.

| Layer | Tech |
|---|---|
| Backend | FastAPI 0.115, Motor 3.6 (MongoDB), Pydantic v2 |
| Model server | FastAPI WebSocket, Qwen3-ASR, FunASR, OpenAI STT |
| Frontend | Next.js 14 App Router, Tailwind CSS 3, Lucide, Axios |
| Desktop | Electron 31 (tray, IPC, deep-link) |
| Mobile | Capacitor 6 (Android foreground service, iOS background audio) |
| Database | MongoDB 7 (Docker) |

---

## Prerequisites

| Tool | Version | Notes |
|---|---|---|
| Python | 3.11 | Conda recommended |
| Node.js | 20+ | |
| Docker + Compose | v2 | MongoDB only |
| CUDA toolkit | 12.1+ | Optional, for local GPU ASR models in `server/` |
| RTK CLI | latest | `cargo install rtk` or from [rtk-ai/rtk](https://github.com/rtk-ai/rtk) |
| Android Studio | Ladybug+ | For mobile builds |

---

## 1 · Clone & environment setup

```bash
git clone https://github.com/nextit/meetai.git
cd NextIT.MeetAI

# Copy and fill in environment variables per service
cp backend/.env.example backend/.env
cp server/.env.example server/.env
cp frontend/.env.local.example frontend/.env.local
# Edit MODEL_SERVER_API_KEY in backend/.env and server/.env to the same value.
```

### Generate JWT secret (if not already set)
```bash
openssl rand -hex 32
# Paste the output as JWT_SECRET in backend/.env
```

---

## 2 · Start MongoDB

```bash
docker compose up -d
# MongoDB → localhost:27017
# Mongo Express UI → http://localhost:8081  (admin / meetai_admin)
```

---

## 3 · Backend (Conda)

```bash
conda create -n meetai python=3.11 -y
conda activate meetai

cd backend

pip install -r requirements.txt

# Run dev server
uvicorn main:app --reload --host 0.0.0.0 --port 8000
# API docs → http://localhost:8000/docs
```

## 4 · Model Server (Live ASR)

```bash
cd server

# If you use local GPU ASR models, install the matching PyTorch build first.
# Example:
pip install torch==2.3.1+cu121 torchaudio==2.3.1+cu121 \
    --index-url https://download.pytorch.org/whl/cu121

pip install -r requirements.txt

uvicorn main:app --reload --host 0.0.0.0 --port 8002
# Model server docs → http://localhost:8002/docs
```

Qwen3-ASR's OpenAI-compatible endpoint still needs to be started manually before
selecting `Qwen3-ASR 1.7B`.

## 5 · Frontend (Next.js)

```bash
cd frontend
npm install
npm run dev          # http://localhost:3005
```

### Mobile static export (for Capacitor)
```bash
npm run build:mobile # sets NEXT_EXPORT=1 → writes to frontend/out/
```

---

## 6 · Electron desktop

```bash
cd electron
npm install

# Requires frontend running on port 3005 first
npm run dev          # opens BrowserWindow loading http://localhost:3005
```

### Production build
```bash
npm run build:linux  # → electron/dist/*.AppImage
npm run build:win    # → electron/dist/*.exe (NSIS)
```

Place icon files in `electron/assets/` before building — see [electron/assets/README.md](electron/assets/README.md).

---

## 6 · Mobile (Capacitor)

```bash
cd mobile
npm install

# First-time setup (builds web, adds platforms, patches manifest, syncs)
CAPACITOR_DEV_URL=http://<your-LAN-IP>:3005 npm run init

# After initial setup, just sync on each change
npm run sync:android

# Open in IDE
npm run open:android   # Android Studio
npm run open:ios       # Xcode (macOS only)
```

---

## Environment variables reference

### Live ASR model server

Live meeting transcription is now split out of the backend. The browser sends
mono PCM16 at 16 kHz directly to the model server WebSocket
`/api/v1/ws/live/{project_id}`. The model server owns VAD, denoise, Qwen3-ASR,
FunASR, and OpenAI STT calls. The backend only validates auth/project ownership
and persists recording/transcript data through internal REST endpoints.

Runtime env is split by service:

```bash
cp backend/.env.example backend/.env
cp server/.env.example server/.env
cp frontend/.env.local.example frontend/.env.local
```

`MODEL_SERVER_API_KEY` must match in `backend/.env` and `server/.env`. The
frontend must point REST traffic to the backend and live WebSocket traffic to
the model server:

```bash
NEXT_PUBLIC_API_URL=http://localhost:8000
NEXT_PUBLIC_WS_URL=ws://localhost:8002
```

Start services manually:

```bash
# terminal 1
cd backend
uvicorn main:app --host 0.0.0.0 --port 8000

# terminal 2
cd server
uvicorn main:app --host 0.0.0.0 --port 8002

# terminal 3
cd frontend
npm run dev
```

The backend no longer auto-starts Qwen3-ASR or any other model process. If the
model server or selected provider is not reachable, the UI reports the error and
no live recording is created.

### Transcript storage

Transcript data belongs in `recording_transcripts`.

- `recording_transcripts` is the only transcript source of truth.
- Do not add new code that reads or writes transcript data through
  `recordings.transcript`.
- New backend/frontend work should assume transcript search, edit, summary,
  AI retrieval, and importance markers operate on `recording_transcripts`.

### Studio design generation

Studio design generation is handled by the backend through the Google Stitch
bridge in `backend/stitch_bridge`. The frontend opens Studio from a project and
talks to `/api/v1/projects/{project_id}/designs/*`.

The recording screen can choose a live recognition model per session. FunASR
WebSocket uses the self-host realtime path. FunASR Whisper v3 Turbo runs through
the model server Python process after VAD finalizes each utterance. Qwen3-ASR
1.7B is the default recording model and runs as a separate OpenAI-compatible
server that receives each finalized VAD utterance through
`/v1/audio/transcriptions`. OpenAI GPT-4o Transcribe and OpenAI Whisper use the
same VAD utterances and transcribe each finalized speech segment.

### Live recording lifecycle

The intended lifecycle is explicit and project-scoped:

- A meeting is created inside a project.
- Only the meeting creator should capture microphone audio to the ASR pipeline.
- Other permitted users join as viewers and only receive live transcript and
  summary updates.
- Do not rely on browser refresh recovery, localStorage recovery, or reconnect
  grace windows as product behavior.
- Resume/continue behavior should be triggered explicitly from project or
  meeting detail UI, not inferred from an unexpected disconnect.

For architecture and product-direction details, follow
`CodexPlans/thay-doi-du-an-PLAN.md` rather than older meeting-centric notes.

For accuracy debugging, turn on `LIVE_STT_DEBUG_AUDIO=true` in `server/.env`
temporarily. The model server will save each finalized VAD utterance as a WAV
file under `server/tmp/live_stt_debug/<recording_id>/` and print audio stats
plus the ASR text to the model server console. Keep it off in normal meetings because it writes a
file for every utterance.

If the debug WAVs contain steady fan noise, you can test utterance-level noise
suppression with DeepFilterNet:

```bash
pip install deepfilternet
```

Then set `LIVE_STT_DENOISE_PROVIDER=deepfilternet` in `server/.env` and restart
the model server. This filters each finalized VAD utterance before Qwen3-ASR,
FunASR local, or OpenAI transcription. FunASR WebSocket still receives raw realtime frames, so use
`qwen_asr`, `funasr_local`, or `openai` while comparing denoise quality.

To test Qwen3-ASR on a GPU machine without changing the model server Python
environment, start the optional Docker profile:

```bash
docker compose --profile qwen-asr up -d qwen3-asr
```

The service maps Qwen's OpenAI-compatible endpoint to `http://localhost:8001/v1`
because the MeetAI backend usually runs on port `8000` and the model server on
`8002`. The compose service
starts `Qwen/Qwen3-ASR-1.7B` by default.

The official `qwenllm/qwen3-asr:latest` image may require a host NVIDIA driver
that supports CUDA 12.8+. If Docker fails with `unsatisfied condition:
cuda>=12.8`, upgrade the NVIDIA driver or keep using `FunASR Whisper v3 Turbo`
on CUDA 12.6 while testing.

To avoid changing the system NVIDIA driver, you can run Qwen3-ASR with the
transformers backend in an isolated Python venv:

```bash
/home/thoa/ENTER/envs/meetai/bin/python -m venv --system-site-packages .venv-qwen-asr
.venv-qwen-asr/bin/python -m pip install -U qwen-asr
QWEN_ASR_MODEL=Qwen/Qwen3-ASR-1.7B QWEN_ASR_DEVICE=cuda:0 \
  .venv-qwen-asr/bin/python server/tools/qwen_asr_server.py

This exposes the same local endpoint, `http://localhost:8001/v1`, without using
the CUDA 12.8 Docker image.

The Python server loads one Qwen3-ASR model per process. Make sure `/v1/models`
reports the same model selected by the recording screen:

```bash
curl http://localhost:8001/v1/models
```

| Variable | Required | Default | Description |
|---|---|---|---|
| `MONGO_URI` | ✅ | — | MongoDB connection string |
| `JWT_SECRET` | ✅ | — | ≥ 256-bit secret. `openssl rand -hex 32` |
| `MODEL_SERVER_API_KEY` | ✅ | — | Shared secret between `backend/.env` and `server/.env` for internal live recording persistence APIs |
| `LLM_BASE_URL` | | `https://openrouter.ai/api/v1` | `backend/.env`: OpenAI-compatible LLM base URL |
| `LLM_API_KEY` | ✅ | — | `backend/.env`: OpenRouter / OpenAI key |
| `LLM_MODEL_NAME` | | `qwen/qwen-vl-plus` | `backend/.env`: model identifier for chatbot and summaries |
| `BACKEND_BASE_URL` | | `http://localhost:8000` | `server/.env`: backend REST URL used by the model server |
| `SERVER_PORT` | | `8002` | `server/.env`: model server HTTP/WebSocket port |
| `STT_BASE_URL` | | `https://api.openai.com/v1` | `server/.env`: Whisper/OpenAI-compatible STT API base URL |
| `STT_API_KEY` | ✅ | — | Whisper API key |
| `STT_MODEL` | | `whisper-1` | Legacy/default STT model |
| `STT_REALTIME_WS_URL` | | `wss://api.openai.com/v1/realtime?intent=transcription` | Legacy OpenAI Realtime URL kept for compatibility |
| `STT_REALTIME_MODEL` | | `Qwen/Qwen3-ASR-1.7B` | Default realtime model used by the live recording picker |
| `STT_LANGUAGE_PROFILE` | | `vi-en` | Allowed language profile for transcription prompts. Use `vi-en`, `ja-en`, or `auto` |
| `STT_LANGUAGE` | | `vi` | Primary ISO-639-1 language hint. Use `vi` for Vietnamese or `ja` for Japanese |
| `STT_TRANSCRIPTION_PROMPT` | | _(profile prompt)_ | Optional custom STT prompt to constrain language/domain terms |
| `STT_TEMPERATURE` | | `0` | STT sampling temperature; lower values are more deterministic |
| `STT_LIVE_PROVIDER` | | `qwen_asr` | Default live provider. The UI can override this per recording |
| `STT_LIVE_PROVIDER_OPTIONS` | | `funasr_ws,funasr_local,qwen_asr,openai` | Providers accepted from the per-recording model picker |
| `STT_LIVE_MODEL_OPTIONS` | | `Qwen/Qwen3-ASR-1.7B,Whisper-large-v3-turbo,gpt-4o-transcribe,whisper-1` | Model IDs accepted from the per-recording model picker |
| `FUNASR_WS_URL` | | `ws://localhost:10095` | Self-host FunASR WebSocket endpoint |
| `FUNASR_MODE` | | `2pass` | FunASR mode used for online partial + per-utterance final text |
| `FUNASR_CHUNK_SIZE` | | `5,10,5` | FunASR chunk size sent in the init JSON |
| `FUNASR_CHUNK_INTERVAL` | | `10` | FunASR chunk interval sent in the init JSON |
| `FUNASR_ITN` | | `true` | Enable inverse text normalization in FunASR |
| `FUNASR_HOTWORDS` | | _(empty)_ | Comma-separated hotwords, e.g. `NextIT:20,MeetAI:20,tên khách hàng:30` |
| `FUNASR_CONNECT_TIMEOUT_SECONDS` | | `3` | Timeout for opening the FunASR WebSocket; prevents recording from hanging while FunASR is still downloading/loading models |
| `FUNASR_RETRY_COOLDOWN_SECONDS` | | `30` | How long to skip reconnect attempts after a FunASR connection failure before trying again |
| `FUNASR_LOCAL_DEVICE` | | `cuda:0` | Device for Python FunASR local models |
| `FUNASR_LOCAL_HUB` | | `auto` | Model hub for Python FunASR local models. `auto` uses Hugging Face for `FunAudioLLM/...` and ModelScope for short ModelScope IDs |
| `FUNASR_LOCAL_LANGUAGE` | | `auto` | Language hint passed to Python FunASR local models |
| `FUNASR_LOCAL_BATCH_SIZE_S` | | `0` | FunASR local batch size in seconds; `0` favors low-latency utterance tests |
| `QWEN_ASR_BASE_URL` | | `http://localhost:8001/v1` | OpenAI-compatible Qwen3-ASR server base URL |
| `QWEN_ASR_API_KEY` | | `EMPTY` | API key sent to the Qwen3-ASR server; vLLM local servers usually accept any non-empty value |
| `QWEN_ASR_LANGUAGE` | | `auto` | Optional Qwen language hint. `auto` maps `STT_LANGUAGE=vi` to `Vietnamese`, `ja` to `Japanese`, and `en` to `English` |
| `QWEN_ASR_TIMEOUT_SECONDS` | | `120` | Per-utterance timeout for Qwen3-ASR HTTP transcription |
| `LIVE_AUDIO_FRAME_MS` | | `60` | Browser PCM frame duration sent over WebSocket |
| `LIVE_VAD_THRESHOLD` | | `0.5` | Silero speech start threshold |
| `LIVE_VAD_THRESHOLD_LOW` | | `0.3` | Lower threshold while already inside speech |
| `LIVE_VAD_MIN_SILENCE_MS` | | `500` | Silence needed before finalizing an utterance |
| `LIVE_VAD_PRE_ROLL_MS` | | `300` | Audio kept before detected speech start |
| `LIVE_UTTERANCE_MAX_SECONDS` | | `25` | Maximum utterance duration before forced finalization |
| `LIVE_STT_DEBUG_AUDIO` | | `false` | Debug-only switch to save every VAD utterance as WAV and print audio stats/transcript |
| `LIVE_STT_DEBUG_DIR` | | `server/tmp/live_stt_debug` | `server/.env`: directory for debug utterance WAV files |
| `LIVE_STT_DEBUG_KEEP_FILES` | | `200` | Number of newest debug WAV files to keep per recording; `0` means no limit |
| `LIVE_STT_DEBUG_LOG_EMPTY` | | `true` | Also print debug lines when ASR returns empty text |
| `LIVE_STT_DEBUG_SAVE_DENOISED_AUDIO` | | `true` | When debug and denoise are enabled, also save the denoised utterance WAV for A/B listening |
| `LIVE_STT_DENOISE_PROVIDER` | | `off` | Optional ASR pre-processing. Use `deepfilternet` to suppress noise before per-utterance ASR |
| `LIVE_STT_DENOISE_MIN_SECONDS` | | `0.5` | Skip denoise for very short utterances |
| `DEEPFILTERNET_MODEL` | | `DeepFilterNet3` | DeepFilterNet pretrained model name |
| `DEEPFILTERNET_POST_FILTER` | | `false` | Enable DeepFilterNet post-filter for stronger suppression |
| `DEEPFILTERNET_ATTEN_LIM_DB` | | `12` | Noise attenuation limit; `0` means unlimited, but may distort speech more |
| `STT_LIVE_BATCH_SECONDS` | | `10` | Deprecated old OpenAI batch setting |
| `STT_LIVE_MIN_BATCH_SECONDS` | | `2` | Deprecated old OpenAI batch setting |
| `STT_VAD_THRESHOLD` | | `0.35` | Deprecated old OpenAI Realtime VAD setting |
| `STT_VAD_PREFIX_PADDING_MS` | | `500` | Deprecated old OpenAI Realtime VAD setting |
| `STT_VAD_SILENCE_DURATION_MS` | | `850` | Deprecated old OpenAI Realtime VAD setting |
| `STT_NOISE_REDUCTION` | | `far_field` | Deprecated old OpenAI Realtime noise reduction setting |
| `ENV` | | `development` | Set `production` to disable `/docs` and enforce CORS |
| `CORS_ORIGINS` | | `http://localhost:3005` | Comma-separated allowed origins |
| `NEXT_PUBLIC_API_URL` | | `http://localhost:8000` | `frontend/.env.local`: backend REST URL seen by browser |
| `NEXT_PUBLIC_WS_URL` | | `ws://localhost:8002` | `frontend/.env.local`: model server WebSocket URL seen by browser |
| `CAPACITOR_DEV_URL` | | _(empty)_ | LAN IP for Capacitor dev server |

---

## Verification checklist

```
docker compose up -d               → MongoDB healthy
cd backend && uvicorn main:app --reload --port 8000 → backend docs
cd server && uvicorn main:app --reload --port 8002  → model server docs
POST /api/v1/auth/register         → 201, access token + httpOnly cookie
POST /api/v1/projects (with token) → 201, project in DB
Live WS /api/v1/ws/live/...        → ready, partial_segment, segment, done
POST /api/v1/projects/{id}/designs/summary → 200, returns BA summary for Studio design
npm run dev (frontend)             → login, record, waveform plays
npx electron . (electron/)         → tray icon visible, loads app
npx cap sync android (mobile/)     → APK builds, background recording works
```

---

## Project structure

```
NextIT.MeetAI/
├── backend/                 FastAPI REST API: auth · DB · service orchestration
│   ├── app/
│   │   ├── api/routes/      auth · projects · project_chat · project_designs · recordings
│   │   ├── core/            config · database · security
│   │   ├── models/          Motor documents
│   │   ├── schemas/         Pydantic I/O schemas
│   │   └── services/        transcript · knowledge · design/model server clients
│   ├── main.py
│   ├── requirements.txt
│   └── Dockerfile
├── server/                  Live ASR model server
│   ├── app/
│   │   ├── api/routes/      WebSocket recording
│   │   ├── core/            model server config
│   │   └── services/        VAD · Qwen · FunASR · OpenAI STT
│   ├── tools/               manual Qwen3-ASR helper server
│   ├── main.py
│   └── requirements.txt
├── frontend/                Next.js 14 App Router
│   └── src/
│       ├── app/             auth · dashboard · projects · recordings
│       ├── components/      layout · projects · recording
│       ├── contexts/        AuthContext
│       ├── hooks/           useAuth · useRecording · useCapacitor
│       └── lib/             api.ts (axios + JWT interceptor)
├── electron/                Desktop wrapper
│   ├── main.js              BrowserWindow · tray · IPC
│   ├── preload.js           contextBridge API
│   └── tray.js
├── mobile/                  Capacitor
│   ├── capacitor.config.ts
│   ├── android/             permissions.xml
│   ├── ios/                 Info.additions.plist
│   └── scripts/init.sh
├── shared/                  Shared TypeScript types
├── docker-compose.yml
├── .env.example
└── README.md
```
