Skip to main content

Roadmap

Yew Search will evolve from a self-hosted homelab solution to an enterprise-grade SaaS platform, following the proven business model of GitLab, Mattermost, and N8N.

For the full breakdown of the newer capabilities referenced below (MCP server, conversational assistant, permissions/API-key auth model, homelab deployment), see the Detailed Roadmap. For supporting software (observability, analytics) rather than product features, see the Operations Roadmap.

Positioning​

Yew Search is a knowledge engine, not just a search engine: "one place for your knowledge" for individuals, and "one place for your company's knowledge" for teams and businesses. Search is the foundation, but the product is the combination of search, summarization, a read-only MCP server other AI tools can connect to, and a conversational assistant that can answer questions grounded in your own content.

This is a reframing of what the product does, not a change to who it's for or the business model below - the existing self-hosted → business → enterprise progression still applies.

Strategy Overview​

V1: Self-hosted homelab MVP - prove core value proposition V2: Multi-tenancy foundation - organizations, teams, and permissions V3: Knowledge engine capabilities - MCP server, conversational assistant, business SaaS launch V4: Enterprise tier - dedicated infrastructure and compliance V5: Turnkey enterprise identity - SSO, provisioning sync, and infrastructure-as-code, so a large customer administers Yew through their existing tooling rather than logging into it directly after initial setup

A real dependency chain runs through V2 and V3, not just a suggested order: the MCP server needs the V2 permissions system to exist first (an API key is a first-class principal, authorized the same way a user is - there is no separate, throwaway auth model to build for it), and the conversational assistant needs the MCP server to exist first (it's built as an MCP client consuming Yew's own knowledge tools, not a second bespoke retrieval implementation). See the Detailed Roadmap for why.

V1 - Self-Hosted MVP​

Target: Individual users, homelab enthusiasts, proof of concept

Goal: Ship a working self-hosted personal knowledge engine that users can deploy on their own hardware with Docker.

Note on sequencing: V1 was originally scoped around PostgreSQL full-text search only, with Elasticsearch deferred to V2. That got pulled forward - a knowledge engine that returns bad search results isn't useful to anyone, homelab audience included, so Elasticsearch (plus hybrid text+vector search) shipped as part of V1 instead of waiting for V2's multi-tenancy work. Out-of-order execution like this is expected and fine when the reason is this direct - it doesn't need to wait for a roadmap rewrite to happen again.

Backend​

  • User authentication system (cookie-based sessions, never JWT)
    • Users are created via CLI command
    • User login endpoints
    • Session management (create, validate, delete)
    • Argon2 password hashing
    • Redis session store for auth lookups
    • PostgreSQL user_session table for session management UI
  • User module
    • User CRUD operations
    • User profile endpoints
    • Active sessions list (view/terminate sessions)
  • Integration architecture
    • Base integration classes and interfaces
    • Integration loader (dynamic plugin loading)
    • OAuth endpoint (unified for all integrations)
    • Task-based execution system
  • Polling system
    • Bull task queue (Redis-backed)
    • Priority-based task scheduling
    • Idempotency checking (contentExists callback)
    • Background worker for executing integration tasks
  • Gmail integration (OAuth)
    • OAuth flow implementation
    • Task types: start, getEmailList, downloadEmail
    • Pagination support
    • Early stop optimization
    • Store email content in user_integration_content table
  • FTP/sFTP integration
    • Credentials-based auth (not OAuth)
    • Directory traversal and file metadata indexing
  • Search system - pulled forward from V2, see note above
    • Elasticsearch full-text search with a deliberate, explicit index mapping (not dynamic-mapped) - see Detailed Roadmap for the mapping/migration approach
    • Hybrid search: Elasticsearch text relevance merged with vector similarity via reciprocal rank fusion
    • Vector search is optional and configurable, not a hard requirement - a self-hoster on modest hardware (Raspberry Pi class) can run Elasticsearch text search alone without also needing to run an embedding model
    • Self-hosted embedding provider support via Ollama, alongside hosted providers (OpenAI, etc.) for anyone who'd rather not run a local model at all
    • Result ranking tuned empirically against a real eval corpus, not just theoretically
  • Database schema
    • user table
    • user_session table
    • user_integration table (with encrypted credentials)
    • user_integration_content table (JSONB content storage)
  • Observability
    • Structured JSON logging
    • Request context (requestId, userId, traceId)
    • Basic error handling and logging
  • Docker setup
    • Backend Dockerfile
    • Portable homelab docker-compose.yaml - a single, self-contained file with everything needed to run Yew Search on your own hardware (Postgres, Redis, Elasticsearch, backend, frontend), separate from any one person's personal homelab configuration. See the Detailed Roadmap.

Frontend​

  • Svelte app setup
    • App router structure
    • Basic layout and navigation
  • Authentication pages
    • Login page
    • Cookie-based auth
  • Search interface
    • Search input and results display, filterable by embedding config and by integration
    • Loading and error states
  • Embedding config management
    • Create, list, and rename embedding configs
  • Integration management
    • "Connect Gmail" button
    • OAuth authorization flow
    • Integration status display
  • Settings page
    • User profile
    • Active sessions (list and terminate)
  • Frontend Dockerfile

Documentation​

  • Getting Started guide (README update)
  • Architecture overview
  • Backend standards (service, controller, DTO, entity)
  • Integration development guide
  • OAuth integration guide
  • Authorization/session management guide
  • Coding style guide
  • Deployment instructions (Docker) - homelab and AWS paths, kept current as the actual deployment process changes

Website​

  • Single-page landing site
    • Project description
    • Key features
    • Link to docs
    • GitHub link

V2 - Multi-Tenancy Foundation​

Target: Teams, small companies (still self-hosted), prepare for SaaS

Goal: Add organizational primitives and a real permission system. This is the unlock for V3's knowledge-engine capabilities, not just an organizational nice-to-have - see the dependency chain in the Strategy Overview above.

Core Features​

  • Database migration system
    • TypeORM migrations
    • Migration CLI commands
    • Rollback support
  • Organizations/companies
    • Organization entity and CRUD
    • User-to-organization relationships
    • Organization settings
  • Teams/groups within organizations
    • Team entity and CRUD
    • Team membership
    • Team-level permissions
  • User permissions system
    • Role-based access control (owner, admin, member, viewer)
    • Permission checks in services
    • Sharing integrations between users
    • Data source access control
    • A unified auth-principal shape: whether a request authenticates via cookie session or API key, both resolve into the same normalized permission object before any authorization check runs - see the Detailed Roadmap
  • API keys
    • First-class principals, not scoped delegations of the user who created them - a key is assigned groups/permissions directly, the same way a user is
    • Foundation for the V3 MCP server's auth model
  • User invitations
    • Email-based invitations
    • Invitation acceptance flow
    • Pending invitations management

Search Improvements​

(Elasticsearch itself moved to V1 - see the note there. What's left here is what still depends on multi-tenancy.)

  • Search filters
    • Filter by date range (API support exists; UI deferred - it's a debugging tool for now, not a user-facing filter)
    • Filter by sender/source
    • Organization/team-scoped search results, once organizations exist

Additional Integrations​

  • Slack integration (OAuth)
    • OAuth flow
    • Channel message syncing
    • Direct message syncing

Infrastructure​

  • Homelab deployment hardening
    • Raspberry Pi / low-resource optimization
    • Memory-optimized PostgreSQL config
    • CPU throttling for background tasks
    • Minimal Docker image sizes
    • Performance testing on Pi 4, including with vector search disabled
  • Environment configuration
    • .env.example with all required vars
    • Configuration validation on startup
    • Better error messages for missing config

UI Improvements​

  • Organization switcher
  • Team management UI
  • Permission management UI
  • API key management UI (create, assign groups/permissions, revoke)
  • Integration settings per user/team

V3 - Knowledge Engine Capabilities & Business SaaS Launch​

Target: Companies that want a hosted solution, plus self-hosters who want the full knowledge-engine experience

Goal: Launch the capabilities that make Yew a knowledge engine rather than a search box, and launch Yew Search as a hosted SaaS product. Self-hosted version remains available with core features - see Feature Differentiation below, though exactly which of the new capabilities are self-hosted-available vs. business-tier-only is still an open call, not yet decided.

Knowledge Engine Capabilities​

(Requires V2's permission system - see the Strategy Overview dependency chain.)

  • Search result summarization
    • Reuses the same provider-routing pattern already built for embeddings (LangChain adapters - OpenAI, Ollama, Voyage, Google GenAI, Mistral AI today; a chat-model equivalent would be where a provider like Anthropic, which has no native embeddings API, would actually show up)
    • Summarize top N results, extract key points
  • Universal read-only MCP server
    • Scoped strictly to knowledge retrieval (search, fetching a document's full content, listing available sources) - explicitly not a control plane for managing Yew itself
    • Authenticated via the V2 API key system, authorized through the same permission model as a logged-in user
    • Two consumers: Yew's own conversational assistant, and external AI assistants from other products a company wants to connect to its knowledge
    • Has standalone value even for someone who never uses Yew's own search UI or assistant
  • Conversational AI assistant
    • Built as an MCP client consuming Yew's own MCP server, not a separate retrieval implementation
    • Multi-turn conversation, not a single one-shot query - the agent can search, look at results, and search again as it reasons
    • Answers grounded in retrieved content with citations back to source documents - an ungrounded or uncited answer undermines the entire "knowledge engine" premise

SaaS Infrastructure​

  • Cloud deployment
    • Production-grade docker-compose or Kubernetes
    • Load balancer setup
    • Database connection pooling
    • Redis clustering
  • Multi-tenant architecture
    • Data isolation per organization
    • Tenant-aware queries (all services check organization)
    • Database per tenant vs shared database decision
  • Billing and subscriptions
    • Stripe integration
    • Subscription plans (Business, Enterprise)
    • Usage tracking (searches, storage, integrations)
    • Resource quotas per plan
    • Billing portal (Stripe Customer Portal)
  • Admin dashboard
    • Organization list and search
    • User activity monitoring
    • System health metrics
    • Feature flag management per customer
  • Onboarding flow
    • Signup for Business tier
    • Organization creation
    • Team setup wizard
    • Integration walkthrough
  • Search collections/groups
    • Group multiple integrations into collections
    • Search within specific collections
    • Share collections with team members
  • Query expansion and synonyms
  • Saved searches
    • Save frequently-used searches
    • Search history per user

More Integrations​

  • Google Drive (OAuth)
  • Dropbox (OAuth)
  • Microsoft 365 (OAuth)
  • At least 3 more integrations based on user demand

UI/UX Improvements​

  • Polish all interfaces
    • Professional design
    • Consistent component library (Shadcn/UI)
    • Mobile responsive
  • Dark mode - already shipped
  • Keyboard shortcuts
  • Advanced search syntax
  • Result preview/quick view
  • Conversational assistant chat interface

Marketing Site​

  • Full marketing website (separate from app)
    • Feature pages
    • Pricing page
    • Documentation
    • Blog
    • Customer testimonials

V4 - Enterprise Tier (6-12 months)​

Target: Large companies with custom needs, compliance requirements

Goal: Launch Enterprise tier with dedicated infrastructure, custom integrations, and compliance.

Enterprise Features​

  • Dedicated infrastructure provisioning
    • Per-customer infrastructure
    • Custom resource allocation
    • Dedicated database
    • Isolated workers
  • Custom integrations
    • Build integrations per customer request
    • Private integrations (not available to other customers)
    • Integration development as a service
  • Advanced admin features
    • Audit logs (all actions logged)
    • Data retention policies
    • Export all data (GDPR compliance)

(SSO/SAML and SCIM-based provisioning moved to V5 - see below. They're identity/provisioning concerns, not infrastructure/compliance ones, and V5 has the actual dependency reasoning for why they land together with Terraform-managed provisioning and possible ACL sync.)

Compliance & Security​

  • SOC2 Type II certification
    • Security audit preparation
    • Compliance documentation
    • Annual audits
  • GDPR compliance enhancements
    • Right to be forgotten
    • Data portability
    • Consent management
  • HIPAA compliance (if needed)
    • BAA agreements
    • Encryption at rest and in transit
    • Access controls
  • Security hardening
    • Penetration testing
    • Vulnerability scanning
    • Incident response plan
    • Security training

Support & Success​

  • Dedicated support
    • Slack channel per customer
    • Response time SLAs
    • Priority bug fixes
  • Customer success manager
    • Regular check-ins
    • Feature adoption tracking
    • Custom training sessions
  • Professional services
    • Integration development
    • Custom feature development
    • Migration assistance

Customer Launch​

  • 1 pilot Enterprise customer
    • Small team (5-10 people)
    • Gather feedback
    • Refine Enterprise offering
  • Case study and testimonial
  • Enterprise sales process documentation

V5 - Turnkey Enterprise Identity & Infrastructure-as-Code​

Target: Large enterprises with an existing identity provider (Okta, Azure AD, Google Workspace) and an infrastructure-as-code culture.

Goal: Let a large customer administer their entire Yew presence - who exists, what teams they're in, what they can access - through tooling they already run, after a single initial setup. Depends on V2's permission model being stable through real V2/V3/V4 usage first - see the Detailed Roadmap.

Identity​

  • SSO (SAML 2.0, OIDC) - authentication only. Changes how a login handshake starts; the session that results is still Yew's own cookie-based session (see docs/docs/backend/authorization.md), not a different auth model.
  • SCIM-based provisioning - a distinct integration from SSO, even though customers usually adopt both together. Lets an IdP push user creation, deactivation, and group membership into Yew automatically, mapping IdP groups to Yew teams.
  • Okta, Azure AD, Google Workspace as example IdPs - the target is the SAML/OIDC/SCIM standards, not any one vendor.

Infrastructure-as-code​

  • A Terraform provider for Yew's own resources (companies, teams, folders, integration assignments, permission grants) - the same pattern already proven by products like Grafana, not a novel idea. Requires the V2 permission API to have stabilized through real usage first; building a provider against a still-changing API means constant breakage.
  • Resources need stable, human-meaningful identifiers (not just UUIDs) for terraform import/idempotent re-apply to work cleanly - a design constraint for the V2 API worth keeping in mind now, even though the provider itself is a V5 deliverable.

Consider at this point: ACL sync​

Whether to let Yew mirror a connected source's own access-control list (the way Glean does - see docs/docs/backend/permissions.md) rather than only using Yew's own permission model, for integrations where the underlying source system already has its own real ACLs (e.g. a domain-wide Google Workspace connection). Not a committed V5 deliverable - explicitly a question worth reconsidering once the identity/provisioning maturity above exists, since that's the same customer profile who'd actually have a source-system ACL worth syncing from. See the open questions in docs/docs/backend/permissions.md.

Optimizations​

Database Query Performance Monitoring​

Context: TypeORM query logging is disabled (logging: false in app.module.ts) because it's noisy and only captures queries from one app instance. Instead, we use PostgreSQL's built-in performance monitoring tools.

The industry standard for query performance analysis. Tracks execution statistics for all SQL statements across all connections.

Setup:

-- Enable the extension (one time)
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- Query slow queries
SELECT
  query,
  calls,
  total_exec_time,
  mean_exec_time,
  max_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 20;

Benefits:

  • Tracks all queries across all connections and app instances
  • Shows execution count, total time, mean time, max time
  • Persists across sessions
  • Near-zero performance overhead
  • Essential for production query optimization

log_min_duration_statement​

Automatically log queries that exceed a duration threshold to PostgreSQL logs.

Setup:

-- In postgresql.conf or via ALTER SYSTEM
ALTER SYSTEM SET log_min_duration_statement = 1000;  -- Log queries > 1 second
SELECT pg_reload_conf();

Benefits:

  • Simple to set up
  • Captures slow queries with execution time
  • Useful for catching extreme outliers
  • Logs include query parameters

auto_explain Module​

Automatically logs execution plans for slow queries. Useful for understanding why specific queries are slow.

Setup:

-- In postgresql.conf (requires restart)
shared_preload_libraries = 'auto_explain'
auto_explain.log_min_duration = 1000  -- Explain queries > 1s
auto_explain.log_analyze = true
auto_explain.log_timing = true

Benefits:

  • Shows EXPLAIN ANALYZE output for slow queries
  • Helps identify missing indexes or inefficient query plans
  • No code changes required

V1 (Development/Self-Hosted):

  • Enable pg_stat_statements for query analysis
  • Set log_min_duration_statement = 2000 to catch very slow queries

V2-V3 (Multi-tenant/SaaS):

  • Enable pg_stat_statements (required)
  • Set log_min_duration_statement = 1000
  • Consider auto_explain for production debugging

V4 (Enterprise):

  • All of the above
  • Query performance monitoring dashboards
  • Automated slow query alerts
  • Per-customer query performance analysis

Future Optimizations​

  • Add query performance metrics to observability stack
  • Create Grafana dashboard for pg_stat_statements data
  • Automated query optimization suggestions
  • Index recommendation system based on slow query patterns

Elasticsearch Query Performance​

Similar principle applies to Elasticsearch as it does to Postgres above - the search-tuning work is empirical (a real eval query set checked into lab/alan/search-tuning/), not theoretical. See the Detailed Roadmap for how the current ranking approach was arrived at. The remaining misses are not static - they shift identity every time the embedding model or ranking weights change, so treat eval-history.jsonl as the current source of truth rather than any specific example cited here or elsewhere.

Test Optimizations​

Currently the e2e tests maintain the atomicity by clearing the database before each test Even for small datasets this can take time.

While the clearDatabase function clears all tables in parallel there will probably be a time when we need to NOT clear the database after each test.

Additionally, the current testing strategy does not allow multiple tests in the same suite to run in parallel since they read and write data to the same table.

Determining how to run tests in parallel while allowing reads and writes to a real database will be required at some point.

Feature Differentiation​

Self-Hosted (Always Free)​

  • Core search functionality - Elasticsearch hybrid search, vector search optional/configurable
  • Gmail integration
  • 1-2 additional basic integrations (FTP, Slack)
  • Single user or family use (< 10 users)
  • Community support only
  • Docker deployment (portable homelab docker-compose.yaml)

Business Tier (SaaS - Paid)​

  • Hosted infrastructure (no self-hosting)
  • Unlimited users per organization
  • Teams and permissions
  • All integrations
  • LLM features (summarization, conversational assistant) - self-hosted availability of these is still TBD, not yet decided
  • MCP server for connecting external AI assistants
  • Search collections
  • SSO (Google, Microsoft)
  • Email support
  • 99.9% uptime SLA
  • Usage analytics
  • Admin dashboard

Enterprise Tier (High-Touch - Custom Pricing)​

  • Everything in Business tier
  • Dedicated infrastructure
  • Custom integrations
  • SAML/SSO (any provider)
  • SOC2/HIPAA compliance
  • Dedicated support (Slack channel)
  • Customer success manager
  • Professional services
  • Custom SLAs
  • Data residency options
  • On-premise deployment option
  • SCIM-based user provisioning, Terraform-managed org/team provisioning, and ACL sync (still TBD) - V5, see above

Long-Term Vision​

Product: A knowledge engine - "one place for your knowledge" for individuals, "one place for your company's knowledge" for teams. Search is the foundation; summarization, a read-only MCP server, and a grounded conversational assistant are what make it a knowledge engine rather than a search box.

Business Model: Fair-code / Open-core

  • Self-hosted version remains genuinely useful forever
  • Core functionality always free
  • Advanced features for companies (not individuals)
  • Commercial license required for business use

Deployment Options:

  1. Local/Homelab - Free for personal use, community supported
  2. Business SaaS - Hosted multi-tenant, standard pricing
  3. Enterprise - Dedicated infrastructure, custom pricing

Target Markets:

  • Phase 1 (V1-V2): Homelab enthusiasts, power users, families
  • Phase 2 (V3): Small-medium businesses (10-100 employees)
  • Phase 3 (V4-V5): Large enterprises (100+ employees) - V5 serves the same market as V4, not a new one; it's a maturity milestone (turnkey identity/infra-as-code) within the Enterprise tier, not a new tier

Success Metrics:

  • V1: 100 active self-hosted deployments
  • V2: 1,000 active self-hosted deployments
  • V3: 50 paying Business customers
  • V4: 5 Enterprise customers
  • V5: 2 Enterprise customers administering Yew primarily via SSO/SCIM/Terraform rather than the admin UI directly

Notes​

  • All versions maintain backward compatibility with self-hosted deployments
  • Breaking changes communicated 90 days in advance
  • Community input welcome on feature prioritization
  • Roadmap updated quarterly based on feedback
  • Out-of-order execution within a version is expected when there's a direct reason (see V1's note on Elasticsearch) - the version numbers describe a rough dependency order, not a rigid schedule