Cut HR First-Response Time in a 2,000+ B2B SaaS Company: A Two-Week RAG Pilot

The Problem: HR First-Response Time in a 2,000+ Employee B2B SaaS Company

A 2,000+ employee B2B SaaS company in Switzerland runs HR and recruiting operations on a mix of Confluence, Notion, and a helpdesk. Employees ask the same 40 questions every week: how to request PTO, how to file an expense report, how to access the staging environment. The current first-response time is 4–6 hours because the answer lives in a Confluence page that no one can find quickly. The goal is to cut first-response time to under 10 minutes by building a retrieval-augmented knowledge assistant that searches the company’s own documentation and returns a sourced answer. The pilot runs for two weeks, uses Anthropic Claude API for generation, and ships with ISO 27001-compliant access controls and audit logging. The delivery model is managed AI operations: Forfis builds, deploys, and monitors the system, and the client’s team owns the content and the feedback loop.

Prerequisites Before Step 1

• Knowledge base access: API credentials for Confluence or Notion, with read access to the relevant workspaces. Confirm the workspace contains the 40 most-asked questions.
• Anthropic API key: A production key with usage limits set. Store it in a secrets manager (HashiCorp Vault, AWS Secrets Manager, or GCP Secret Manager), not in code.
• Communication channel: Slack or Microsoft Teams workspace where employees ask questions. Confirm webhook or API access is available.
• Vector database: A managed instance (Pinecone, Weaviate, or pgvector on Postgres) with sufficient capacity for the knowledge base size. For a 2,000+ employee company, expect 5,000–20,000 documents.
• ISO 27001 documentation: Access control policies, audit logging requirements, and data retention rules. The pilot must comply with these before go-live.
• Baseline data: A one-week log of HR questions, current first-response times, and resolution rates. This is the before/after measurement point.

Step 1: Ingest the Knowledge Base

Export all relevant Confluence or Notion pages to a structured format. Use the Confluence REST API (/rest/api/content?spaceKey=HR) or the Notion API (/v1/databases/{database_id}/query) to pull pages. Store the output as JSON files in a staging directory. Each document should include: id, title, body (Markdown), last_updated, and owner. For a 2,000+ employee company, expect 5,000–20,000 pages. Filter out pages marked as deprecated or restricted. The ingestion script should run in under 30 minutes for a typical workspace. Log the number of pages ingested and any errors to a CSV file for the audit trail.

Step 2: Build the Retrieval Pipeline

Split each document into chunks of 256–512 tokens, with a 50-token overlap. Use a semantic chunking strategy: split on headings first, then on paragraphs. For each chunk, generate an embedding using the text-embedding-3-small model (OpenAI) or bge-large-en (open-weight, if the data cannot leave the building). Store the embeddings in the vector database with metadata: document_id, chunk_index, title, last_updated. For a 10,000-document knowledge base, expect 50,000–100,000 chunks. The indexing process should take under 2 hours on a managed vector database. Verify the index by running 10 test queries and confirming that the top-5 results are relevant.

Step 3: Configure the Generation Layer

Configure the Anthropic Claude API call with the following parameters: model: claude-sonnet-4-20250514, max_tokens: 1024, temperature: 0.2. The system prompt should instruct the model to answer only from the retrieved context, cite the source document, and say “I don’t know” if the answer is not in the context. The user prompt should include: the employee’s question, the top-5 retrieved chunks (with titles and URLs), and a request for a concise answer with a source link. Test the pipeline with 20 real questions from the baseline log. Measure: (1) retrieval precision (are the top-5 chunks relevant?), (2) generation accuracy (is the answer correct?), (3) latency (should be under 3 seconds end-to-end). Iterate on the chunking and prompt until accuracy is above 80%.

Step 4: Integrate with the Communication Channel

Integrate the assistant with Slack or Microsoft Teams. In Slack, create a custom app with a /ask slash command. The command sends the question to the RAG pipeline, waits for the response, and posts it back to the channel. In Teams, use a bot framework (Microsoft Bot Framework) with a similar flow. The response should include: the answer, a link to the source document, and a feedback button (thumbs up/down). The feedback button sends a structured event to a logging endpoint. For ISO 27001 compliance, log every query with: timestamp, user_id, question, retrieved_chunks, model_response, feedback. Store the logs in a read-only database with a 12-month retention policy. Restrict access to the assistant via SSO: only authenticated employees can use it.

Step 5: Run the Two-Week Pilot

Run the pilot for two weeks with a defined scope: one department (HR or recruiting), one knowledge source (Confluence or Notion), one channel (Slack or Teams). Track five metrics daily: (1) first-response time (target: under 10 minutes, baseline: 4–6 hours), (2) resolution rate (target: 70%, baseline: 30–40%), (3) accuracy (target: 80%, measured by user feedback), (4) retrieval precision (target: 85%, measured by manual review of 50 queries), (5) user satisfaction (target: 4/5, measured by post-answer rating). At the end of week two, produce a report with: before/after metrics, a list of the top 10 unanswered questions, and a recommendation for rollout. The report should be reviewed by the client’s HR lead and the Forfis delivery team.