The problem: manual triage across three languages
Your support team handles 150 to 400 tickets per day across English, Spanish, and German. Each ticket is read, categorized, and routed by a human agent before any response is drafted. The median cycle time from ticket creation to first human action is 42 minutes. You want to cut that number without adding headcount, and you want the routing to work across all three languages without a separate team per locale. The constraint is that you cannot replace your helpdesk or CRM. The model must plug into the REST API and webhook endpoints you already expose, and the pilot must be scoped so that you know the total cost and the success criteria before the first sprint starts.
Prerequisites before step 1
Before the first sprint, confirm the following are in place:
- Helpdesk API access. A service account with read and write permissions on ticket objects. The account must be able to create, update, and query tickets via REST. Verify that the API rate limit is at least 100 requests per minute.
- Webhook endpoint. A publicly reachable HTTPS URL that accepts POST requests with a JSON body. The endpoint must return a 200 status within 5 seconds. If your helpdesk does not natively support webhooks, you will need a lightweight relay service.
- Historical ticket data. At least 500 labeled tickets per language, exported as CSV or JSON. Each record must include the ticket body, the final category, the assigned team, and the language tag. This is the training set for the scoring model.
- OpenAI API key. A key with access to the GPT-4o or GPT-4o-mini model. The key must have sufficient credits for the pilot volume. For 300 tickets per day over 14 days, budget for roughly 4,200 API calls.
- A named owner. One person on your side who can approve scope changes, answer integration questions, and sign off on the pilot results. This person should have authority over the helpdesk configuration.
Step 1: Export and label your historical tickets
Export 500 to 1,000 tickets per language from your helpdesk. Each record must contain the ticket body, the final category assigned by a human, the team that handled it, and the language tag. If your helpdesk does not store a language tag, infer it from the ticket body using a language-detection library such as langdetect or fasttext. Save the export as tickets_train.csv with columns: ticket_id, body, category, team, language. Split the file into a 70% training set and a 30% validation set. The validation set is used to measure routing accuracy before the model goes live. If any category has fewer than 50 examples, merge it with a related category or flag it for manual review in the pilot.
Step 2: Configure the OpenAI scoring model
Build a scoring function that takes a ticket body and returns a category label, a confidence score from 0 to 100, and a language tag. Use the OpenAI API with the GPT-4o model. The prompt should include the list of valid categories, the language of the ticket, and the instruction to return JSON with fields category, confidence, and language. Set the temperature parameter to 0.1 to reduce variance. Set max_tokens to 200. The function should handle API errors by retrying up to three times with exponential backoff (1 second, 5 seconds, 25 seconds). If all three retries fail, return a default category of unclassified with a confidence score of 0. Log every API call with the ticket ID, the model version, and the latency in milliseconds. Store the logs in a file or a lightweight database for the pilot review.
Step 3: Build the webhook-to-helpdesk router
Write a webhook handler that receives the scoring result and calls your helpdesk REST API to update the ticket’s routing field. The handler should accept a POST request with a JSON body containing ticket_id, category, confidence, and language. It should call the helpdesk API endpoint PATCH /tickets/{ticket_id} with a JSON body that sets the routing field to the predicted category and the priority field based on the confidence score. If the confidence score is 85 or above, set the priority to auto. If the score is between 60 and 84, set the priority to review. If the score is below 60, set the priority to manual. The handler must return a 200 status to the caller within 5 seconds. If the helpdesk API returns an error, log the error and retry up to three times. After the third failure, write the ticket ID to a dead-letter queue file.
Step 4: Validate routing accuracy on the holdout set
Run the scoring model on the 30% validation set from step 1. For each ticket, compare the predicted category to the human-assigned category. Calculate the routing accuracy as the percentage of tickets where the predicted category matches the human category. Calculate the median confidence score for correctly routed tickets and for incorrectly routed tickets. If the routing accuracy is below 80%, review the misclassified tickets and adjust the prompt or the category definitions. If the median confidence for correct tickets is below 70, lower the confidence threshold for auto-routing. Document the final thresholds in a configuration file named triage_config.json with fields auto_threshold, review_threshold, and manual_threshold. This file is read by the webhook handler at startup.
Step 5: Run the two-week pilot
Deploy the webhook handler to a staging environment that mirrors your production helpdesk configuration. Send 50 test tickets through the full pipeline: ticket creation in the helpdesk, webhook trigger, scoring model call, routing update. Verify that each ticket is routed to the correct team and that the priority field is set according to the confidence thresholds. Check the dead-letter queue file for any failed deliveries. Monitor the API latency for each scoring call. The median latency should be under 800 milliseconds. If the median latency exceeds 1,200 milliseconds, reduce the max_tokens parameter or switch to the GPT-4o-mini model. Once all 50 test tickets pass, promote the handler to production and enable the webhook on your live helpdesk instance.
Leave a Reply