A RAG-powered prompt engineering platform and research program covering prompt structure, retrieval strategy, model behavior, judged evaluations, architecture, Azure ML costs, deployment history, and documented tradeoffs.
The metrics below separate different kinds of value: downstream output quality, retrieval orchestration cost, model/runtime choice, and prompt-length efficiency. That keeps the case study honest about what each percentage actually measures.
| Metric | Scope | Result | Impact | Interpretation |
|---|---|---|---|---|
| PromptTriage vs generic prompt | Study D IFEval, Gemini 3.1 Pro, 50 deterministic instruction-following tasks | 82% vs 72% | +10 percentage points, +13.9% relative | Best direct comparison against a simple generic system prompt. Claude was less helped on this specific deterministic slice, so this is an upper-bound task-specific result. |
| PromptTriage vs no system prompt | Study C v2, Nemotron 120B downstream tasks, 13 paired tasks | 30.92 vs 27.54 | +3.38 / 40, +12.3% relative | Largest model-level downstream lift. Gemini was more prompt-invariant, which is why the case study keeps model sensitivity explicit. |
| Naive RAG vs judge RAG | Study A, 180 prompt-generation outputs judged on a /50 rubric | 42.27 vs 38.97 | +8.5% relative quality and 57.3% faster | This is a retrieval-orchestration result, not a downstream model result. It shows simple retrieval beat heavier judge-based retrieval for this pipeline. |
| Naive RAG vs agentic RAG | Study A, same RAG ablation matrix | 42.27 vs 39.40 | +7.3% relative quality and 42.5% faster | Useful as an engineering tradeoff metric: more agent steps did not produce better prompts in this experiment. |
| Short structured prompts vs long prompts | Study E v2, 1,080 format/length evaluations across five models | 32.03 vs 26.76 | +19.7% relative quality with 86.7% fewer words | This supports a resource-efficiency claim: PromptTriage should prefer compact, front-loaded structure instead of long generic instruction walls. |
| Qwen3 14B vs tested MoE path | Study B, prompt-generation benchmark outputs | 41.77 vs 41.47 | Slightly higher score and about 64x faster | This is a model/runtime selection metric. It justifies the production-candidate choice without pretending bigger models always win. |
Honorable mention: the research also supports a practical prompt-engineering rule for multimodal and agentic workflows. Important instructions should be front-loaded, with role, constraints, input priority, output format, and safety boundaries ordered from most important to least important. Models react differently to token order, so how context is fed to the model can matter as much as the wording itself.
The largest gains appeared when the task punished default model behavior: strict format constraints, anti-trend design instructions, timezone reasoning, domain boundaries, security review, and exact instruction following.
Pricing dashboard requested stark white brutalist HTML/CSS: no dark mode, gradients, rounded corners, shadows, or external frameworks.
+30 / 40, +600% relative
PromptTriage helped the model escape common AI UI defaults and obey negative constraints that generic prompting missed.
Ocean poem with exactly 14 lines and exactly 8 syllables per line.
+18 / 40, +257% relative
The biggest formatting gains appeared where the task required exact structural compliance rather than fluent prose.
Earliest 1-hour meeting slot across EST and GMT+8 with weekday and time-window constraints.
+17 / 40, +131% relative
The prompt structure helped with constraint ordering and assumption control on a multi-step reasoning task.
Three-box gold puzzle where only one statement is true.
+13 / 40, +72% relative
PromptTriage helped most when the response needed systematic hypothesis testing instead of intuitive guessing.
Node API review with a default live-looking secret key and payment processing call.
+5 / 50, +11.6% relative
The behavioral benchmark showed better role expertise, edge-case coverage, boundaries, specificity, and format on a security-sensitive task.
Postgres on AWS RDS replica lag rising while primary CPU is at 92%.
+4 / 50, +9.8% relative
The improvement clustered around edge cases, specificity, and operational formatting, which are the dimensions that matter in incident response.
Thyroid fatigue triage and vacation savings goal planning.
+7.0% to +7.3% relative
PromptTriage improved boundary setting and edge-case awareness on domains where confident generic advice can become risky.
IFEval tasks involving titles, exact sentence/word counts, placeholders, bullet-list counts, quotations, and no-comma constraints.
82% PromptTriage vs 72% simple / 76% bare
The measurable wins came from discrete compliance failures that generic prompts often leave to chance.
PromptTriage was built around several connected studies, not a single benchmark. The public articles cover the most readable findings, while the local research logs preserve model, RAG, judge-bias, and stopped/resumed run details.
Analyzed a large corpus of production system prompts to identify recurring anti-patterns: persona bloat, contradictory long-prompt clauses, and emotional-pressure scaffolds that distracted from the actual task.
Hashnode 28K writeup, release chart, Pinecone corpus evidence.Curated publicly leaked system prompts attributed to major AI products and labs, including Anthropic, OpenAI, Google, Perplexity, Cursor, v0, Windsurf, and Gemini CLI, then used them as reference patterns instead of republishing private prompt text on the portfolio page.
Local PromptTriage repo references, training data notes, RAG corpus docs.The public writeup describes five frontier models evaluated across 12 task domains, six output formats, and three prompt lengths. The published article says each result was blindly scored by a three-judge LLM panel; local research notes also preserve earlier Study E run details and cross-model judging variants.
Hashnode Format Wars article, release charts, local RESEARCH_PROGRESS.md.Several studies exposed judge bias, especially self-preference from the original Gemini judge. Later passes used neutral Llama 4 Maverick judging or cross-model judging, so the case study records both positive results and evaluator limitations.
PromptTriage research progress log and benchmark notes.The full research writeups currently live on Hashnode. This page treats them as source evidence and connects them to the structured case-study template.
Hashnode article covering the 28K prompt corpus, production anti-patterns, and why prompt hygiene became the core product thesis.
Read on Hashnode →Hashnode article covering the 1,080-eval format study and the measurable impact of prompt structure across models and domains.
Read on Hashnode →Release assets from the PromptTriage repository are shown directly here so the case study has visible evidence instead of only outbound article links.
The app showcase is now available as a GitHub release asset. Because it is about 121 MB and marketing-heavy, this portfolio links to it deliberately instead of autoplaying or treating it as primary benchmark evidence.
The release video is useful as a product walkthrough, but the case study treats it as supporting demo evidence. The main technical proof remains the benchmark logs, Azure ML runs, research charts, and implementation source.
Open release video →Azure cost evidence now includes the EUR 178.12 credit spend screenshot plus CLI-reviewed Azure ML job metadata for the highest-signal benchmark runs.
Cost screenshot groups spend under Virtual Machines in East US 2 for the qwentrain workspace period.
MoE-only Study B benchmark, showing why naive inference became the cost lesson.
Local Azure ML logs show PyTorch, CUDA, and NVIDIA A100 80GB PCIe runtime evidence.
The cost API can break the research-period VM spend into meters, but it does not directly attach billed euros to each Azure ML run ID. For reviewer accuracy, this page shows meter-level cost plus job runtime evidence instead of pretending the portal gives clean per-job billing.
Primary Azure VM meter behind the gpu-a100 benchmark and training work.
Small supporting VM meter surfaced in the same East US 2 cost query.
qwen3_30b_a3b benchmark run on Azure ML; this is the main expensive slow-inference evidence.
Qwen3-14B run across six RAG levels for 180 prompt-generation combinations.
QLoRA fine-tuning run for the dense-vs-MoE comparison.
Some runs failed, while others were intentionally stopped and resumed with the next job at the planned continuation point.
This is the technical layer behind the case study: not just an app wrapper, but a research workflow using fine-tuning, retrieval, benchmarks, and judged evaluation.
Used to patch Qwen-family training and inference paths for faster fine-tuning and lower VRAM pressure on Azure ML GPU runs.
Fine-tuned compact adapters for Qwen variants instead of retraining full base models, keeping experiments feasible on limited credits.
Ran generation and evaluation workloads on Azure ML gpu-a100 compute, including the long MoE benchmark that exposed inference cost.
Compared six retrieval levels and found simple Pinecone retrieval more useful than heavier orchestration for this prompt-generation path.
Used judged outputs and re-judging passes to compare model, prompt format, and strategy quality while tracking judge-bias risks.
Some Azure ML jobs were stopped intentionally and resumed in follow-up jobs, so not every non-completed job represents a broken experiment.
This separates benchmark infrastructure from app hosting so reviewers do not confuse the research environment with the product runtime.
Current-build view based on the local PromptTriage source: Supabase Auth and Stripe sit in the app layer, FastAPI handles retrieval, Pinecone stores prompt vectors, and GitHub Actions deploys containers to Azure.
The table follows the case-study template and separates risks, implemented controls, and current evidence sources.
| Risk | Control | Evidence |
|---|---|---|
| Unauthenticated use of paid or rate-limited workflows | Supabase Auth sessions are checked by server routes before analyze, refine, and system-prompt generation. | Local source: Supabase middleware and API route auth checks. |
| Subscription state drift between payment and app access | Stripe checkout, billing portal, and webhooks write subscription state into Supabase. | Local source: Stripe webhook and subscription routes. |
| Secret leakage from deployment pipeline | Azure OIDC is used for GitHub Actions deployment, with runtime secrets stored outside source code. | Local source: deploy-frontend.yml and deploy-backend.yml. |
| Vector database or model provider exposure | Pinecone and provider keys are server-side environment variables; the browser only talks to application routes. | Local source: backend RAG service and environment examples. |
The implementation evidence now points to Azure Container Apps rather than the older deployment notes.
The case study follows the condensed professional template already documented in this site repo. The PromptTriage repository also follows the expected professional repo pattern, with the remaining work clearly labeled as evidence polish rather than hidden gaps.
The page includes the required sequence: executive summary, problem, constraints, requirements, architecture, security, deployment, operations, cost, results, tradeoffs, failures, next improvements, links, interview explanation, and resume bullets.
PromptTriage has README, deployment notes, security policy, research progress, benchmark JSON, Azure ML logs, and release assets. The remaining gap is public-safe screenshots and a runbook rather than missing source evidence.
The page now separates downstream quality, RAG orchestration, model/runtime selection, and prompt-length/token-efficiency metrics so reviewers can see exactly what each percentage describes.
These are the items from the professional case-study template that still need more source material before the PromptTriage page becomes fully audit-ready.
Job-level Azure ML cost allocation that maps run IDs to billed spend. Current evidence itemizes the EUR 178.12 VM meters, but Azure Cost Management does not directly attach every euro to each ML job.
Public-safe screenshots or summaries for monitoring, alerts, rollback, incident response, and restore paths.
Screenshots or redacted config evidence for Supabase Auth, Stripe webhook verification, secrets, and deployment identity.
Visual evidence for Azure Container Apps, ACR, Google Cloud Run history, GitHub Actions runs, image scanning, and rollback.
Exportable SVG is now included; optional PNG, Excalidraw, or Draw.io source can be added if you want a diagram asset that is easier to reuse in decks.
The public benchmark-impact table is now included. A future polish pass can attach the exact benchmark JSON/CSV exports as download assets next to the PDF.
PromptTriage is a RAG-powered prompt engineering platform built to turn rough ideas into production-ready prompts across text, image, video, and system-prompt workflows. The current build combines a Next.js interface, Supabase Auth, Stripe subscription flows, FastAPI retrieval services, Pinecone vector search, modality-specific metaprompts, and Azure Container Apps deployment. The strongest evidence is not only the app surface, but the documented research process behind retrieval strategy, prompt format, model selection, evaluation bias, and cost tradeoffs.
Most prompt tools behave like thin wrappers over a model API. The project needed to show whether structured prompt generation, retrieval context, and modality-specific templates could produce better, more reliable prompts than generic chat UX. The user problem is practical: builders need reusable prompts that include constraints, output format, assumptions, and evaluation criteria without manually writing every section from scratch.
The project was developed as a solo engineering effort with limited budget, evolving model availability, provider API instability, GPU quota constraints, and a need to publish evidence without exposing secrets or private datasets. Most benchmark and training work ran on Azure ML compute clusters, while the app moved through Azure Container Apps and Google Cloud Run deployment phases. Research notes show cost-sensitive experimentation across fine-tuned Qwen variants, RAG strategies, and LLM-as-judge workflows.
Functional requirements included prompt analysis, dynamic clarification questions, modality routing, retrieval from a prompt corpus, final prompt synthesis, one-click iteration, paid-plan capability checks, and optional current-documentation lookup through MCP tooling. Non-functional requirements included secure authentication, environment-based secrets, responsive UI, reproducible prompt versions, structured logging, and a containerized path from GitHub to Azure runtime. Research requirements included Unsloth-based QLoRA experiments, Azure ML benchmark jobs, and repeatable evaluation outputs.
The architecture separates the user-facing workflow from retrieval and prompt-engineering services. The UI collects intent and modality, Next.js API routes enforce Supabase session and subscription context, the backend retrieves relevant examples from Pinecone, and modality-specific metaprompts synthesize the final result. Azure Container Apps hosts the containerized frontend and backend services through images built and pushed by GitHub Actions.
The current security model centers on Supabase Auth, server-side subscription checks, Stripe webhook verification, environment-based secrets, Azure OIDC deployment, and no publication of private provider keys or vector indexes. The risk/control table below is intentionally evidence-oriented so reviewers can see which controls are implemented and which supporting artifacts still need screenshots.
The local workflows document a frontend and backend Docker deployment path to Azure Container Apps through Azure Container Registry. Historically, PromptTriage also ran on Google Cloud Run. The Azure app runtime and ACR were later accidentally removed during CLI cleanup, which is now part of the operational lessons learned rather than hidden from the case study.
Operational evidence includes API logging, health routes, error feedback UX, dataset ingestion scripts, research progress notes, and failure tracking around model/provider behavior. The next operational step is to attach Azure Container Apps logs, alert thresholds, incident response notes, rollback proof, and cost-monitoring screenshots to make the operations section audit-ready.
The Azure cost screenshot shows EUR 178.12 of credits spent for the PromptTriage research period, grouped under Virtual Machines in East US 2 for the qwentrain workspace. Azure ML job metadata shows the longest completed MoE benchmark ran for 38.19 hours on the gpu-a100 compute target, while Study A ran for 5.35 hours. The main cost lesson is that benchmark design, retry behavior, and inference implementation can matter more than raw model size.
Documented results include a 28K+ prompt evidence base, Pinecone retrieval, multiple completed studies, qwen3_14b selected as a production candidate in the research notes, Unsloth/QLoRA model experiments, and a finding that naive RAG performed better than more complex RAG variants for this pipeline. The strongest public metrics are task-specific: up to +13.9% relative instruction-following improvement versus a generic prompt, up to +12.3% downstream quality improvement versus no system prompt, and +19.7% quality for short structured prompts versus long prompts while using 86.7% fewer words. Some downstream benchmark results were nuanced, so the case study preserves both positive findings and null results instead of overstating uniform gains.
The main tradeoff was simplicity versus orchestration complexity. Naive RAG was easier to operate and outperformed more agentic retrieval variants in the documented study, while complex corrective or judge-based RAG added noise. The project also trades broad provider coverage for a more controlled, research-backed prompt workflow.
The research logs are valuable because they document failures and controlled interruptions: judge bias, provider/model availability issues, expensive slow MoE inference, failed Azure ML runs, planned stop/resume benchmark jobs, overfitting on small datasets, prompt strategies that did not move downstream scores, and accidental cloud-resource teardown. These failures make the project stronger as engineering evidence because they show measurement discipline rather than only demo polish.
Next improvements should include a formal runbook, security model page, threat model, Azure cost report, architecture diagram export stored in the repo, API rate limiting, clearer public benchmark reports, and a more complete deployment checklist. Those additions align directly with the documentation standard from the career execution pack.
The public demo has historically been hosted at prompttriage.kaelux.dev and the repository is available on GitHub. The strongest supporting evidence is the README, backend research notes, Azure ML logs, benchmark output JSON, deployment workflows, security policy, release charts, and the marketing-oriented app showcase video.
I built PromptTriage because prompt quality is usually treated as subjective, but production AI systems need measurable prompt structure, retrieval context, and evaluation discipline. The architecture uses a Next.js interface, Supabase Auth, Stripe billing, API orchestration, FastAPI retrieval, Pinecone vectors, Azure Container Apps, and modality-specific metaprompts. The most important lesson was that more complex RAG is not automatically better: in this project, simple retrieval produced stronger documented results than agentic retrieval traces.
Designed and benchmarked PromptTriage, a RAG-powered prompt engineering platform, across local research studies covering prompt strategy, RAG complexity, format sensitivity, model selection, and judge bias. Demonstrated up to +13.9% relative instruction-following improvement versus generic prompts, up to +12.3% downstream quality lift versus no system prompt, and short-prompt configurations that scored +19.7% higher while using 86.7% fewer words. Documented cost, failure modes, and tradeoffs to make the project inspectable by engineering reviewers.
The main case-studies page separates PromptTriage from Kaelux.dev, ViperMesh, and n8n Automation Atlas so each project can grow into its own structured page.