At IBM I built a series of generative-AI proofs of concept for enterprise clients. Each one started from a concrete business problem and had to work on the client's own messy data, not a clean demo set, before anyone would consider taking it to production.
The largest was a retrieval-augmented assistant that could answer questions across every kind of document a company keeps. Alongside it I built natural-language-to-code tools, code translation between languages, an aspect-level sentiment pipeline for course feedback, and multi-step agents that tied these capabilities together.
POC 01
A document assistant that reads everything
Employees were spending a large part of their day hunting for answers spread across PDFs, slide decks, spreadsheets, wiki pages, email threads and recorded meetings. Keyword search failed because the answer was often in a table, a scanned page or a slide's speaker notes.
The goal was a chat assistant that answers in plain language, cites the exact page, slide or timestamp it used, respects who is allowed to see which document, and says so when it does not know.
Ingestion: one pipeline, every format
The quality of a RAG system is decided at ingestion. A format router inspects each file and sends it to the parser best suited to it. Every parser outputs the same normalized Markdown, so everything downstream is format-agnostic.
How each document type is handled
| Source | How it is parsed | What it becomes |
|---|---|---|
| Native PDF | Docling layout analysis for reading order, headings and figures; PyMuPDF / pdfplumber as a fast fallback when layout is simple. | Markdown sections with page numbers and a heading path. |
| Scanned PDF, photos of documents | OpenCV deskew, denoise and binarize, then OCR (Tesseract or EasyOCR through Docling). Pages below a confidence threshold are flagged instead of silently indexed. | Text with per-page OCR confidence in metadata. |
| Tables, in any file | Docling's table-structure model; Camelot for ruled PDF tables. Merged header cells are flattened into column names. | A Markdown table plus one sentence per row, so a single row is retrievable on its own. |
| Word (.docx) | python-docx / Docling, keeping heading styles, numbered lists, footnotes and comments. | A section tree that drives chunk boundaries. |
| PowerPoint (.pptx) | python-pptx for slide titles, body text, tables and speaker notes; slide images go to the vision step. | One chunk per slide, with its notes attached. |
| Excel / CSV | pandas and openpyxl, with header-row detection and per-sheet profiling (column types, ranges). | Sheet summaries for search. Numeric questions are routed to the NL-to-code path instead of RAG. |
| HTML, wiki and intranet pages | Trafilatura / BeautifulSoup to strip navigation and boilerplate while keeping heading hierarchy. | Clean text with the source URL for citation. |
| Email (.eml / .msg) | Python's email parser and extract-msg; quoted replies are de-duplicated across a thread and attachments are sent back through the router. | Message chunks with sender, date and thread metadata. |
| Charts, diagrams, figures | A vision-language model on watsonx.ai describes the figure, using the surrounding caption as context. | A text description linked back to the page it came from. |
| Audio and video | Speech-to-text for the transcript, plus frame sampling and captioning, the approach from my prior-art publication. | Timestamped transcript chunks that cite the exact moment. |
Chunking: follow the document's own structure
I compared several chunking strategies on the same question set rather than picking one up front:
- Fixed-size with overlap: the baseline. Easy, but it splits tables and procedures mid-step.
- Recursive, heading-aware: splits on headings first, then paragraphs, then sentences, so a chunk rarely crosses a topic boundary.
- Semantic: breaks where embedding similarity between neighbouring sentences drops. Useful for long unstructured transcripts.
- Parent-child: small chunks are embedded for precise matching, but the model receives the surrounding parent section. This gave the best answers on policy-style documents.
- Table-aware: a table is never split, and each row also exists as its own sentence-style chunk.
Every chunk carries metadata: source file, page or slide, heading path, document date, format, and the access-control groups copied from the source system.
Retrieval: hybrid, filtered, reranked
Pure vector search misses exact identifiers such as part numbers, policy codes and names, while pure keyword search misses paraphrases. I ran both in Elasticsearch and merged them:
- Rewrite follow-up questions into standalone queries, and expand vague ones into a few alternative phrasings.
- Run BM25 and dense kNN search in parallel, both filtered by the user's access groups and any date or document-type constraints in the question.
- Merge the two result lists with reciprocal rank fusion.
- Rerank the top candidates with a cross-encoder and keep only the best handful.
- Expand each winner to its parent section and compress away sentences that do not relate to the question.
For quick local prototypes I used ChromaDB, and I evaluated Milvus as an alternative vector store. Elasticsearch won for production because the clients already ran it and it handled keyword, vector and filtering in one query.
Answering: grounded or not at all
The prompt hands the model numbered sources and instructs it to answer only from them and cite each claim. After generation, a check confirms every citation points to a retrieved chunk. If it does not, or if retrieval scores were too low to begin with, the assistant declines and shows the nearest documents instead of guessing. Models were served on watsonx.ai, which made it easy to compare IBM Granite against open models such as Llama and Mixtral on the same pipeline.
Measuring it instead of eyeballing it
With subject-matter experts I built a golden set of real questions and expected sources. Every pipeline change ran against it, scored with RAGAS-style metrics: context precision and recall for retrieval, faithfulness and answer relevance for generation. An LLM judge handled scale, and human reviewers spot-checked its verdicts. User thumbs-up and thumbs-down, logged with the full trace, fed new hard cases back into the set.
POC 02
Natural language to code
Not every question belongs in RAG. "What were repeat orders by region last quarter?" is a database query. This POC let business users ask such questions in plain English and get back a result, a chart, and the code that produced it.
The model never sees the whole database. Table and column descriptions are indexed like documents, so only the relevant slice of the schema goes into the prompt, together with a few similar, previously verified question-and-SQL pairs. Generated SQL is validated with EXPLAIN before it runs, executed under a read-only role with row limits and timeouts, and automatically repaired from the error message when validation fails. The same pattern also generated pandas code for analysis over uploaded spreadsheets.
POC 03
Translating code from one language to another
Clients with ageing codebases wanted to know whether an LLM could carry logic from a legacy language into a modern one. Translating a whole file in one prompt produces code that looks right and quietly behaves differently, so I treated it as an engineering pipeline:
- Parse the source into functions and classes and build a dependency graph, then translate leaves first so callers can see the already-translated signatures.
- Translate one unit at a time, with its dependencies' interfaces and a style guide for the target language in context.
- Generate tests from the original code's behaviour, then run the original and the translation on the same inputs and compare outputs.
- Feed compiler errors and failing cases back to the model for a bounded number of repair rounds, and flag anything still failing for a developer.
The point was not to claim full automation. It was to show how much of a migration could be done safely, with tests as evidence, and to hand developers a precise list of what still needed human attention.
POC 04
Reading the logic behind course feedback
An education client had thousands of free-text student comments about professors. A single positive or negative score was useless, because one comment can praise a professor's clarity and criticise their grading in the same sentence.
I built an aspect-based pipeline instead. Each comment is split into statements, each statement is mapped to an aspect (teaching clarity, course organisation, workload, grading fairness, approachability), and each aspect gets a sentiment with a one-line justification quoted from the text. The model returns strict JSON that is validated against a schema, so the output can be aggregated reliably.
I benchmarked this LLM approach against classic baselines (lexicon-based scoring and a fine-tuned transformer classifier) on a hand-labelled sample. The LLM was clearly better on mixed, sarcastic and comparative comments, which are exactly the ones that matter. Student identifiers were removed before any text reached a model, and results were only reported per professor and course, never per student.
POC 05
Agents that combine the pieces
The final step was letting a single assistant decide which of these capabilities a request needs. I built agent workflows in LangGraph for stateful, inspectable control flow, and used CrewAI for role-based crews (researcher, writer, reviewer) where tasks benefited from that division of labour. Enterprise systems such as ticketing and CRM were exposed to agents through MCP servers, so tools could be added without changing agent code.
Three rules kept the agents trustworthy: every step is traced and visible to the user, loops have hard limits on iterations and cost, and anything that changes data in a real system requires a human to approve it first.
Common ground
From proof of concept to something deployable
Each POC shipped as a FastAPI service with streaming responses, containerised with Docker and deployed on Kubernetes, so a successful demo was already close to a production shape. Prompts were versioned like code, every pipeline had an evaluation set, and every request produced a trace. That made it possible to show clients not only that a system worked, but how well, on their own data, and what it would take to run it.
Technology
- Models
- IBM Granite, Llama, Mixtral and embedding models on watsonx.ai; cross-encoder rerankers; vision-language models for figures
- Parsing
- Docling, PyMuPDF, pdfplumber, Tesseract / EasyOCR, OpenCV, Camelot, python-docx, python-pptx, pandas, openpyxl, Trafilatura, BeautifulSoup, speech-to-text
- Retrieval
- Elasticsearch (BM25 + dense vectors), ChromaDB, Milvus, reciprocal rank fusion
- Orchestration
- LangGraph, CrewAI, MCP, LangChain
- Evaluation
- Golden question sets, RAGAS-style metrics, LLM-as-judge with human review, A/B prompt testing
- Serving
- Python, FastAPI, Flask, Docker, Kubernetes, React, Streamlit
Client names, data and internal details are confidential. The examples on this page are representative of the work rather than copies of client material.