Helping Hand Questions for Databricks-Generative-AI-Engineer-Associate

The most effective way to preprocess prompts using custom code is to write a custom model, such as an MLflow PyFunc model . Here’s a breakdown of why this is the correct approach:

MLflow PyFunc Models : MLflow is a widely used platform for managing the machine learning lifecycle, including experimentation, reproducibility, and deployment. A PyFunc model is a generic Python function model that can implement custom logic, which includes preprocessing prompts.

Preprocessing Prompts : Preprocessing could include various tasks like cleaning up the user input, formatting it according to specific rules, or augmenting it with additional context before passing it to the LLM. Writing this preprocessing as part of a PyFunc model allows the custom code to be managed, tested, and deployed easily.

Modular and Reusable : By separating the preprocessing logic into a PyFunc model, the system becomes modular, making it easier to maintain and update without needing to modify the core LLM or retrain it.

Why Other Options Are Less Suitable :

A (Modify LLM’s Internal Architecture) : Directly modifying the LLM ' s architecture is highly impractical and can disrupt the model’s performance. LLMs are typically treated as black-box models for tasks like prompt processing.

B (Avoid Custom Code) : While it’s true that LLMs haven ' t been explicitly trained with preprocessed prompts, preprocessing can still improve clarity and alignment with desired input formats without confusing the model.

C (Postprocessing Outputs) : While postprocessing the output can be useful, it doesn ' t address the need for clean and well-formatted inputs, which directly affect the quality of the model ' s responses.

Thus, using an MLflow PyFunc model allows for flexible and controlled preprocessing of prompts in a scalable way, making it the most effective method.

To design an LLM-based application that can answer employee HR questions using HR PDF documentation, the most effective approach is option D. Here ' s why:

Chunking and Vector Store Embedding : HR documentation tends to be lengthy, so splitting it into smaller, manageable chunks helps optimize retrieval. These chunks are then embedded into a vector store (a database that stores vector representations of text). Each chunk of text is transformed into an embedding using a transformer-based model, which allows for efficient similarity-based retrieval.

Using Vector Search for Retrieval : When an employee asks a question, the system converts their query into an embedding as well. This embedding is then compared with the embeddings of the document chunks in the vector store. The most semantically similar chunks are retrieved, which ensures that the answer is based on the most relevant parts of the documentation.

LLM to Generate a Response : Once the relevant chunks are retrieved, these chunks are passed into the LLM, which uses them as context to generate a coherent and accurate response to the employee ' s question.

Why Other Options Are Less Suitable :

A (Calculate Averaged Embeddings) : Averaging embeddings might dilute important information. It doesn ' t provide enough granularity to focus on specific sections of documents.

B (Summarize HR Documentation) : Summarization loses the detail necessary for HR-related queries, which are often specific. It would likely miss the mark for more detailed inquiries.

C (Interaction Matrix and ALS) : This approach is better suited for recommendation systems and not for HR queries, as it’s focused on collaborative filtering rather than text-based retrieval.

Thus, option D is the most effective solution for providing precise and contextual answers based on HR documentation.

The problem describes a Retrieval-Augmented Generation (RAG) system for HR policy Q & A where responses are incomplete and unstructured due to the retriever failing to return sufficient context. The engineer has already tried different embedding and response-generating LLMs without success, suggesting the issue lies in the retrieval process—specifically, how documents are chunked and indexed. Let’s evaluate the options.

Option A: Add the section header as a prefix to chunks

Adding section headers provides additional context to each chunk, helping the retriever understand the chunk’s relevance within the document structure (e.g., “Leave Policy: Annual Leave” vs. just “Annual Leave”). This can improve retrieval precision for structured HR policies.

Databricks Reference : " Metadata, such as section headers, can be appended to chunks to enhance retrieval accuracy in RAG systems " ( " Databricks Generative AI Cookbook, " 2023).

Option B: Increase the document chunk size

Larger chunks include more context per retrieval, reducing the chance of missing relevant information split across smaller chunks. For structured HR policies, this can ensure entire sections or rules are retrieved together.

Databricks Reference : " Increasing chunk size can improve context completeness, though it may trade off with retrieval specificity " ( " Building LLM Applications with Databricks " ).

Option C: Split the document by sentence

Splitting by sentence creates very small chunks, which could exacerbate the problem by fragmenting context further. This is likely why the current system fails—it retrieves incomplete snippets rather than cohesive policy sections.

Databricks Reference : No specific extract opposes this, but the emphasis on context completeness in RAG suggests smaller chunks worsen incomplete responses.

Option D: Use a larger embedding model

A larger embedding model might improve vector quality, but the question states that experimenting with different embedding models didn’t help. This suggests the issue isn’t embedding quality but rather chunking/retrieval strategy.

Databricks Reference : Embedding models are critical, but not the focus when retrieval context is the bottleneck.

Option E: Fine tune the response generation model

Fine-tuning the LLM could improve response coherence, but if the retriever doesn’t provide complete context, the LLM can’t generate full answers. The root issue is retrieval, not generation.

Databricks Reference : Fine-tuning is recommended for domain-specific generation, not retrieval fixes ( " Generative AI Engineer Guide " ).

Conclusion : Options A and B address the retrieval issue directly by enhancing chunk context—either through metadata (A) or size (B)—aligning with Databricks’ RAG optimization strategies. C would worsen the problem, while D and E don’t target the root cause given prior experimentation.

The goal is to make fine-tuned LLMs from a production (prod) Databricks workspace available for testing in a development (dev) workspace, leveraging Unity Catalog and MLflow, while ensuring cost-effectiveness and security. Let’s analyze the options.

Option A: Use an external model registry which can be accessed from all workspaces

An external registry adds cost (e.g., hosting fees) and complexity (e.g., integration, security configurations) outside Databricks’ native ecosystem, reducing security compared to Unity Catalog’s governance.

Databricks Reference : " Unity Catalog provides a centralized, secure model registry within Databricks " ( " Unity Catalog Documentation, " 2023).

Option B: Setup a script to export the model from prod and import it to dev

Export/import scripts require manual effort, storage for model artifacts, and repeated execution, increasing operational cost and risk (e.g., version mismatches, unsecured transfers). It’s less efficient than a native solution.

Databricks Reference : Manual processes are discouraged when Unity Catalog offers built-in sharing: " Avoid redundant workflows with Unity Catalog’s cross-workspace access " ( " MLflow with Unity Catalog " ).

Option C: Setup a duplicate training pipeline in dev, so that an identical model is available in dev

Duplicating the training pipeline doubles compute and storage costs, as it retrains the model from scratch. It’s neither cost-effective nor necessary when the prod model can be reused securely.

Databricks Reference : " Re-running training is resource-intensive; leverage existing models where possible " ( " Generative AI Engineer Guide " ).

Option D: Use MLflow to log the model directly into Unity Catalog, and enable READ access in the dev workspace to the model

Unity Catalog, integrated with MLflow, allows models logged in prod to be centrally managed and accessed across workspaces with fine-grained permissions (e.g., READ for dev). This is cost-effective (no extra infrastructure or retraining) and secure (governed by Databricks’ access controls).

Databricks Reference : " Log models to Unity Catalog via MLflow, then grant access to other workspaces securely " ( " MLflow Model Registry with Unity Catalog, " 2023).

Conclusion : Option D leverages Databricks’ native tools (MLflow and Unity Catalog) for a seamless, cost-effective, and secure solution, avoiding external systems, manual scripts, or redundant training.