Generative AI Engineer Databricks-Generative-AI-Engineer-Associate Databricks Study Notes

When deploying an agent via the Mosaic AI Agent Framework (which leverages Databricks Model Serving), operational metrics are captured automatically by default. These include system-level telemetry such as the number of requests per second (volume), the time taken for the model to respond (latency), and the rate of 4xx/5xx HTTP errors. These are essential for monitoring Service Level Agreements (SLAs). However, Quality metrics (B), such as correctness, groundedness, or adherence to custom guidelines, cannot be determined " automatically " by the serving infrastructure because they require either human feedback or an LLM-as-a-judge evaluation (using Databricks Agent Evaluation). While Databricks makes it easy to generate quality metrics using the mlflow.evaluate API or the inference table, they are not " default operational metrics " that appear without additional evaluation configuration.

When deploying an LLM application for customer service inquiries, the primary focus is on measuring the operational efficiency and quality of the responses. Here ' s why A is the correct metric:

Number of customer inquiries processed per unit of time : This metric tracks the throughput of the customer service system, reflecting how many customer inquiries the LLM application can handle in a given time period (e.g., per minute or hour). High throughput is crucial in customer service applications where quick response times are essential to user satisfaction and business efficiency.

Real-time performance monitoring : Monitoring the number of queries processed is an important part of ensuring that the model is performing well under load, especially during peak traffic times. It also helps ensure the system scales properly to meet demand.

Why other options are not ideal:

B. Energy usage per query : While energy efficiency is a consideration, it is not the primary concern for a customer-facing application where user experience (i.e., fast and accurate responses) is critical.

C. Final perplexity scores for the training of the model : Perplexity is a metric for model training, but it doesn ' t reflect the real-time operational performance of an LLM in production.

D. HuggingFace Leaderboard values for the base LLM : The HuggingFace Leaderboard is more relevant during model selection and benchmarking. However, it is not a direct measure of the model ' s performance in a specific customer service application in production.

Focusing on throughput (inquiries processed per unit time) ensures that the LLM application is meeting business needs for fast and efficient customer service responses.

A managed Databricks MCP server provides tools for accessing Databricks services without requiring the team to host an external server. For structured table queries, a suitable managed SQL or Genie tool uses Databricks-governed data access. Unity Catalog remains responsible for authorization instead of delegating security decisions to the language model. When permissions must reflect the individual requester, the application must use the appropriate user authorization configuration so row filters and column masks are evaluated for that identity. Passing credentials in prompts exposes secrets unnecessarily. A self-managed virtual machine violates the infrastructure requirement. Direct JDBC access also fails the requested MCP architecture and introduces custom integration work. Managed hosting simplifies deployment while preserving the platform’s access-control model. Databricks tools documentation , authentication documentation