Free Practice Questions for Databricks Databricks-Generative-AI-Engineer-Associate Exam

When addressing concerns of hallucination and data leakage in an LLM application for internal company policies, fine-tuning the model on internal data with the hope it learns data boundaries can be problematic:

Risk of Data Leakage : Fine-tuning on sensitive or confidential data does not guarantee that the model will not inadvertently include or reference this data in its outputs. There’s a risk of overfitting to the specific data details, which might lead to unintended leakage.

Hallucination : Fine-tuning does not necessarily mitigate the model ' s tendency to hallucinate; in fact, it might exacerbate it if the training data is not comprehensive or representative of all potential queries.

Better Approaches:

A , C , and D involve setting up operational safeguards and constraints that directly address data leakage and ensure responses are aligned with specific user needs and security levels.

Fine-tuning lacks the targeted control needed for such sensitive applications and can introduce new risks, making it an unsuitable approach in this context.

According to Generative AI engineering standards for Retrieval-Augmented Generation (RAG), chunking strategy is a critical optimization variable. Setting the chunk size to match the model ' s maximum context window (e.g., 4k or 8k tokens) is a poor practice and should be discarded. Large chunks introduce significant " noise " into the LLM ' s context, as only a small portion of a massive chunk usually contains the answer to a specific query. This leads to the " lost in the middle " phenomenon where LLMs struggle to extract relevant information from bloated contexts. Furthermore, large chunks reduce the precision of the vector search. Standard best practices involve using smaller, semantically meaningful chunks (typically 256–512 tokens) with overlap to maintain context. In contrast, metadata filtering (B) is essential for narrowing searches to specific car years, fine-tuning embeddings (C) improves retrieval accuracy for domain-specific technical terms, and few-shot examples (D) guide the LLM ' s output format and tone.

The task is to benchmark LLMs for text translation using an evaluation set with known high-quality examples, requiring a performant metric. Let’s evaluate the options.

Option A: ROUGE metric

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures overlap between generated and reference texts, primarily for summarization. It’s less suited for translation, where precision and word order matter more.

Databricks Reference : " ROUGE is commonly used for summarization, not translation evaluation " ( " Generative AI Cookbook, " 2023).

Option B: BLEU metric

BLEU (Bilingual Evaluation Understudy) evaluates translation quality by comparing n-gram overlap with reference translations, accounting for precision and brevity. It’s widely used, performant, and appropriate for this task.

Databricks Reference : " BLEU is a standard metric for evaluating machine translation, balancing accuracy and efficiency " ( " Building LLM Applications with Databricks " ).

Option C: NDCG metric

NDCG (Normalized Discounted Cumulative Gain) assesses ranking quality, not text generation. It’s irrelevant for translation evaluation.

Databricks Reference : " NDCG is suited for ranking tasks, not generative output scoring " ( " Databricks Generative AI Engineer Guide " ).

Option D: RECALL metric

Recall measures retrieved relevant items but doesn’t evaluate translation quality (e.g., fluency, correctness). It’s incomplete for this use case.

Databricks Reference : No specific extract, but recall alone lacks the granularity of BLEU for text generation tasks.

Conclusion : Option B (BLEU) is the best metric for translation evaluation, offering a performant and standard approach, as endorsed by Databricks’ guidance on generative tasks.

Problem Context : After receiving positive feedback for the RAG application prototype, the next step is to formally evaluate the system to pinpoint areas for improvement.

Explanation of Options :

Option A : While cosine similarity scores are useful, they primarily measure similarity rather than the overall performance of an RAG system.

Option B : This option provides a systematic approach to evaluation by testing both retrieval and generation components separately. This allows for targeted improvements and a clear understanding of each component ' s performance, using MLflow’s metrics for a structured and standardized assessment.

Option C : Benchmarking multiple LLMs does not focus on evaluating the existing system’s components but rather on comparing different models.

Option D : Using an LLM as a judge is subjective and less reliable for systematic performance evaluation.

Option B is the most comprehensive and structured approach, facilitating precise evaluations and improvements on specific components of the RAG system.

 Context : Deploying an application that uses an MLflow Pyfunc model involves managing sensitive information such as secrets and credentials securely.

 Explanation of Options :

Option A: Use spark.conf.set() : While this method can pass configurations within Spark jobs, using it for secrets is not recommended because it may expose them in logs or Spark UI.

Option B: Pass variables using the Databricks Feature Store API : The Feature Store API is designed for managing features for machine learning, not for handling secrets or credentials.

Option C: Add credentials using environment variables : This is a common practice for managing credentials in a secure manner, as environment variables can be accessed securely by applications without exposing them in the codebase.

Option D: Pass the secrets in plain text : This is highly insecure and not recommended, as it exposes sensitive information directly in the code.

Therefore, Option C is the best method for securely passing secrets and credentials to an application, protecting them from exposure.

When deploying an LLM application for customer service inquiries, the primary focus is on measuring the operational efficiency and quality of the responses. Here ' s why A is the correct metric:

Number of customer inquiries processed per unit of time : This metric tracks the throughput of the customer service system, reflecting how many customer inquiries the LLM application can handle in a given time period (e.g., per minute or hour). High throughput is crucial in customer service applications where quick response times are essential to user satisfaction and business efficiency.

Real-time performance monitoring : Monitoring the number of queries processed is an important part of ensuring that the model is performing well under load, especially during peak traffic times. It also helps ensure the system scales properly to meet demand.

Why other options are not ideal:

B. Energy usage per query : While energy efficiency is a consideration, it is not the primary concern for a customer-facing application where user experience (i.e., fast and accurate responses) is critical.

C. Final perplexity scores for the training of the model : Perplexity is a metric for model training, but it doesn ' t reflect the real-time operational performance of an LLM in production.

D. HuggingFace Leaderboard values for the base LLM : The HuggingFace Leaderboard is more relevant during model selection and benchmarking. However, it is not a direct measure of the model ' s performance in a specific customer service application in production.

Focusing on throughput (inquiries processed per unit time) ensures that the LLM application is meeting business needs for fast and efficient customer service responses.

The task is to extend a cinema chatbot to provide movie showtime information using a RAG application, leveraging user location and a continuously updated Delta table, with minimal effort and high performance. Let’s evaluate the options.

Option A: Create a Feature Serving Endpoint from a FeatureSpec that references an online store synced from the Delta table. Query the Feature Serving Endpoint as part of the agent logic / tool implementation

Databricks Feature Serving provides low-latency access to real-time data from Delta tables via an online store. Syncing the Delta table to a Feature Serving Endpoint allows the chatbot to query showtimes efficiently, integrating seamlessly into the RAG agent’s tool logic. This leverages Databricks’ native infrastructure, minimizing effort and ensuring performance.

Databricks Reference : " Feature Serving Endpoints provide real-time access to Delta table data with low latency, ideal for production systems " ( " Databricks Feature Engineering Guide, " 2023).

Option B: Query the Delta table directly via a SQL query constructed from the user ' s input using a text-to-SQL LLM in the agent logic / tool

Using a text-to-SQL LLM to generate queries adds complexity (e.g., ensuring accurate SQL generation) and latency (LLM inference + SQL execution). While feasible, it’s less performant and requires more effort than a pre-built serving solution.

Databricks Reference : " Direct SQL queries are flexible but may introduce overhead in real-time applications " ( " Building LLM Applications with Databricks " ).

Option C: Write the Delta table contents to a text column, then embed those texts using an embedding model and store these in the vector index. Look up the information based on the embedding as part of the agent logic / tool implementation

Converting structured Delta table data (e.g., showtimes) into text, embedding it, and using vector search is inefficient for structured lookups. It’s effort-intensive (preprocessing, embedding) and less precise than direct queries, undermining performance.

Databricks Reference : " Vector search excels for unstructured data, not structured tabular lookups " ( " Databricks Vector Search Documentation " ).

Option D: Set up a task in Databricks Workflows to write the information in the Delta table periodically to an external database such as MySQL and query the information from there as part of the agent logic / tool implementation

Exporting to an external database (e.g., MySQL) adds setup effort (workflow, external DB management) and latency (periodic updates vs. real-time). It’s less performant and more complex than using Databricks’ native tools.

Databricks Reference : " Avoid external systems when Delta tables provide real-time data natively " ( " Databricks Workflows Guide " ).

Conclusion : Option A minimizes effort by using Databricks Feature Serving for real-time, low-latency access to the Delta table, ensuring high performance in a production-ready RAG chatbot.