Free Practice Questions for Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam

Comprehensive and Detailed Explanation From Exact Extract:

The Spark UI is the standard and most effective way to inspect executor logs, task time, input size, and shuffles.

From the Databricks documentation:

“You can monitor job execution via the Spark Web UI. It includes detailed logs and metrics, including task-level execution time, shuffle reads/writes, and executor memory usage.”

(Source: Databricks Spark Monitoring Guide)

Option A is incorrect: logs are not guaranteed to be in/tmp, especially in cloud environments.

B.—verbosehelps during job submission but doesn't give detailed executor logs.

D.spark-sqlis a CLI tool for running queries, not for inspecting logs.

Hence, the correct method is using the Spark UI → Stages tab → Executor logs.

Comprehensive and Detailed Explanation From Exact Extract:

The correct output mode for streaming aggregations that need to output the full updated results at each trigger is"complete".

From the official documentation:

“complete: The entire updated result table will be output to the sink every time there is a trigger.”

This is ideal for aggregations, such as counts or averages grouped by a key, where the result table changes incrementally over time.

append: only outputs newly added rows

replace and aggregate: invalid values for output mode

Comprehensive and Detailed Explanation From Exact Extract:

The question requires creating a dictionary where keys areregionvalues and values are the correspondingregion_idintegers. Furthermore, it asks to retrieve only the smallest 3region_idvalues.

Key observations:

select('region', 'region_id')puts the column order as expected bydict()— where the first column becomes the key and the second the value.

sort('region_id')ensures sorting in ascending order so the smallest IDs are first.

take(3)retrieves exactly 3 rows.

Wrapping the result indict(...)correctly builds the required Python dictionary:{ 'AFRICA': 0, 'AMERICA': 1, 'ASIA': 2 }.

Incorrect options:

Option B flips the order toregion_idfirst, resulting in a dictionary with integer keys — not what's asked.

Option C uses.limit(3)without sorting, which leads to non-deterministic rows based on partition layout.

Option D sorts in descending order, giving the largest rather than smallestregion_ids.

Hence, Option A meets all the requirements precisely.

Comprehensive and Detailed Explanation From Exact Extract:

To extract unique values from a column and sort them alphabetically:

distinct()is required to remove duplicate values.

orderBy()is needed to sort the results alphabetically (ascending by default).

Correct code:

df.select("Name").distinct().orderBy(df["Name"])

This is directly aligned with standard DataFrame API usage in PySpark, as documented in the official Databricks Spark APIs. Option A is incorrect because it may not remove duplicates. Option C omits sorting. Option D sorts in descending order, which doesn't meet the requirement for alphabetical (ascending) order.

Comprehensive and Detailed Explanation:

Adaptive Query Execution (AQE) is a Spark 3.x feature that dynamically optimizes query plans at runtime. One of its core features is:

Dynamically switching join strategies (e.g., from sort-merge to broadcast) based on runtime statistics.

Other AQE capabilities include:

Coalescing shuffle partitions

Skew join handling

Option A is correct.

Option B refers to statistics collection, which is not AQE's primary function.

Option C is too broad and not AQE-specific.

Option D refers to Delta Lake optimizations, unrelated to AQE.

Final Answer: A

Comprehensive and Detailed Explanation From Exact Extract:

The correct approach to perform a parallelizedgroupByoperation across Spark worker nodes using Pandas API is viaapplyInPandas. This function enables grouped map operations using Pandas logic in a distributed Spark environment. It applies a user-defined function to each group of data represented as a Pandas DataFrame.

As per the Databricks documentation:

"applyInPandas()allows for vectorized operations on grouped data in Spark. It applies a user-defined function to each group of a DataFrame and outputs a new DataFrame. This is the recommended approach for using Pandas logic across grouped data with parallel execution."

Option A is correct and achieves this parallel execution.

Option B (mapInPandas) applies to the entire DataFrame, not grouped operations.

Option C uses built-in aggregation functions, which are efficient but not customizable with Pandas logic.

Option D creates a scalar Pandas UDF which does not perform a group-wise transformation.

Therefore, to run agroupBywith parallel Pandas logic on Spark workers, Option A usingapplyInPandasis the only correct answer.

Comprehensive and Detailed Explanation From Exact Extract:

df.cache()is shorthand fordf.persist(StorageLevel.MEMORY_AND_DISK)

df.persist()allows specifying any storage level such asMEMORY_ONLY,DISK_ONLY,MEMORY_AND_DISK_SER, etc.

By default,persist()usesMEMORY_AND_DISK, unless specified otherwise.

Comprehensive and Detailed Explanation From Exact Extract:

Operations that trigger data movement across partitions (like groupBy, join, repartition) result in a shuffle and a new stage.

From Spark documentation:

“groupBy and aggregation cause data to be shuffled across partitions to combine rows with the same key.”

Option A (groupBy + agg) → causes shuffle.

Options B, C, and D (filter, withColumn, select) → transformations that do not require shuffling; they are narrow dependencies.

Final Answer: A

Pandas API on Spark (formerly Koalas) offers:

Familiar Pandas-like syntax

Distributed execution using Spark under the hood

Scalability for large datasets across the cluster

It provides the power of Spark while retaining the productivity of Pandas.

f.to_date() converts a timestamp or string to a DateType.

Ideal for extracting the date component (year-month-day) from a full timestamp.

Example:

frompyspark.sql.functionsimportto_date

dates_df.withColumn("date", to_date("timestamp"))