Free Practice Questions for Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam

Apache Spark employs a lazy evaluation model for transformations. This means that when transformations (e.g., map(), filter()) are applied to a DataFrame, Spark does not execute them immediately. Instead, it builds a logical plan (lineage) of transformations to be applied.

Execution is deferred until an action (e.g., collect(), count(), save()) is called. At that point, Spark's Catalyst optimizer analyzes the logical plan, optimizes it, and then executes the physical plan to produce the result.

This lazy evaluation strategy allows Spark to optimize the execution plan, minimize data shuffling, and improve overall performance by reducing unnecessary computations.

In Apache Spark, broadcast variables are used to efficiently distribute large, read-only data to all worker nodes. However, broadcasting very large datasets can lead to memory issues on executors if the data does not fit into the available memory.

According to the Spark documentation:

"Broadcast variables allow the programmer to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks. This can greatly reduce the amount of data sent over the network."

However, it also notes:

"Using the broadcast functionality available in SparkContext can greatly reduce the size of each serialized task, and the cost of launching a job over a cluster. If your tasks use any large object from the driver program inside of them (e.g., a static lookup table), consider turning it into a broadcast variable."

But caution is advised when broadcasting large datasets:

"Broadcasting large variables can cause out-of-memory errors if the data does not fit in the memory of each executor."

Therefore, if the broadcasted DataFrame containing millions of rows exceeds the memory capacity of the executors, the job may fail due to memory constraints.

To process data in fixed micro-batch intervals, use the .trigger(processingTime="interval") option in Structured Streaming.

Correct usage:

query = df.writeStream \

outputMode("append") \

trigger(processingTime="5 seconds") \

start()

This instructs Spark to process available data every 5 seconds.

Why the other options are incorrect:

B: continuous triggers are for continuous processing mode (different execution model).

C: once=True runs the stream a single time (batch mode).

D: Default trigger runs as fast as possible, not fixed intervals.

To sort a Spark DataFrame by multiple columns, use .orderBy() (or .sort()) with column expressions.

Correct syntax for descending and ascending mix:

from pyspark.sql.functions import col

df1.orderBy(col("count").desc(), col("Name").asc())

This sorts primarily by count in descending order and secondarily by Name in ascending order (alphabetically).

Why the other options are incorrect:

B/C: Default sort order is ascending; won’t place highest counts first.

D: Reverses sorting logic — sorts Name descending, not required.

The correct output mode for streaming aggregations that need to output the full updated results at each trigger is "complete".

From the official documentation:

“complete: The entire updated result table will be output to the sink every time there is a trigger.”

This is ideal for aggregations, such as counts or averages grouped by a key, where the result table changes incrementally over time.

append: only outputs newly added rows

replace and aggregate: invalid values for output mode

The default broadcast join threshold in Spark is:

spark.sql.autoBroadcastJoinThreshold = 10MB

Since df2 is only 8 MB (less than 10 MB), Spark will automatically apply a broadcast join without requiring explicit hints.

From the Spark documentation:

“If one side of the join is smaller than the broadcast threshold, Spark will automatically broadcast it to all executors.”

A is incorrect because Spark does support auto broadcast even with static plans.

B is correct: Spark will automatically broadcast df2.

C and D are incorrect because Spark's default logic handles this optimization.

Final Answer: B

To compare a column value with a Python literal constant in a DataFrame expression, use F.lit() to convert it into a Spark literal.

Correct refactor:

from pyspark.sql import functions as F

min_price = 110.50

result_df = prices_df.filter(F.col("price") > F.lit(min_price)).agg(F.count("*"))

This avoids type mismatches and ensures Spark executes the filter expression on the cluster.

Why the other options are incorrect:

B: where() syntax is valid, but F.lit("price") is incorrect — wraps string literal, not a column.

C: withColumn adds a column, not needed for this aggregation.

D: Comparison logic reversed.

Spark Connect enables remote execution of Spark jobs by decoupling the client from the driver using the Spark Connect protocol (gRPC).

It allows users to run Spark code from different environments (like notebooks, IDEs, or remote clients) while executing jobs on the cluster.

Key Features:

Enables remote interaction between client and Spark driver.

Supports interactive development and lightweight client sessions.

Improves developer productivity without needing driver resources locally.

Why the other options are incorrect:

A: Spark Connect is not limited to ingestion tasks.

B: It allows multi-language clients (Python, Scala, etc.) but runs via Spark Connect API, not arbitrary remote code.

C: Uses gRPC protocol, not REST.

When using the MEMORY_AND_DISK storage level, Spark attempts to cache as much of the DataFrame in memory as possible. If the DataFrame does not fit entirely in memory, Spark will store the remaining partitions on disk. This allows processing to continue, albeit with a performance overhead due to disk I/O.

As per the Spark documentation:

"MEMORY_AND_DISK: It stores partitions that do not fit in memory on disk and keeps the rest in memory. This can be useful when working with datasets that are larger than the available memory."

— Perficient Blogs: Spark - StorageLevel

This behavior ensures that Spark can handle datasets larger than the available memory by spilling excess data to disk, thus preventing job failures due to memory constraints.