Free Practice Questions for Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam

Comprehensive and Detailed Explanation From Exact Extract:

The correct way to load specific columns from an ORC file is to first load the file using.load()and then apply.select()on the resulting DataFrame. This is valid with.read.format("orc")or the shortcut.read.orc().

df = spark.read.format("orc").load("/file/test_data.orc").select("col1","col2")

Why others are incorrect:

Aperforms selection after filtering, but doesn’t match the intention to minimize memory at load.

Bincorrectly tries to use.select()before.load(), which is invalid.

Cuses a non-existent.selected()method.

Dcorrectly loads and then selects.

Comprehensive and Detailed Explanation From Exact Extract:

In Spark, when a CSV row does not match the provided schema, Spark does not raise an error by default. Instead, it returnsnullfor fields that cannot be parsed correctly.

In the first row,"hello"cannot be cast to Integer for theagefield → Spark setsage=None

In the second row,"20"is a valid integer →age=20

So the output will be:

[Row(name='bambi', age=None), Row(name='alladin', age=20)]

Final Answer: C

Comprehensive and Detailed Explanation From Exact Extract:

The message"GC overhead limit exceeded"typically indicates that the JVM is spending too much time in garbage collection with little memory recovery. This suggests that the driver or executor is under-provisioned in memory.

The most effective remedy is to increase the driver memory using:

--driver-memory 4g

This is confirmed in Spark's official troubleshooting documentation:

“If you see a lot ofGC overhead limit exceedederrors in the driver logs, it’s a sign that the driver is running out of memory.”

—Spark Tuning Guide

Why others are incorrect:

Amay help but does not directly address the driver memory shortage.

Bis not a valid action; GC cannot be disabled.

Dincreases memory usage, worsening the problem.

The most efficient and correct approach is to define a schema using StructType and pass it tospark.read.schema(...).

This avoids schema inference overhead and ensures proper data types are enforced during read.

Example:

frompyspark.sql.typesimportStructType, StructField, StringType, DoubleType

schema = StructType([

StructField("order_id", StringType(),True),

StructField("amount", DoubleType(),True),

])

df = spark.read.schema(schema).json("path/to/json")

— Source:Databricks Guide – Read JSON with predefined schema

Comprehensive and Detailed Explanation From Exact Extract:

The.coalesce(numPartitions)function is used to reduce the number of partitions in a DataFrame. It does not increase the number of partitions. If the specified number of partitions is greater than the current number, it will not have any effect.

From the official Spark documentation:

“coalesce() results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim one or more of the current partitions.”

However, if you try to increase partitions using coalesce (e.g., from 10 to 20), the number of partitions remains unchanged.

Hence,df.coalesce(20)will still return a DataFrame with 10 partitions.

Comprehensive and Detailed Explanation From Exact Extract:

In Spark’s client mode, the driver runs on the local machine that submitted the job. If that machine is resource-constrained (e.g., low memory), performance degrades.

From the Spark documentation:

"In cluster mode, the driver runs inside the cluster, benefiting from cluster resources and scalability."

Option A is incorrect — executors do not help the driver directly.

Option B might help short-term but does not scale.

Option C is correct — switching to cluster mode moves the driver to the cluster.

Option D (local mode) is for development/testing, not production.

Final Answer: C

Comprehensive and Detailed Explanation:

Spark Connect introduces a decoupled client-server architecture. Its key feature is enabling Spark job submission and execution from remote clients — in Python, Java, etc.

From Databricks documentation:

“Spark Connect allows remote clients to connect to a Spark cluster and execute Spark jobs without being co-located with the Spark driver.”

A is close, but "any language" is overstated (currently supports Python, Java, etc., not literally all).

B refers to REST, which is not Spark Connect's mechanism.

D is incorrect; Spark Connect isn’t focused on ingestion.

Final Answer: C

Which operation is supported with streaming_df?

A. streaming_df.select(countDistinct("Name"))

B. streaming_df.groupby("Id").count()

C. streaming_df.orderBy("timestamp").limit(4)

D. streaming_df.filter(col("count") < 30).show()

Answer: B

Comprehensive and Detailed Explanation:

In Structured Streaming, only a limited subset of operations is supported due to the nature of unbounded data. Operations like sorting (orderBy) and global aggregation (countDistinct) require a full view of the dataset, which is not possible with streaming data unless specific watermarks or windows are defined.

Review of Each Option:

A. select(countDistinct("Name"))

Not allowed — Global aggregation like countDistinct() requires the full dataset and is not supported directly in streaming without watermark and windowing logic.

Comprehensive and Detailed Explanation From Exact Extract:

Spark Connect enables decoupling of the client and Spark driver processes, allowing remote access. Spark supports configuring the remote Spark Connect server in multiple ways:

From Databricks and Spark documentation:

Option B (--remote "sc://localhost") is a valid command-line argument for thepysparkshell to connect using Spark Connect.

Option C (settingSPARK_REMOTEenvironment variable) is also a supported method to configure the remote endpoint.

Option A is incorrect because Spark Connect uses thesc://protocol, nothttps:// .

Option D requires modifying the code, which the question explicitly avoids.

Option E configures the port on the server side but doesn’t start a client connection.

Final Answers: B and C