Spark Datasets provide a way to work with structured and semi-structured data in a type-safe manner. They combine the benefits of RDDs (distributed processing, fault tolerance) with the convenience and optimization of structured data processing. Datasets are essentially optimized DataFrames with added type safety.
A Dataset is a strongly typed collection of domain-specific objects that can be transformed in parallel using functional or relational operations. Each Dataset also has an untyped view called a DataFrame, which is a Dataset of Row. Operations available on Datasets are divided into transformations and actions. Transformations are the ones that produce new Datasets, and actions are the ones that trigger computation and return results. Example transformations include map, filter, select, and aggregate (groupBy). Example actions count, show, or writing data out to file systems. Datasets are “lazy”, i.e. computations are only triggered when an action is invoked. Internally, a Dataset represents a logical plan that describes the computation required to produce the data. When an action is invoked, Spark’s query optimizer optimizes the logical plan and generates a physical plan for efficient execution in a parallel and distributed manner. source

​
Key Features of Datasets

  • Type Safety: Datasets offer type safety, meaning that the schema of the data is known at compile time. This allows for better performance and error detection. This is a significant advantage over DataFrames, which are schema-aware but not type-safe.
  • Optimized Execution: Spark’s optimizer can perform more aggressive optimizations on Datasets due to the type information. This leads to improved performance compared to RDDs.
  • Integration with Spark SQL: Datasets seamlessly integrate with Spark SQL, allowing you to use SQL queries to process your data.
  • Encoders: Encoders are used to convert between JVM objects and Spark’s internal representation. They are crucial for enabling type safety in Datasets.
  • Lazy Evaluation: Like RDDs, Datasets also utilize lazy evaluation, meaning transformations are only executed when an action is called.

​
Creating Datasets

Datasets can be created from various sources, including:
  • Existing DataFrames: You can easily convert an existing DataFrame to a Dataset.
  • Case classes: Define a case class (in Scala or a similar construct in other languages like Python’s namedtuple) representing your data structure. Spark can then infer the schema from the case class.
  • JSON, CSV, Parquet, etc.: Spark can infer the schema from various file formats and create a Dataset directly.

​
Datasets vs. DataFrames

While Datasets offer type safety and potential performance benefits, DataFrames are often more convenient for quick data exploration and manipulation, especially when dealing with complex or less structured data. The choice between Datasets and DataFrames depends on the specific needs of your application. For many use cases, the performance difference is negligible. If type safety is a priority, or if you need the maximum performance optimization, Datasets are the better choice.

​
QnA

Q: Can Datasets be used with Spark SQL?