Type Members

1.  class ExpressionEvaluator[T <: Product, R] extends AnyRef

   ExpressionEvaluator can evaluate a Spark SQL expression against a case class

   ExpressionEvaluator can evaluate a Spark SQL expression against a case class

   T

   class of object the expression should be evaluated on

   R

   class of expressions expected return type. This might also be set Any, in that case the result type check is omitted and complex datatypes will not be mapped to case classes, as they are not specified.

2.  class RowDecoder[T <: Product] extends Serializable

   Spark row to case class Decoder.

   Spark row to case class Decoder. This can be used for example in UDF's with struct parameters, which are mapped as Row in Spark 2.x (Spark 3.x supports case classes as parameters directly) If you have to decode multiple rows (like in a UDF) it's important to instantiate the RowDecoder only once (outside the UDF) and reuse it for all rows.

   T

   the case class to be produced

3.  case class SetNullable(child: Expression, forcedNullable: Boolean) extends UnaryExpression with Product with Serializable
4.  case class UnsafeUnaryUdfExpression(child: Expression, udf: (Any) => Any, tgtDataType: DataType) extends UnaryExpression with Product with Serializable

   With Spark 3.0 the API for Udf's was made more typesafe.

   With Spark 3.0 the API for Udf's was made more typesafe. It's no longer possible to create a Udf and give it's return type as StructType. This implements an explicitly unsafe unary udf, which takes a function with signature Any -> Any as transformation. UnsafeUnaryUdfExpression is the Udf expression that is embedded and executed in Spark logical plan. Use UnsafeUnaryUdf to create the udf function to be used in Spark DataFrame API.