Type Members

1.  class ConfluentJsonConnector extends ConfluentConnector

   Provides Spark SQL functions from/to_confluent for decoding/encoding confluent json messages.

2.  class JsonToSparkSchemaConverter extends AnyRef

   Schema Converter for getting schema in json format into a spark Structure

   Schema Converter for getting schema in json format into a spark Structure

   The given schema for spark has almost no validity checks, so it will make sense to combine this with the schema-validator. For loading data with schema, data is converted to the type given in the schema. If this is not possible the whole row will be null (!). A field can be null if its type is a 2-element array, one of which is "null". The converted schema doesn't check for 'enum' fields, i.e. fields which are limited to a given set. It also doesn't check for required fields or if additional properties are set to true or false. If a field is specified in the schema, than you can select it and it will be null if missing. If a field is not in the schema, it cannot be selected even if given in the dataset.

3.  case class SchemaType(typeName: String, nullable: Boolean) extends Product with Serializable
4.  case class StructsToJsonWithConfluent(options: Map[String, String], child: Expression, confluentClient: ConfluentClient[JsonSchema], topic: String, subjectType: SubjectType, updateAllowed: Boolean = false, mutualReadCheck: Boolean = false, eagerCheck: Boolean = false, timeZoneId: Option[String] = None) extends UnaryExpression with TimeZoneAwareExpression with CodegenFallback with ExpectsInputTypes with NullIntolerant with Product with Serializable

   As schema from an expression can only be retrieved at execution time, we need customize Sparks StructsToJsonWithConfluent operator.

   As schema from an expression can only be retrieved at execution time, we need customize Sparks StructsToJsonWithConfluent operator. This is copied from spark.sql.catalyst.expressions.jsonExpressions.scala.