io.delta.storage

Class S3DynamoDBLogStore

java.lang.Object

io.delta.storage.LogStore

io.delta.storage.HadoopFileSystemLogStore

io.delta.storage.BaseExternalLogStore

io.delta.storage.S3DynamoDBLogStore

public class S3DynamoDBLogStore
extends BaseExternalLogStore

A concrete implementation of BaseExternalLogStore that uses an external DynamoDB table to provide the mutual exclusion during calls to `putExternalEntry`. DynamoDB entries are of form - key -- tablePath (HASH, STRING) -- filename (RANGE, STRING) - attributes -- tempPath (STRING, relative to _delta_log) -- complete (STRING, representing boolean, "true" or "false") -- commitTime (NUMBER, epoch seconds)

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	BASE_CONF_PREFIX
static java.lang.String	DDB_CLIENT_CREDENTIALS_PROVIDER
static java.lang.String	DDB_CLIENT_REGION
static java.lang.String	DDB_CLIENT_TABLE
static java.lang.String	DDB_CREATE_TABLE_RCU
static java.lang.String	DDB_CREATE_TABLE_WCU
static java.lang.String	SPARK_CONF_PREFIX Configuration keys for the DynamoDB client.

Constructor Summary

Constructors

Constructor and Description
S3DynamoDBLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Modifier and Type	Method and Description
protected java.util.Optional<ExternalCommitEntry>	getExternalEntry(java.lang.String tablePath, java.lang.String fileName) Return external store entry corresponding to delta log file with given `tablePath` and `fileName`, or `Optional.empty()` if it doesn't exist.
protected java.util.Optional<ExternalCommitEntry>	getLatestExternalEntry(org.apache.hadoop.fs.Path tablePath) Return the latest external store entry corresponding to the delta log for given `tablePath`, or `Optional.empty()` if it doesn't exist.
protected static java.lang.String	getParam(org.apache.hadoop.conf.Configuration hadoopConf, java.lang.String name, java.lang.String defaultValue) Get the hadoopConf param $name that is prefixed either with $SPARK_CONF_PREFIX or $BASE_CONF_PREFIX.
protected void	putExternalEntry(ExternalCommitEntry entry, boolean overwrite) Write to external store in exclusive way.

Methods inherited from class io.delta.storage.BaseExternalLogStore

createTemporaryPath, fixDeltaLogCopyTempFile, fixDeltaLogPutCompleteDbEntry, getTablePath, isPartialWriteVisible, listFrom, write, writeActions, writeCopyTempFile, writePutCompleteDbEntry

Methods inherited from class io.delta.storage.HadoopFileSystemLogStore

createTempPath, read, resolvePathOnPhysicalStorage, writeWithRename

Methods inherited from class io.delta.storage.LogStore

initHadoopConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SPARK_CONF_PREFIX

public static final java.lang.String SPARK_CONF_PREFIX

Configuration keys for the DynamoDB client. Keys are either of the form $SPARK_CONF_PREFIX.$CONF or $BASE_CONF_PREFIX.$CONF, e.g. spark.io.delta.storage.S3DynamoDBLogStore.ddb.tableName or io.delta.storage.S3DynamoDBLogStore.ddb.tableName

See Also:

Constant Field Values

BASE_CONF_PREFIX

public static final java.lang.String BASE_CONF_PREFIX

See Also:

Constant Field Values

DDB_CLIENT_TABLE

public static final java.lang.String DDB_CLIENT_TABLE

See Also:

Constant Field Values

DDB_CLIENT_REGION

public static final java.lang.String DDB_CLIENT_REGION

See Also:

Constant Field Values

DDB_CLIENT_CREDENTIALS_PROVIDER

public static final java.lang.String DDB_CLIENT_CREDENTIALS_PROVIDER

See Also:

Constant Field Values

DDB_CREATE_TABLE_RCU

public static final java.lang.String DDB_CREATE_TABLE_RCU

See Also:

Constant Field Values

DDB_CREATE_TABLE_WCU

public static final java.lang.String DDB_CREATE_TABLE_WCU

See Also:

Constant Field Values

Constructor Detail

S3DynamoDBLogStore

public S3DynamoDBLogStore(org.apache.hadoop.conf.Configuration hadoopConf)
                   throws java.io.IOException

Throws:

java.io.IOException

Method Detail

putExternalEntry

protected void putExternalEntry(ExternalCommitEntry entry,
                                boolean overwrite)
                         throws java.io.IOException

Description copied from class: BaseExternalLogStore

Write to external store in exclusive way.

Specified by:

putExternalEntry in class BaseExternalLogStore

Throws:

java.io.IOException

getExternalEntry

protected java.util.Optional<ExternalCommitEntry> getExternalEntry(java.lang.String tablePath,
                                                                   java.lang.String fileName)

Description copied from class: BaseExternalLogStore

Return external store entry corresponding to delta log file with given `tablePath` and `fileName`, or `Optional.empty()` if it doesn't exist.

Specified by:

getExternalEntry in class BaseExternalLogStore

getLatestExternalEntry

protected java.util.Optional<ExternalCommitEntry> getLatestExternalEntry(org.apache.hadoop.fs.Path tablePath)

Description copied from class: BaseExternalLogStore

Return the latest external store entry corresponding to the delta log for given `tablePath`, or `Optional.empty()` if it doesn't exist.

Specified by:

getLatestExternalEntry in class BaseExternalLogStore

Parameters:

tablePath - TODO

getParam

protected static java.lang.String getParam(org.apache.hadoop.conf.Configuration hadoopConf,
                                           java.lang.String name,
                                           java.lang.String defaultValue)

Get the hadoopConf param $name that is prefixed either with $SPARK_CONF_PREFIX or $BASE_CONF_PREFIX. If two parameters exist, one for each prefix, then an IllegalArgumentException is thrown. If no parameters exist, then the $defaultValue is returned.