public class S3DynamoDBLogStore extends BaseExternalLogStore
BaseExternalLogStore that uses an external DynamoDB table
to provide the mutual exclusion during calls to `putExternalEntry`.
DynamoDB entries are of form
- key
-- tablePath (HASH, STRING)
-- filename (RANGE, STRING)
- attributes
-- tempPath (STRING, relative to _delta_log)
-- complete (STRING, representing boolean, "true" or "false")
-- commitTime (NUMBER, epoch seconds)| Modifier and Type | Field and Description |
|---|---|
static java.lang.String |
BASE_CONF_PREFIX |
static java.lang.String |
DDB_CLIENT_CREDENTIALS_PROVIDER |
static java.lang.String |
DDB_CLIENT_REGION |
static java.lang.String |
DDB_CLIENT_TABLE |
static java.lang.String |
DDB_CREATE_TABLE_RCU |
static java.lang.String |
DDB_CREATE_TABLE_WCU |
static java.lang.String |
SPARK_CONF_PREFIX
Configuration keys for the DynamoDB client.
|
| Constructor and Description |
|---|
S3DynamoDBLogStore(org.apache.hadoop.conf.Configuration hadoopConf) |
| Modifier and Type | Method and Description |
|---|---|
protected java.util.Optional<ExternalCommitEntry> |
getExternalEntry(java.lang.String tablePath,
java.lang.String fileName)
Return external store entry corresponding to delta log file with given `tablePath` and
`fileName`, or `Optional.empty()` if it doesn't exist.
|
protected java.util.Optional<ExternalCommitEntry> |
getLatestExternalEntry(org.apache.hadoop.fs.Path tablePath)
Return the latest external store entry corresponding to the delta log for given `tablePath`,
or `Optional.empty()` if it doesn't exist.
|
protected static java.lang.String |
getParam(org.apache.hadoop.conf.Configuration hadoopConf,
java.lang.String name,
java.lang.String defaultValue)
Get the hadoopConf param $name that is prefixed either with $SPARK_CONF_PREFIX or
$BASE_CONF_PREFIX.
|
protected void |
putExternalEntry(ExternalCommitEntry entry,
boolean overwrite)
Write to external store in exclusive way.
|
createTemporaryPath, fixDeltaLogCopyTempFile, fixDeltaLogPutCompleteDbEntry, getTablePath, isPartialWriteVisible, listFrom, write, writeActions, writeCopyTempFile, writePutCompleteDbEntrypublic static final java.lang.String SPARK_CONF_PREFIX
public static final java.lang.String BASE_CONF_PREFIX
public static final java.lang.String DDB_CLIENT_TABLE
public static final java.lang.String DDB_CLIENT_REGION
public static final java.lang.String DDB_CLIENT_CREDENTIALS_PROVIDER
public static final java.lang.String DDB_CREATE_TABLE_RCU
public static final java.lang.String DDB_CREATE_TABLE_WCU
public S3DynamoDBLogStore(org.apache.hadoop.conf.Configuration hadoopConf)
throws java.io.IOException
java.io.IOExceptionprotected void putExternalEntry(ExternalCommitEntry entry, boolean overwrite) throws java.io.IOException
BaseExternalLogStoreputExternalEntry in class BaseExternalLogStorejava.io.IOExceptionprotected java.util.Optional<ExternalCommitEntry> getExternalEntry(java.lang.String tablePath, java.lang.String fileName)
BaseExternalLogStoregetExternalEntry in class BaseExternalLogStoreprotected java.util.Optional<ExternalCommitEntry> getLatestExternalEntry(org.apache.hadoop.fs.Path tablePath)
BaseExternalLogStoregetLatestExternalEntry in class BaseExternalLogStoretablePath - TODOprotected static java.lang.String getParam(org.apache.hadoop.conf.Configuration hadoopConf,
java.lang.String name,
java.lang.String defaultValue)