io.delta.storage

Class BaseExternalLogStore

java.lang.Object

io.delta.storage.LogStore

io.delta.storage.HadoopFileSystemLogStore

io.delta.storage.BaseExternalLogStore

Direct Known Subclasses:

S3DynamoDBLogStore

public abstract class BaseExternalLogStore
extends io.delta.storage.HadoopFileSystemLogStore

A base LogStore implementation for cloud stores (e.g. Amazon S3) that do not provide mutual exclusion.

This implementation depends on child methods, particularly `putExternalEntry`, to provide the mutual exclusion that the cloud store is lacking.

Constructor Summary

Constructors

Constructor and Description
BaseExternalLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Modifier and Type	Method and Description
protected java.lang.String	createTemporaryPath(org.apache.hadoop.fs.Path path) Generate temporary path for TransactionLog.
protected void	fixDeltaLogCopyTempFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Wrapper for `copyFile`, called by the `fixDeltaLog` method.
protected void	fixDeltaLogPutCompleteDbEntry(ExternalCommitEntry entry) Wrapper for `putExternalEntry`, called by the `fixDeltaLog` method.
protected abstract java.util.Optional<ExternalCommitEntry>	getExternalEntry(java.lang.String tablePath, java.lang.String fileName) Return external store entry corresponding to delta log file with given `tablePath` and `fileName`, or `Optional.empty()` if it doesn't exist.
protected abstract java.util.Optional<ExternalCommitEntry>	getLatestExternalEntry(org.apache.hadoop.fs.Path tablePath) Return the latest external store entry corresponding to the delta log for given `tablePath`, or `Optional.empty()` if it doesn't exist.
protected org.apache.hadoop.fs.Path	getTablePath(org.apache.hadoop.fs.Path path) Returns the base table path for a given Delta log entry located in e.g.
java.lang.Boolean	isPartialWriteVisible(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf)
java.util.Iterator<org.apache.hadoop.fs.FileStatus>	listFrom(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) First checks if there is any incomplete entry in the external store.
protected abstract void	putExternalEntry(ExternalCommitEntry entry, boolean overwrite) Write to external store in exclusive way.
void	write(org.apache.hadoop.fs.Path path, java.util.Iterator<java.lang.String> actions, java.lang.Boolean overwrite, org.apache.hadoop.conf.Configuration hadoopConf) If overwrite=true, then write normally without any interaction with external store.
protected void	writeActions(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, java.util.Iterator<java.lang.String> actions) Write file with actions under a specific path.
protected void	writeCopyTempFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Wrapper for `copyFile`, called by the `write` method.
protected void	writePutCompleteDbEntry(ExternalCommitEntry entry) Wrapper for `putExternalEntry`, called by the `write` method.

Methods inherited from class io.delta.storage.HadoopFileSystemLogStore

createTempPath, read, resolvePathOnPhysicalStorage, writeWithRename

Methods inherited from class io.delta.storage.LogStore

initHadoopConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BaseExternalLogStore

public BaseExternalLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Detail

listFrom

public java.util.Iterator<org.apache.hadoop.fs.FileStatus> listFrom(org.apache.hadoop.fs.Path path,
                                                                    org.apache.hadoop.conf.Configuration hadoopConf)
                                                             throws java.io.IOException

First checks if there is any incomplete entry in the external store. If so, tries to perform a recovery/fix. Then, performs a normal listFrom user the `super` implementation.

Overrides:

listFrom in class io.delta.storage.HadoopFileSystemLogStore

Throws:

java.io.IOException

write

public void write(org.apache.hadoop.fs.Path path,
                  java.util.Iterator<java.lang.String> actions,
                  java.lang.Boolean overwrite,
                  org.apache.hadoop.conf.Configuration hadoopConf)
           throws java.io.IOException

If overwrite=true, then write normally without any interaction with external store. Else, to commit for delta version N: - Step 1: Ensure that N-1.json exists. If not, perform a recovery. - Step 2: PREPARE the commit. - Write `actions` into temp file T(N) - Write with mutual exclusion to external store and entry E(N, T(N), complete=false) - Step 3: COMMIT the commit to the delta log. - Copy T(N) into N.json - Step 4: ACKNOWLEDGE the commit. - Overwrite entry E in external store and set complete=true

Specified by:

write in class io.delta.storage.LogStore

Throws:

java.io.IOException

isPartialWriteVisible

public java.lang.Boolean isPartialWriteVisible(org.apache.hadoop.fs.Path path,
                                               org.apache.hadoop.conf.Configuration hadoopConf)

Specified by:

isPartialWriteVisible in class io.delta.storage.LogStore

writeActions

protected void writeActions(org.apache.hadoop.fs.FileSystem fs,
                            org.apache.hadoop.fs.Path path,
                            java.util.Iterator<java.lang.String> actions)
                     throws java.io.IOException

Write file with actions under a specific path.

Throws:

java.io.IOException

createTemporaryPath

protected java.lang.String createTemporaryPath(org.apache.hadoop.fs.Path path)

Generate temporary path for TransactionLog.

getTablePath

protected org.apache.hadoop.fs.Path getTablePath(org.apache.hadoop.fs.Path path)

Returns the base table path for a given Delta log entry located in e.g. input path of $tablePath/_delta_log/00000N.json would return $tablePath

putExternalEntry

protected abstract void putExternalEntry(ExternalCommitEntry entry,
                                         boolean overwrite)
                                  throws java.io.IOException

Write to external store in exclusive way.

Throws:

java.nio.file.FileAlreadyExistsException - if path exists in cache and `overwrite` is false

java.io.IOException

getExternalEntry

protected abstract java.util.Optional<ExternalCommitEntry> getExternalEntry(java.lang.String tablePath,
                                                                            java.lang.String fileName)
                                                                     throws java.io.IOException

Return external store entry corresponding to delta log file with given `tablePath` and `fileName`, or `Optional.empty()` if it doesn't exist.

Throws:

java.io.IOException

getLatestExternalEntry

protected abstract java.util.Optional<ExternalCommitEntry> getLatestExternalEntry(org.apache.hadoop.fs.Path tablePath)
                                                                           throws java.io.IOException

Return the latest external store entry corresponding to the delta log for given `tablePath`, or `Optional.empty()` if it doesn't exist.

Parameters:

tablePath - TODO

Throws:

java.io.IOException

writeCopyTempFile

protected void writeCopyTempFile(org.apache.hadoop.fs.FileSystem fs,
                                 org.apache.hadoop.fs.Path src,
                                 org.apache.hadoop.fs.Path dst)
                          throws java.io.IOException

Wrapper for `copyFile`, called by the `write` method.

Throws:

java.io.IOException

writePutCompleteDbEntry

protected void writePutCompleteDbEntry(ExternalCommitEntry entry)
                                throws java.io.IOException

Wrapper for `putExternalEntry`, called by the `write` method.

Throws:

java.io.IOException

fixDeltaLogCopyTempFile

protected void fixDeltaLogCopyTempFile(org.apache.hadoop.fs.FileSystem fs,
                                       org.apache.hadoop.fs.Path src,
                                       org.apache.hadoop.fs.Path dst)
                                throws java.io.IOException

Wrapper for `copyFile`, called by the `fixDeltaLog` method.

Throws:

java.io.IOException

fixDeltaLogPutCompleteDbEntry

protected void fixDeltaLogPutCompleteDbEntry(ExternalCommitEntry entry)
                                      throws java.io.IOException

Wrapper for `putExternalEntry`, called by the `fixDeltaLog` method.

Throws:

java.io.IOException