nsrdb.file_handlers.file_system.NSRDBFileSystem

class NSRDBFileSystem(path, anon=False, profile=None, use_rex=False, **kwargs)

Bases: FileSystem

Custom FileSystem handler for NSRDB with unique logic to open .h5 and .nc files in AWS S3

Parameters:

• path (str) – S3 object path or file path

• anon (bool, optional) – Whether to use anonymous credentials, by default False

• profile (str, optional) – AWS credentials profile, by default None

• use_rex (bool) – Flag to uss rex/nsrdb Resource as the .h5 handler instead of the h5py.

Methods

copy(src_path, dst_path[, anon, profile])	Copy file(s) from src_path to dst_path.
cp(dst, **kwargs)	Copy file to given destination
exists()	Check if file path exists
glob(**kwargs)	Find all file paths matching the given pattern
isdir()	Check if path is a directory
isfile()	Check if path is a file
ls()	List everyting under given path
mkdirs(**kwargs)	Make desired directory and any intermediate directories
mv(dst, **kwargs)	Move file or all files in directory to given destination
open()	open filesystem handler, if file is a .h5 or .nc file also open a h5py.File or netCDF4.Dataset handler respectively.
rm(**kwargs)	Delete file or files in given directory
size()	Get file size in bytes
walk()	Recursively search directory and all sub-directories

Attributes

path	File path to perform filesystem operation on

open()

open filesystem handler, if file is a .h5 or .nc file also open a h5py.File or netCDF4.Dataset handler respectively.

Returns:

_file_handler (obj) – Proper file handler, either: - local file path - open s3fs.File object - open h5py.File object - open netCDF4.Dataset object

classmethod copy(src_path, dst_path, anon=False, profile=None, **kwargs)

Copy file(s) from src_path to dst_path. Either can be local or in the cloud.

Parameters:

• src_path (str) – Source path to copy file(s) from, can be local or in the cloud

• dst_path (str) – Destination path to copy file(s) to, can be local or in the cloud

• anon (bool, optional) – Whether to use anonymous credentials, by default False

• profile (str, optional) – AWS credentials profile, by default None

cp(dst, **kwargs)

Copy file to given destination

Parameters:

• dst (str) – Destination path

• kwargs (dict) – kwargs for s3fs.S3FileSystem.copy

exists()

Check if file path exists

Returns:

bool

glob(**kwargs)

Find all file paths matching the given pattern

Parameters:

kwargs (dict) – kwargs for s3fs.S3FileSystem.glob

Returns:

list

isdir()

Check if path is a directory

Returns:

bool

isfile()

Check if path is a file

Returns:

bool

ls()

List everyting under given path

Returns:

list

mkdirs(**kwargs)

Make desired directory and any intermediate directories

Parameters:

kwargs (dict) – kwargs for s3fs.S3FileSystem.mkdirs

mv(dst, **kwargs)

Move file or all files in directory to given destination

Parameters:

• dst (str) – Destination path

• kwargs (dict) – kwargs for s3fs.S3FileSystem.mv

property path

File path to perform filesystem operation on

Returns:

str

rm(**kwargs)

Delete file or files in given directory

Parameters:

kwargs (dict) – kwargs for s3fs.S3FileSystem.rm

size()

Get file size in bytes

Returns:

float

walk()

Recursively search directory and all sub-directories

Returns:

• path (str) – Root path

• directory (list) – All directories in path

• file (list) – All files in path