Skip to content
This repository has been archived by the owner on Sep 10, 2022. It is now read-only.

agutil (main module)

Jump to bottom
Aaron Graubert edited this page Mar 28, 2019 · 7 revisions

agutil utility methods

The following utility methods are available directly from the agutil module.

  • agutil.intToBytes(num, padding_length=0):

    Converts the int num to its big-endian byte representation. If the length of the converted bytestring is less than padding_length, 0-bytes (\x00) are added to the beginning of the string. This has to do with an inherent issue with int<->byte conversion: 0-bytes in more-significant end of a bytestring are equivalent to adding zeroes to the left of an integer. These bytes are lost in a conversion from bytes->int. padding_length provides a solution if the length of the original bytestring is known. For conversions of int->bytes->int, the padding_length parameter is not needed. For conversions of bytes->int->bytes, it may be necessary to either communicate the desired length in advance, or prepend a non-zero byte to the start of the string

  • agutil.bytesToInt(num):

    Converts the byte sequence num into an integer

  • agutil.byte_xor(b1, b2):

    Returns a byte sequence equivalent to the XOR of each byte in b1 with the corresponding byte in b2

  • agutil.clump(seq, length):

  • agutil.split_iterable(seq, length):

    Yields iterables which take from seq in chunks up to length. Each iterable returned will yield up to length items. If chained together, the iterables returned would iterate the same sequence as seq. Note: You should not use the returned iterables out of order. Each iterable will always yield the correct number of elements, but elements will be out of order if you do not completely exhaust one iterable before moving to the next

  • agutil.splice(seq):

    Takes an iterable, seq, which yields N items, each of length M and returns a list of M generators which each yield N items. This is essentially the opposite of the builtin zip. The width of seq is determined by its first element, and splice will provide a number of iterators equal to that width.

  • agutil.hashfile(filepath, algorithm='sha1', length=None):

    Opens the file at the requested filepath and generates a hash using the specified algorithm. If filepath cannot be found, it raises a FileNotFoundError. If algorithm is not available on the current platform, it raises a ValueError. If length is provided and is not None, it is passed to algorithm.digest() for variable length digest algorithms (shake_128 and shake_256)

  • agutil.byteSize(n):

    Returns a string with n converted to the highest unit of Bytes where n would be at least 1 (caps at ZiB), rounded to 1 decimal place. Examples:

    • byteSize(1) = 1B
    • byteSize(1023) = 1023B
    • byteSize(1024) = 1KiB
    • byteSize(856633344) = 816.9MiB
    • byteSize(12379856472314232739172) = 10.5ZiB

  • agutil.cmd(expr, display=True):

    Executes expr in a background shell and returns a agutil.ShellReturnObject afterwards. expr can either be a single command string, or a list/tuple of command arguments (which will be quoted and escaped if necessary). If display is True, the stdout and stderr from the background shell will be displayed live in the interpreter in addition to being captured in the returned agutil.ShellReturnObject

  • agutil.first(iterable, predicate):

    Takes an iterable and returns the first item for which predicate(item) is True. If predicate is callable, it must be a function which takes a single argument (if the result of the function is 'Truthly' the item in the iterable will be returned). Alternatively, predicate may be an object, in which case first() will return the first item in the iterable which compares equal to the given predicate object

  • agutil.context_lock(lock, timeout=-1): (Context Manager)

    Returns a context manager object. When entering the context it attempts to acquire the lock. If timeout is a non-negative number, it waits at most timeout seconds before raising agutil.LockTimeoutExceeded. When exiting the context, it always attempts to release the lock, regardless of any exceptions which may have occurred within the context.

  • agutil.TimeoutExceeded: (Exception)

    OSError -> socket.timeout -> agutil.TimeoutExceeded This exception is raised by agutil.ActiveTimeout when the timeout has expired

  • agutil.LockTimeoutExceeded: (Exception)

    OSError -> socket.timeout -> agutil.TimeoutExceeded -> agutil.LockTimeoutExceeded This exception is raised by agutil.context_lock when it fails to acquire the lock within the provided timeout

Argparse File Types:

The following classes can be passed to argparse.ArgumentParser.add_argument as the type parameter. They can be used to accept Files, Files of File Names, and Directories, and then convert the string argument into a variety of types to handle different requirements

agutil.FileType

  • FileType(*extensions, compression=False, output=None, existence=os.path.isfile, **kwargs) (Constructor)

    Initializes a new FileType object. When called on a string, the FileType instance will check that the string is a valid filepath ending in one of the provided extensions. If extensions is empty, any valid filepath will be accepted regardless of extension. If compression is True, it will also accept filepaths which end in one of the provided extensions followed by ".gz", ".bgz", or ".bz2". If compression is a list, it is taken to be the set of allowed compression extensions. output and arbitrary keyword arguments control the type of output from calling the FileType instance. For descriptions of those arguments, see FileType.__call__. existence should be a callable, which is used to determine if a given argument exists. It defaults to os.path.isfile

  • FileType.__call__(arg) (Call operator)

    Checks that the provided arg is a valid filepath and ends in one of the allowed extensions (as specified in the constructor). If both of those are true, return the absolute filepath for arg. Otherwise raise an argparse.ArgumentTypeError. If output was set to a String in the constructor, this method will return a File-Like object instead of an absolute filepath. The value of output is used as the mode argument to open. Additionally, the value of kwargs in the constructor will be unpacked as keyword arguments to open. If output was a String, arg ends in ".gz", ".bgz", or ".bz2", and that compression extension existed in the compression argument to the constructor, the appropriate compression library will be used to open the file, so that the returned File-Like object reads decompressed data. The constructor's output and kwargs arguments will also be used when opening a compressed file.

agutil.DirType

  • DirType(existence=os.path.isdir) (Constructor)

    Initializes a new DirType object. existence should be a callable, which is used to determine if a given argument exists. It defaults to os.path.isdir

  • DirType.__call__(arg) (Call operator)

    Checks that the provided arg is a valid path to a directory and returns the absolute filepath. Otherwise it raises an argparse.ArgumentTypeError.

agutil.FOFNType

  • FOFNType(*extensions, min_paths=1, as_list=False, as_handle=False, allow_direct=False, existence=os.path.isfile, **kwargs) (Constructor)

    Initializes a new FOFNType instance. extensions should be a list of str, FileType, DirType, or FOFNType instances, together representing the set of acceptable files in the FOFN. Remaining arguments are stored and affect the behavior of FOFNTYPE.__call__. existence should be a callable, which is used to determine if a given argument exists. It defaults to os.path.isfile

  • FOFNType.__call__(arg) (Call operator)

    Checks that the provided arg is a valid filepath. Returns the absolute filepath of arg. If min_paths was greater than 0 in the constructor, the file will be opened and the first min_paths lines will be checked against the extensions provided in the constructor. If there are less than min_paths lines or any of the checked lines is rejected by all extensions, raise an argparse.ArgumentTypeError. If as_list is True, all lines will be checked and this method will return a list of the absolute filepaths contained in the FOFN instead of the absolute filepath of arg. If as_handle is True, this method will return an open File-Like object to arg instead of the absolute filepath. The value of kwargs in the constructor is unpacked when calling open. If both as_list and as_handle were true, this method will return a list of File-Like objects each open to one of the files listed in the FOFN. When checking the lines of the FOFN, if any of the checks fail (raising an argparse.ArgumentTypeError) and allow_direct was True in the constructor, attempt to check arg directly against the set of extensions (instead of treating it like a file containing files to check). If arg matches one of the extensions, this method continues as if it was parsing a FOFN containing arg on one line. In other words, it returns the same result that it would return if this method had been called on a filepath to a file containing arg and nothing else. Depending on the return type, a temporary file may be created, in which case, it will automatically be deleted when the interpreter exits

Clone this wiki locally