Copy data from USB

Talking about OS and emulation core development
Post Reply
User avatar
Meewan
Posts: 2
Joined: Sun Feb 23, 2020 12:52 am
languages_spoken: english, french
Has thanked: 0
Been thanked: 3 times
Contact:

Copy data from USB

Post by Meewan » Sun Feb 23, 2020 1:14 am

I have made a simple python script to copy games from USB to the SD card.

It is compatible with any filesystem supported by the operating system (only tested with vfat but it tries all available systems from /proc/filesystems that are not flagged with nodev).
It explores all mountable partitions from the USB to find compatible "games" (".sh" script are consided as games in EmulationStation) and put them all in the correct folder in /roms (except the ".zip" that are ignored as they could go to multiple places)

TODO
  • support /opt/system to be able to import games at anytime (the current version only import games on boot) this probably needs to add a root daemon to mount the usb stick
  • installer (a single file would be easier to install and make ease the developement of scripts) done
  • give the user a hint about when the game are all uploaded (maybe by playing a sound ?)

Installation
copy the attached file as

Code: Select all

 /etc/cron.d/intsaller_ma
installer : https://framadrop.org/r/aAe_3RNkD7#GWg7 ... Xh6Lwy1CU=

code
/usr/lib/aarch64-linux-gnu/odroid_go_advance_ma/odroid_go_advance_ma.py

Code: Select all

#!/usr/bin/python3
import ctypes
import ctypes.util
import html.parser
import os
import re
import shutil
import syslog
import traceback

# configuration
DEBUG = False
# where to mount the found devices
MOUNT_FOLDER = "/mnt"
# regex that matches the devices in /dev
USB_STICK = r"^sd[a-z][1-9]$"
# path to the EmulationStation xml configuration file
ES_CONFIG_FILE = "/etc/emulationstation/es_systems.cfg"

# libc management
MS_RDONLY = 1
libc = ctypes.CDLL(ctypes.util.find_library('c'), use_errno=True)
libc.mount.argtypes = (
    ctypes.c_char_p,
    ctypes.c_char_p,
    ctypes.c_char_p,
    ctypes.c_ulong,
    ctypes.c_char_p,
)
libc.umount.argtypes = (
    ctypes.c_char_p,
)


def automount(source, target, read_only=False):
    """Auto mount a device by guessing its fs from /proc/filesystems

    inspired from https://stackoverflow.com/questions/1667257/how-do-i-mount-a-filesystem-using-python/34812552 # noqa

    :param str source: special file to mount (example "/dev/sda")
    :param str target: folder where to mount de file system
    :param boolean read_only: wether or not mount the file system as read only
    :raise: OSError
    """
    with open("/proc/filesystems") as fp:
        for line in fp.readlines():
            table_line = line.split("\t")
            if not len(table_line) == 2 or table_line[0] == "nodev":
                continue
            else:
                fs = table_line[1].strip("\n")
                ret = libc.mount(
                    source.encode('utf-8'),
                    target.encode('utf-8'),
                    fs.encode('utf-8'),
                    MS_RDONLY if read_only else 0,
                    b''
                )
                if ret != 0:
                    if DEBUG:
                        syslog.syslog(f"DEBUG : {source} is not {fs}")
                else:
                    if DEBUG:
                        syslog.syslog(f"DEBUG : {source} mounted with {fs}")
                    break
        else:
            raise RuntimeError(
                f"Error mounting {source} no compatible filesystem found"
            )


def umount(target):
    """Unmount a file system

    :param boolean read_only: wether or not mount the file system as read only
    :raise: OSError
    """
    ret = libc.umount(
        target.encode('utf-8')
    )
    if ret < 0:
        errno = ctypes.get_errno()
        raise OSError(
            errno,
            f"Error unmounting {target} : {os.strerror(errno)}"
        )


def get_partitions_to_mount(usb_stick):
    """Get all partitions from /dev matching the USB_STICK regex

    :param str usb_stick: regex to find a partition tou mount
    :rtype: list(str)
    :returns: a list of path to special device to mount
    """
    partitions = []
    dev = os.listdir("/dev")
    for candidate in dev:
        if re.match(usb_stick, candidate):
            partitions.append(os.path.join("/dev", candidate))
    if DEBUG:
        syslog.syslog(
            "%s match found in /dev: '%s'" % (
                len(partitions),
                ', '.join(partitions)
            )
        )
    return partitions


def get_file_extension_folder_dict(config_file):
    """Retrieve a dict to match a file extention with its destination folder.
    This function works by reading the EmulationStation config file given in
    argument.

    :param str config_file: path to the EmulationStation configuration file
    :rtype: dict(str:str)
    :return: a dict with the file extension in key and the target folder as
        value
    :raise: ValueError if the configuration file is not found
    """
    if not os.path.exists(config_file):
        syslog.syslog(f"configuration file {config_file} not found")
        raise ValueError(f"configuration file {config_file} not found")
    parser = ConfigParser()
    with open(config_file, 'r') as fp:
        parser.feed(fp.read())
    return parser.get_extension_path_dict()


def get_remote_files(root, extension_folder_dict):
    """List all files to copy from the given folder. and match them with the
    correct targt name.

    :param str root: absolute path to the folder to explore
    :param dict extension_folder_dict: dict that match the target folder for
        each supported extension
    :rtype: dict(str, str)
    :return: the list of absolute path of the files to explore
    """
    remote_files = {}
    for dirpath, dirnames, filenames in os.walk(root):
        for filename in filenames:
            if filename.startswith('.') or '.' not in filename:
                # ignore hidden files and files without extensions
                continue
            extention = '.' + filename.split('.')[-1]
            if extention in extension_folder_dict:
                absolute_origin_path = os.path.join(dirpath, filename)
                absolute_target_path = os.path.join(
                    extension_folder_dict[extention],
                    filename
                )
                remote_files[absolute_origin_path] = absolute_target_path
    return remote_files


def copy_remote_files(remote_files):
    """Copy remote files if the file does not already exists

    :param dict remote_files: A dict with the origin path as key and the target
        path as value
    """
    copied = 0
    for origin, target in remote_files.items():
        if os.path.exists(target):
            if DEBUG:
                syslog.syslog(
                    f"ignoring {origin} file {target} already exists"
                )
            continue
        try:
            shutil.copyfile(origin, target)
            copied += 1
            if DEBUG:
                syslog.syslog(f"copied {origin} to {target}")
        except (IOError, OSError):
            syslog.syslog(f"ERROR could not copy {origin} file {target}")
            if DEBUG:
                syslog.syslog(traceback.format_exc())

    syslog.syslog(f"copied {copied} files")


def main():
    partitions = get_partitions_to_mount(USB_STICK)
    mounted = []
    if not partitions:
        return
    extension_folder_dict = get_file_extension_folder_dict(ES_CONFIG_FILE)
    remote_files = {}
    for partition in partitions:
        target = "/mnt/" + partition.split("/")[-1]
        if not os.path.exists(target):
            os.mkdir(target)
        try:
            automount(
                source=partition,
                target=target,
                read_only=True,
            )
        except OSError:
            syslog.syslog(f"WARNING: could not mount partition {target}")
            if DEBUG:
                syslog.syslog(traceback.format_exc())
        else:
            mounted.append(target)

    for target in mounted:
        remote_files.update(
            get_remote_files(
                root=target,
                extension_folder_dict=extension_folder_dict,
            )
        )
    copy_remote_files(remote_files)

    for target in mounted:
        umount(target)


class ConfigParser(html.parser.HTMLParser):

    def __init__(self, ignore_extensions=frozenset(['.zip'])):
        """Parse the es_systems.cfg and extract an association dict and the
        list of system properties

        :param iterable ignore_extensions: list of extention (with dot) to
            ignore.These extensions are case sensitive.
        """
        self._current_system = None
        self._current_tag = []
        self.ignore_extensions = ignore_extensions
        self.content = []
        self.dict = {}
        super(ConfigParser, self).__init__()

    def handle_starttag(self, tag, attrs):
        self._current_tag.append(tag)
        if tag == "system":
            self._current_system = {}

    def handle_endtag(self, tag):
        if tag == "system":
            for extension in self._current_system['extension'].split(' '):
                if extension not in self.ignore_extensions:
                    self.dict[extension] = self._current_system['path']
            self.content.append(self._current_system)
        self._current_tag.pop()

    def handle_data(self, data):
        if len(self._current_tag) > 1 and data.strip():
            self._current_system[self._current_tag[-1]] = data

    def get_extension_path_dict(self):
        """Return a dict with extensions in key and a folder path for this
        device as value.

        :rtype: dict(str, str)
        """
        return self.dict

    def get_configuration(self):
        """Return the list of systems with all their properties unaltered

        :rtype:(list(dict(str,str)))
        """
        return self.content


if __name__ == "__main__":
    try:
        main()
    except Exception:
        syslog.syslog(traceback.format_exc())

/etc/cron.d/odroid_go_advance_ma

Code: Select all

@reboot root /usr/bin/python3 /usr/lib/aarch64-linux-gnu/odroid_go_advance_ma/odroid_go_advance_ma.py
License
This software is distributed under CC0 (public domain)
Last edited by Meewan on Fri Feb 28, 2020 7:58 pm, edited 2 times in total.
These users thanked the author Meewan for the post (total 3):
rooted (Wed Feb 26, 2020 4:22 am) • SnowFarmers (Thu Feb 27, 2020 8:06 pm) • DrewfusTwofus (Sun Mar 08, 2020 2:31 am)

User avatar
rooted
Posts: 7744
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 720 times
Been thanked: 208 times
Contact:

Re: Copy data from USB

Post by rooted » Wed Feb 26, 2020 4:25 am

Very interesting, this would certainly help a lot of people if they knew about it.

Nice work.

User avatar
Meewan
Posts: 2
Joined: Sun Feb 23, 2020 12:52 am
languages_spoken: english, french
Has thanked: 0
Been thanked: 3 times
Contact:

Re: Copy data from USB

Post by Meewan » Fri Feb 28, 2020 8:01 pm

Thanks,

Updated with an installer. I am working on the daemon to support import at any time.

On the technical part the installer is just an awful hack with base64 to launch a heavy command (more than 999 char) with cron. The first leve of base64 is just a python file, the second is a tbz2 (tar + bz2) of the files to install.

User avatar
rooted
Posts: 7744
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 720 times
Been thanked: 208 times
Contact:

Re: Copy data from USB

Post by rooted » Mon Mar 02, 2020 8:25 am

Have you thought of adapting this to include copying BIOS files into the correct location also?
These users thanked the author rooted for the post:
DrewfusTwofus (Sun Mar 08, 2020 2:31 am)

neights
Posts: 1
Joined: Tue May 26, 2020 10:56 am
languages_spoken: english
Has thanked: 0
Been thanked: 0
Contact:

Re: Copy data from USB

Post by neights » Thu Jun 04, 2020 10:16 am

Hello, the download address has failed, can we get a new one?

PaulF8080
Posts: 64
Joined: Fri Feb 14, 2020 9:40 am
languages_spoken: english
ODROIDs: go, go advance, xu4
Has thanked: 0
Been thanked: 8 times
Contact:

Re: Copy data from USB

Post by PaulF8080 » Fri Jun 05, 2020 2:15 pm

Is there any way you could install it without buying a wifi stick?

Post Reply

Return to “Platform development”

Who is online

Users browsing this forum: No registered users and 2 guests