Scripting thread

[mctom]

I have a question that does not fit to any other category, so there you go, a thread for Q&A around script snippets.

So, I have a backup script that cron launches every day, and creates incremental copy of my whole root (except sys, dev, tmp, swapfile etc). Each copy is stored in a separate directory:

Code: Select all

[mctom@Tomusiomat daily]$ ll
  inode Permissions  Size Blocks User Group Date Modified Name
7866507 drwxr-xr-x      -      - root root  14 lis  2021  2021-11-13_06:25:01
8524203 drwxr-xr-x      -      - root root  14 lis  2021  2021-11-14_18:27:22
7880564 drwxr-xr-x      -      - root root  14 lis  2021  2021-11-15_10:17:01
7880983 drwxr-xr-x      -      - root root  14 lis  2021  2021-11-16_21:47:01
7881162 drwxr-xr-x      -      - root root  14 lis  2021  2021-11-17_10:45:01
7881339 drwxr-xr-x      -      - root root  14 lis  2021  2021-11-18_09:19:01
7881514 drwxr-xr-x      -      - root root  14 lis  2021  2021-11-19_09:22:01
7881691 drwxr-xr-x      -      - root root  14 lis  2021  2021-11-20_12:48:01
7881869 drwxr-xr-x      -      - root root  14 lis  2021  2021-11-21_12:09:01
7882044 drwxr-xr-x      -      - root root  21 lis  2021  2021-11-22_08:49:01
7885328 drwxr-xr-x      -      - root root  21 lis  2021  2021-11-23_08:24:02
7886174 drwxr-xr-x      -      - root root  21 lis  2021  2021-11-24_10:26:01
7889336 drwxr-xr-x      -      - root root  21 lis  2021  2021-11-25_09:33:01

Obviously storing all that requires substantial disk space, even if it's just incremental, so I was thinking I could delete most of the older copies.

I'd like to add to my bash script some code that deletes directories older than 1 month, that were not created on Wednesday. This way I'll keep just weekly copies of really old stuff.

The problem is, the script has to extract date from directory name, determine whether that's friday and more than 30 days ago, and then decide whether to keep it or not.

Any tips will be appreciated!

And here's the full script for the curious:

Code: Select all

[mctom@Tomusiomat cron.daily]$ cat backup 
#!/bin/bash

# A script to perform incremental backups using rsync

# Logging
exec > /tmp/backup.logfile 2>&1
set -x

set -o errexit
set -o nounset
set -o pipefail

readonly BACKUP_UUID="1ebac215-0666-baca-0000-0000000000ad"
readonly BACKUP_MOUNT_DIR="/mnt/backup"
readonly SOURCE_DIR="/"
readonly BACKUP_DIR="${BACKUP_MOUNT_DIR}/daily"
readonly DATETIME="$(date '+%Y-%m-%d_%H:%M:%S')"
readonly BACKUP_PATH="${BACKUP_DIR}/${DATETIME}"
readonly LATEST_LINK="${BACKUP_DIR}/latest"

# unmount in case it is currently mounted somewhere
umount /dev/disk/by-uuid/${BACKUP_UUID} || true
# mount 
mkdir -p ${BACKUP_MOUNT_DIR}
mount /dev/disk/by-uuid/${BACKUP_UUID} ${BACKUP_MOUNT_DIR}

mkdir -p "${BACKUP_DIR}"

nice -n15 rsync -av --delete \
   "${SOURCE_DIR}/" \
    --link-dest "${LATEST_LINK}" \
    --exclude=".cache" \
    --exclude="/dev" \
    --exclude="/media" \
    --exclude="/mnt" \
    --exclude="/mount" \
    --exclude="/proc" \
    --exclude="/run" \
    --exclude="/sys" \
    --exclude="/tmp" \
    --exclude="/var" \
    --exclude="/home/swapfile" \
    --exclude="/home/mctom/Downloads" \
    "${BACKUP_PATH}"

rm -rf "${LATEST_LINK}"
ln -s "${BACKUP_PATH}" "${LATEST_LINK}"

mv /tmp/backup.logfile ${BACKUP_DIR}/backup.log

umount ${BACKUP_MOUNT_DIR}

[mad_ady]

Based on what your script does, I'd recommend ditching it and switching to a backup tool like borg-backup. Last year I invested in it and was amazed by the savings in disk space due to block-level deduplication and compression. It can also expire old backups based on rules (e.g. keep 1 yearly backup, 4 weekly backups and 7 daily).

For your problem, if you want to keep your script, I'd consider building a perl oneliner that parses the date in a Date object, compares it to your criteria and returns true or false. If you want shell only, date --date="1 month ago" +%s can solve part of the problem.

Let me know if you want a working oneliner. I'll try it tomorrow, because I need access to my keyboard.

[mctom]

Thanks for the idea, but I like this way of keeping my backups, because retrieving anything doesn't rely on external tools. Most of my data (size-wise) are personal photos and videos, that I recently transcoded to x264 to save a lot of space. I don't think further compression would do any good.
As a matter of fact, my backup drive is an old 2,5" 500GB HDD that spins up only once a day for this. I think I could just get a 1TB drive instead and solve my problem for next 20 years, but I'd like to flex my bash muscle once in a while.

Thanks for pointing out date. That's a good start. I think I should iterate through a list of directories and parse their names. How can I process the output of ls in bash?

[L67GS]

WOOHOO! A scripting thread!
Here's a little one I wrote last year or so to check the number of processor cores and set the appropriate background image for the computer the hard drive is in. I'm really weird about wallpaper and use it as a template to organize my icons into groups and the resolution was different between my laptop and lab computer.
Possibly this could be of use to someone:

setwallpaper.sh

Code: Select all

 #! /bin/bash

COUNT=$(nproc)

LAPTOP="4"
if [ "$LAPTOP" == "$COUNT" ];
then
cp /home/tom/Documents/laptop.png /home/tom/Documents/background.png
else
cp /home/tom/Documents/portablepc.png /home/tom/Documents/background.png
fi

#call in .profile

Here's my .profile calling it:

Code: Select all

# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.

# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022

# if running bash
if [ -n "$BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
	. "$HOME/.bashrc"
    fi
fi
# Set wallpaper for appropriate computer as per Tom
./.setwallpaper.sh

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$PATH"
fi

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

[L67GS]

mctom, this might be a starting point for your script modification, but it doesn't look like fun as it is. Perhaps just write a small program in a preferred language to scroll through the filesystem and use a system command to get the age of the file, then delete a file matching the criteria.
https://stackoverflow.com/questions/969 ... ell-script

[mad_ady]

If you don't want to parse dates is to use find . -type d -depth 1 -ctime 30, but you still need to exclude wednesdays, so you still need to pipe it to date, parse it and decide.
The stat command should extract creation/modification date of a file/dir if you don't want to parse it. And date +%a should tell you when it's Wed.

[mctom]

Here's a little one I wrote last year or so to check the number of processor cores

When I was playing with distcc (a compiler distributing its work across multiple machines), I wrote a bash script that returned either 2 or 24, depending on local IP of my netbook. I named it "nproc" and put in ~/.local/bin
Otherwise some build scripts ignored my distcc setup and even though the job distribution worked fine, there were only 2 jobs at a time (the netbook had 2 threads).

use a system command to get the age of the file

That is not what I want to do. As shown in my sample listing, directories are created each day, but their modification date remain largely the same. I have to depend solely on their names.
Would be much easier to include weekday in a directory name, but well..

At least I did the right thing and used constant width fields!

So here are my bits that I came up with so far. I can list directories, sort them, and isolate those older than a month:

find * -maxdepth 0 -type d | sort | grep -B 256 $(date --date="1 month ago" +%Y-%m-%d)


But that works only if a copy exactly one month ago exists, otherwise grep fails.

I'll give it more thought tomorrow..

[mad_ady]

For your date parsing - here's a oneliner that makes use of perl's DateTime module (needs to be installed with sudo apt-get install libdatetime-perl).
Here's some test structure:

Code: Select all

$  ls -1p
2021-11-18_09:19:01/
2021-11-20_12:48:01/
2021-11-22_08:49:01/
2021-11-24_10:26:01/
2021-12-08_10:00:00/
2022-01-11_10:00:00/
2022-01-12_10:00:00/

And the requirement is to return only lines older than 30 days which are not wednesdays, right? Here's my take on it, but please try it yourself on your data to make sure it's correct:

Code: Select all

$ find . -maxdepth 1 -type d | sort | perl -MDateTime -ne 'if (/([0-9]{4})-([0-9]{2})-([0-9]{2})/){ my $dt = DateTime->new( year => $1, month => $2, day => $3); my $now = DateTime->now(); if($now->epoch() - $dt->epoch() > 30 * 24 * 60 * 60 && $dt->day_of_week != 3){ print $_;}}'
./2021-11-18_09:19:01
./2021-11-20_12:48:01
./2021-11-22_08:49:01

Now, it's ugly, but oneliners tend to be that way (and unreadable, unmaintainable...). What it does is the following (for non perl speakers): it parses its input line by line, uses a regex to match yyyy-mm-dd structure and uses parenthesis to save the values matched as $1 for year, $2 for month and $3 for day. If the line matches this regex, it creates a DateTime object based on those values and a "now" object based on local time. It then tests the difference between the unixtime representation of $now and the unixtime representation of $dt and sees if it's more than 30 days. Also, if the day of week is not wednesday, it enters the last if and prints the original line.
So, it should work as a "grep" filter in your case.

Regarding borg-backup, here's some stats about deduplication of content based on backups at work, maybe I can change your mind...


Also, remember! A backup is only as good as its last test restore... So maybe there should be an international backup day where all sysadmins in the world test backups all day.

[mctom]

Thanks again for your comprehensive input. Well, I didn't plan to learn a new language in the process, but I'll consider it, given it might be better suited for the task.

Seeing that your backup has some 80% compression ratio I'm guessing your data is a lot of databases or something.

Also, remember! A backup is only as good as its last test restore...

Made me think I should at least do fsck on it sometimes..
But then again, restore always works for me so far. I sometimes use it to retrieve accidentally deleted files or restoring older versions of my kicad projects (git isn't the best suited tool for that really). Ever since I lost my handcrafted printer driver I do backups of all persistent data, not just home.
I can't even tell how much space is saved on "deduplicated" data. both ncdu and baobab treat each hardlink as a separate file instance and counts it in. Even df reports the filesystem completely full even though it happily accepts new files. Unless there is something very wrong already

That's the rsync summary of today's backup:

Code: Select all

sent 452,120,088 bytes  received 97,231 bytes  1,442,479.49 bytes/sec
total size is 266,492,880,276  speedup is 589.30

And judging from timestamps it took some 5 minutes.

[rooted]

Compression of non-compressed files is great but deduplication is where it gets interesting.

[mctom]

Okay, I tried that borg thing. Now I know what was meant by "investment" regarding this software

So I created a test repository with my home directory. I really want to see my data "deduplicated" in any relevant way.

As it was working on it, I noticed this on their website:

EXPECT THAT WE WILL BREAK COMPATIBILITY REPEATEDLY WHEN MAJOR RELEASE NUMBER CHANGES (like when going from 0.x.y to 1.0.0 or from 1.x.y to 2.0.0).
NOT RELEASED DEVELOPMENT VERSIONS HAVE UNKNOWN COMPATIBILITY PROPERTIES.
THIS IS SOFTWARE IN DEVELOPMENT, DECIDE YOURSELF WHETHER IT FITS YOUR NEEDS.

Not great for long-term storage, but that is probably not the intended use. I honestly don't know what I'm going to do with my copies, perhaps keep an annual one just to browse through it when I retire.

Code: Select all

Creating archive at ".::testrepo"
------------------------------------------------------------------------------
Archive name: testrepo
Archive fingerprint: f51d863743...
Time (start): Tue, 2022-01-18 21:28:15
Time (end):   Tue, 2022-01-18 22:12:31
Duration: 44 minutes 15.57 seconds
Number of files: 72518
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              164.00 GB            152.36 GB            142.63 GB
All archives:              164.00 GB            152.36 GB            142.63 GB

                       Unique chunks         Total chunks
Chunk index:                  114356               128048
------------------------------------------------------------------------------

13% deflation, not bad

Mounting is pretty straightforward, but requires diving into the repository dir and mounting it using command line, and creating a mount point before that, and so on. Since they advertised access via file manager I expected at least mount.sh or something.
Also password is mandatory in every repo, as far as I can tell.
So thanks again, worth knowing there is a tool like that out there, but that's not for me. I'll stick to my physical local copy with ext4 that hopefully will still be around in 2060s.

[brad]

So thanks again, worth knowing there is a tool like that out there, but that's not for me. I'll stick to my physical local copy with ext4 that hopefully will still be around in 2060s.

As long as you don't hit the 2038 bug

Code: Select all

~$ dmesg | grep ext4
[   12.884481] ext4 filesystem being mounted at /boot supports timestamps until 2038 (0x7fffffff)

I think you have gone down the right path parsing the directory names for your current setup to work.

I do agree with mad_ady though in regards to considering a more sophisticated backup strategy if you really value the data you want to back up. I've found out the hard way before, I documented my new process here viewtopic.php?p=312572#p312572

I have 4Tb space (2x4Tb drives mirrored), urbackup essentially hardlinks files that are identical between backups and it is able to use btrfs filesystem to make writable snapshots of individual files so small changes to large files such as image backups, databases, or pst files takes up little space for additional copies / changes. Non of this is full de duplication though that needs to be done at a filesystem level where we find blocks or files that are the same on the entire disk and make them point to the same location, not just between backup copies. I don't do full dedup on the HC4 (not enough RAM for caching dedup tables).

Have backups going back to 2020 of my machines / desktops and it comes in very handy (ever wanted that command you ran last year but it already rolled out of your .bash_history )

[mad_ady]

Since they advertised access via file manager I expected at least mount.sh or something.

Mounting is relatively simple... Once you create your mount.sh
borg mount .::testrepo /mnt/borg

Also password is mandatory in every repo, as far as I can tell.

No, you can bypass it with some environment variables:

Code: Select all

export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes

[mars24568]

Yeah, scripting
A programmer once decided to solve a problem using regex. Now he has two problems...

But what about

Code: Select all

find * -maxdepth 0 -ctime +30 -type d -regextype sed -printf "%Ta %f\n" | sed -n -E -e 's/^Mo |Di |Do |Fr |Sa |So //p'

Finds all directories with a ctime older than 30 days that are NOT on Wednesday. Of course you have to adopt the days-of-week names to your locale. And then pipe the result to

Code: Select all

xargs /bin/rm

or so...

[mctom]

That snippet may come in handy as well, thanks. But it has two problems: I can't use ctime to determine directory age, and I don't want it to be locale dependent. But for the last one, I already found that %u returns weekday number instead.

Yes, I'm afraid I'll have to familiarize with sed in the end.. I do have regex cheatsheet on my board, and I should handle it unless there are any substantial differences.

Also, since I'm no programmer I still count both problems as one.

No, you can bypass it with some environment variables:

That is certainly good to know as well. I'll just assume it can do anything from now on.
I actually spreaded the word about borg to my pal, who is much more paranoid than I am and he loved the idea. Incidentally he's about to ditch zfs due to near-backup-loss experience he doesn't want to talk about.

[mctom]

Here's a little bit of trivia: I saved my little bash experiments by echoing them into *.sh files for later. But when I wanted to run one of them, of course, I had to chmod +x, and then run with ./ prefix.. But i found a quicker way, just bash xx.sh

Anyway, I got tired with that pointless idea of keeping copies from one arbitrarily selected weekday. It's easier to just keep copies from roughly every 10 days, so I'm going to do that instead.

This is what I came up with:

Code: Select all

{ find * -maxdepth 0 -type d; 
echo $(date --date="2 months ago" +%Y-%m-%d)damn_old_backups;} 
| sort 
| sed /damn_old_backups/Q 
| grep -v 20..-..-.5
| xargs rm -r

This is the reasoning:
- list all directories
- append another item to the list, a date two months ago with "damn_old_backups" suffix <- this will ensure script will work properly even if there was no backup at that day
- sort
- print lines before "damn_old_backups" suffix
- do reverse grep, so exclude dates ending with "5" - yields 3 reasonably spaced copies each month

In more detail:
- find is instructed to not go into the directories
- sed just prints stdin until it reaches the line with my magic string, then it exits
- grep -v ignores lines matching pattern "20..-..-.5", where dot is any single character

When directories contain spaces in their names, xargs breaks them into separate arguments. I tried to force xargs to use \n as a delimiter, but then rm would complain, so I gave up. My directories have no spaces, so why would I bother.

Anyway, the script seems to work fine, here's the example with one month old backups removed. I'll wait a few days for any expert veto on my script before I push it "into production".

Code: Select all

[mctom@Tomusiomat test]$ ll
   inode Permissions  Size Blocks User  Group Date Modified Name
 9176986 .rw-r--r--     49      8 mctom mctom 17 sty 20:53  1.sh
 9176987 .rw-r--r--     59      8 mctom mctom 17 sty 20:53  2.sh
11929362 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-11-15_10:17:01
11929380 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-11-25_09:33:01
11929389 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-05_12:17:01
11929387 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-21_10:35:01
11929373 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-22_10:20:01
11929403 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-23_10:06:01
11929382 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-24_03:25:01
11929375 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-25_03:37:01
11929376 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-26_03:21:01
11929407 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-27_03:11:01
11929406 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-28_03:09:01
11929381 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-29_12:46:01
11929395 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-30_10:39:01
11929393 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2021-12-31_13:18:01
11929390 drwxr-xr-x      -      - mctom mctom 17 sty 20:53  2022-01-01_16:40:01
...

There is one backup missing, 2021-12-15, because that was such a busy day I didn't switch on my computer at all

I am amazed how people are able to do everything with sed. That's another item to my bucket list, definitely.

[mctom]

Here's a little bit of trivia: I saved my little bash experiments by echoing them into *.sh files for later. But when I wanted to run one of them, of course, I had to chmod +x, and then run with ./ prefix.. But i found a quicker way, just bash xx.sh

Found even quicker way:

Code: Select all

. xx.sh

It's so convenient it renders execution permissions on bash scripts useless..

anyway, I pushed the code into my backup script. Will it work? we shall see about that tomorrow.

Also, about that:

Even df reports the filesystem completely full even though it happily accepts new files. Unless there is something very wrong already

It turned out the df report was actually correct: I have less than 5% of free space on that drive. I wasn't aware 5% is reserved for root until today.
So, my backup reducing scheme came just in time. Alas I fear it may not save me enough space, especially when I want to keep one every 10 days.
Well, in that case.. borg backups, I think. With a decent mounting script.
I wonder if I could stuff existing backups into this.

[mad_ady]

There's a subtle difference between executing a script (./script) and sourcing it (. script) explained here: https://superuser.com/questions/176783/ ... ourcing-it

As for the 5% - the value can be tweaked with tune2fs

[mctom]

Yep, I learned about sourcing yesterday when I was wondering if I can use "cd" in bash script, and google led me to basic sourcing examples.

I'm fine with 5% reserved for root - after all root does the backup.

Also, the script actually works with the changes, and reclaimed 1% of free space!
But I think borg is the destiny - lately I installed and uninstalled 6GB package with kicad 3d models a few times - and I'll end up with this many copies of it. :/

[mctom]

So I got a pulseoxymeter today and didn't hesitate to make it worthwhile.

It's Creative PC-60FW that I paid some $30 for. Not THAT Creative brand you might be thinking about.
Far from medical grade, but on the other hand I did have some experience with similar medical measurement techniques back at the internship in Finland. Techniques are simple enough I trust a $30 device may actually work well.
Too bad I can't alter my saturation at will to test it, but I am known to drop it significantly in my sleep, so we shall see the results shortly.

The device has "wireless link" which is BLE, but they fail to mention it anywhere - not on a box, in the manual or anywhere. Perhaps because it would fail miserably at any attempt of certification.
The Android App has so poor reviews I didn't even bother cheching it out. Also my Mandarin is still not fluent enough
Radio cannot be turned off to save battery, so I might as well log all the data it spits out.
I bought antennas for my Dell Wyse terminal, and I'm really surprised how good reception it has. A metal frame of my bed should not be a problem.

I found a script prepared by some guy that does the job, but is more of a proof of concept. So I did some fixing, tweaking, and added CSV logging for my needs, and of course pushed all that stuff on github. (and pull request too)
https://github.com/tomek-szczesny/viatom_pc60fw

Now the script is in good enough condition to try some testing on a living patient. I can run it, it waits for the device to be discovered, logs data and exits once device disconnects.
I'd like to make this script run all the time, so I won't forget to run it manually before sleep. Something that will restart it once it exits..
I'm thinking about systemd service. Is this a correct way to do it?

[rooted]

I just use the oxygen meter on my watch, I have one for the finger as well and they are both within the margin of error.

Is there a reason you are monitoring your o2?

[mad_ady]

I'm thinking about systemd service.

That's the recommended way, yes. Something like ths that restarts after 5s

Code: Select all

#  cat /etc/systemd/system/ir-ac-mqtt-agent.service
[Unit]
Description=IR AC MQTT Agent
After=network.target

[Service]
ExecStart=/home/adrianp/bin/ir-ac-mqtt-agent.py
Type=simple
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

[mctom]

Stupid me, I edited out my own post, instead of replying.

I just use the oxygen meter on my watch, I have one for the finger as well and they are both within the margin of error.

Is there a reason you are monitoring your o2?

That's good news, thanks. I'd compare my pulseoximeter against the one in a hospital, if I get admitted there again.
For now I see it has different results for different fingers - I get up to 99% on a middle one, and just got 88% on a thumb.

I was diagnosed with a very bad case of sleep apnea. Before I can hope for any treatment to take effect, I got this pulseoximeter with alarm function to wake me up every time I stop breathing in my sleep. The reasoning here is to at least stop suffocating at night and feeling like a zombie every morning.
Logging is not critical for my cause, but as I said, I cannot turn off radio in this device, so I might as well log it. It may provide some statistics whether my condition is improving or not, or additional data for my doctor.
Last night the alarm thing woke me up some 10-20 times. I think the 95% threshold was too high and kicked in at random.
It also dropped connection every 7-8 minutes so I had to cat the results together.

That's the recommended way, yes. Something like ths that restarts after 5s

For the last night, I used watch to restart the script every time it stopped. But yeah, I'll do it the proper way. Thanks for help!

On a second though, I want it to log results into a home directory of a user. Is there a way to run a service as a non-root user?

[mad_ady]

Last night the alarm thing woke me up some 10-20 times.

When the kids were little they'd wake me up about 3-4 times each night and I'd feel like crap in the morning. I can't imagine how waking up 10-20 times a night makes you feel like... Somehow I doubt that's the answer to your problem.

Regarding running systemd as a user - sure. Add

Code: Select all

User=xxx

and you're done.

[mctom]

Oh yes, when my cat is about to barf (always at night, about once a week) I've got to jump down the bed and try catching the payload before he does The night is always ruined.
However this pulseoximeter is very, very silent - could be unnoticed during the day. The point is to nudge me if I'm not breathing, which would wake me up eventually anyway. Not that bad, given I always wake up a few times during the night.

Thanks for that, I'll construct this service tonight.

[L67GS]

You could use that thing to trigger a stun gun tucked in your underwear at night. Then sleeping would be such a terrifying prospect that you may learn to live without it.

[mctom]

Just to let you know, while you were enjoying yourselves throughout the weekend I pulled the last hair out my dyed head, working on a desktop video stream across LAN!
(Which started as a "how to Netflix on XU4" thread but OK..)

Here is the fresh repository, and there is a long discussion.
Thanks again @mad_ady, @rooted, @odroid for all your inspiration and input!

[joerg]

Thanks to @mctom existing this thread.
I have a question about passing random filename to ffmpeg as output filename. Trying and trying and trying..., but I can't get it work.
This I am doing in a loop:

Code: Select all

            rand=$RANDOM | md5sum | head -c 20;
            export filename=$rand.mp3
            echo $filename
            ffmpeg -nostdin -f pulse -i combined.monitor -codec:a libmp3lame -qscale:a 0 $filename

But the result is always a '.mp3' file.
I have tried with named pipe, with substitution, but I can't get it work.
I want to have a random filename, I can get the actual track from playerctl and when it changes I want to rename the random file.
But I can't get rid of '.mp3' or 'At least one output file must be specified'.

[mctom]

Well, how about creating a file with a known name, and renaming it into a random one after ffmpeg is done?
Also, perhaps ffmpeg has an explicit option to set output filename, in that case it wouldn't have to guess that the last bit is indeed a filename.

[joerg]

To explain more exactly, the player changes the metadata to null between two songs. This is the trigger to stop ffmpeg and start it again. At this moment the next song titel is unkown. I need a random filename vor ffmpeg. The former random file I can now rename to the former title.
I have read so much about how to pass a variable to a subshell, but everything I tried, the variable stays empty.
And no, ffmpeg wants the filename as last argument. But it is not a ffmpeg problem, the problem ist to pass a variable to subshell. I have the same when I try parec and twolame combination.

[mctom]

Oh, okay, I misunderstood, sorry.

I think there's an error in line #1. It uses $RANDOM as the first command in the pipeline (which obviously cannot be found), thus weird stuff is going on. When I tested this code, I always got the same "random" file name.

Here's something that works for me:

Code: Select all

[mctom@Tomusiomat tmp]$ bat test.sh 
───────┬──────────────────────────────────────────────────────────────────────────
       │ File: test.sh
       │ Size: 128 B
───────┼──────────────────────────────────────────────────────────────────────────
   1   │ #!/bin/bash
   2   │ for i in 1 2 3
   3   │ do
   4   │     rand= echo $RANDOM | md5sum | head -c 20;
   5   │     filename=$rand.mp3
   6   │     echo $filename
   7   │ done
───────┴──────────────────────────────────────────────────────────────────────────
[mctom@Tomusiomat tmp]$ ./test.sh 
c6d2be47ed1378353730.mp3
d55824e6054e20be53df.mp3
083833ef0d75f1a38791.mp3

EDIT:
By the way, the same filename I always got was: d41d8cd98f00b204e980.mp3. This is the value that md5sum returns with no input:

Code: Select all

[mctom@Tomusiomat tmp]$ echo -n "" |md5sum
d41d8cd98f00b204e9800998ecf8427e  -

[joerg]

@mctom, thank you to guide me to the right direction. Yes, I saw also that the random filename was always the same. With your variant it generates always new random string, but it don't wanted to work at all to pass this variable to sub shell, always empty. So there must be something with $RANDOM. I searched for other possibility to create random string and came to rand=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 20 | head -n 1) that works now and the created variable can be passed to sub shell. Here is the complete script that grabs the pulseaudio stream, waits until track change, stops and starts a new grab session and renames the grabbed to song title:

Code: Select all

#!/bin/sh
echo "grabbing script"
titleold=$(playerctl metadata title)
while true
do
    title=$(playerctl metadata title)
    if [ "$title" != "$titleold" ]; then
        echo "title: ${title}"
        if [ -z "${title}" ]; then
            kill -9 $important_pid
            mv $filename "./$titleold.mp3"
            rand=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 20 | head -n 1)
            filename="$rand.mp3";
            parec -d combined.monitor | twolame -r -V 10 - ${filename} &
            important_pid=$!;
        fi
        titleold=$title
    fi
done

I know it's beginners code. I am no scripting enthusiast.
I will fine adjust it more and more, to get also artist and maybe album to save the tracks to the right folder. And also some checks if a variable is empty to not get some error messages, eg. when at first run the pid does not exist kill errors out.

[mctom]

So adding "echo" to the first line of your original script didn't resolve the problem?
I understand you echo'ed $filename and saw you are getting random file names. And then when using the same variable in "parec", you got empty file names again?
That's extraordinarily odd. I don't think it has anything to do with $RANDOM, but really can't think of any explanation either.

Unless you had empty "filename" variable exported in previous experiments, and sub-shell used that instead?

Nevertheless I'm glad you found a solution that works for you.

[joerg]

Yes when I use $RANDOM as your example I get empty variable in sub shell. This problem has made me loosing half day yesterday ...

[mctom]

Experience is knowledge gained in the worst possible way.

[mctom]

"Coding" FPGA is such a different paradigm it does impact abilities of doing the "traditional" coding afterwards. It's too easy and tempting to get used to that everything works in parallel and at the same time.
So I had a little idea to implement some formulas from ancient books of electromagnetic wisdom. Quite complicated ones too, so needed to be broken into pieces. And I had too hard time trying to stuff them into functions and doing function matryoshka, and keeping the execution order in check.

In the end I wrote a C++ program that is entirely #define and cout Give or take three functions that in fact are aliases to std::comp_ellint_1 because it's darn too long to remember.

https://github.com/tomek-szczesny/edge_ ... d_cpwg.cpp