Cloning EMMC/SD partitioning layout

Post Reply
User avatar
mad_ady
Posts: 5419
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Contact:

Cloning EMMC/SD partitioning layout

Unread post by mad_ady » Fri Feb 05, 2016 5:27 pm

Hello all,

I want to make backups of my EMMC/SD card in the most efficient way. I'd like to be able to clone and restore MBR, bootloader and any partitions with the added bonus of being able to restore them on a different size card.
I've tried dd-ing the whole card, and while it worked, it was slow for 32GB...

I've also looked at loboris' scripts because they have backup/restore functionality, but it only works with his partition layout.

Before I go out and reinvent the wheel, I'd like to ask if there's already a way of doing what I'd like.

From what I've seen, it seems that:
1. The MBR is the first 512 bytes from the disk, extractable with

Code: Select all

dd if=/dev/mmcblk0 of=mbr.bin bs=1 count=512
2. The bootloader appears to be before partition 1, in the first 3072 blocks (about 1.6MB). The bootloader starts immediately after the MBR, followed by an empty section and then its configuration, followed by more zeros. Also easily extractable with dd.
3. Each partition is extractable with rsync, or partimage. Except partimage can't handle ext4. If I use rsync, I have flexibility when it comes to destination partition size (after fixing the mbr). But the problem is I have to keep track of UUIDs and fix boot.ini and /etc/fstab if uuid changes. Rsync would be nice because you could easily refresh an older backup without transferring everything.

My question is - what is your preferred backup/restore method for your cards? Maybe I missed something obvious...
Thanks

User avatar
meveric
Posts: 9762
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by meveric » Fri Feb 05, 2016 7:43 pm

I just use dd to backup the entire SD/eMMC, or better to say, I shrink the partition to a minimum and only copy as much sectors as there are in use, this keeps the images to a minimum and is easy/fast to flash.
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.

User avatar
mad_ady
Posts: 5419
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by mad_ady » Fri Feb 05, 2016 9:46 pm

Too much work... surely there must be an easier way/tool...

User avatar
mad_ady
Posts: 5419
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by mad_ady » Fri Feb 05, 2016 11:20 pm

I'm going to give fsarchiver a try... Looks nice: http://www.fsarchiver.org/Fsarchiver_vs_partimage

User avatar
meveric
Posts: 9762
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by meveric » Sat Feb 06, 2016 2:08 am

I just checked on fsarchiver. I don't think that's what you want.

You could do the very same with just using tar. Cause it does nothing besides copying all files on a partition and save them in a compressed archive.
That's about it.. probably some meta information as well, but you can't create a working ODROID image out of it.

The fat boot partition is not even supported from fsarchiver (at least not in the Ubuntu 12.04) version. Which means you can't even backup that.
Nor will it backup what you call the "MBR". Which means you would have to do both separately.
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.

User avatar
mad_ady
Posts: 5419
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by mad_ady » Sat Feb 06, 2016 3:08 pm

Well, I was thinking of using dd to copy and restore mbr and bootloader and fsarchiver to backup and restore vfat/ext4 partitions. It operates at file level like tar but should save all attributes. I've also read it needs mkfs.vfat, mkfs.ext4 to support those filesystems.

I haven't tried it, but I will as soon as I get my next emmc and I'll report back.

User avatar
meveric
Posts: 9762
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by meveric » Sat Feb 06, 2016 4:17 pm

mad_ady wrote:Well, I was thinking of using dd to copy and restore mbr and bootloader and fsarchiver to backup and restore vfat/ext4 partitions. It operates at file level like tar but should save all attributes. I've also read it needs mkfs.vfat, mkfs.ext4 to support those filesystems.
mad_ady wrote:Too much work... surely there must be an easier way/tool...
and in what way is that "less work" then just using dd to make a backup of the entire card with all the same information which doesn't need mkfs and of other things? O_o
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.

User avatar
mad_ady
Posts: 5419
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by mad_ady » Sun Feb 07, 2016 2:09 am

dd can take forever to complete for large cards, over network and also potentially wastes a lot of space (unless you pipe it through gzip or something). The last backup I made with dd over network (from a downclocked xu3) took a few hours to complete for 32GB of mostly zeros. Other than that dd is pretty easy to use.

LiquidAcid
Posts: 1091
Joined: Fri Oct 11, 2013 11:07 pm
languages_spoken: english
ODROIDs: X2
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by LiquidAcid » Sun Feb 07, 2016 2:29 am

mad_ady wrote:But the problem is I have to keep track of UUIDs and fix boot.ini and /etc/fstab if uuid changes. Rsync would be nice because you could easily refresh an older backup without transferring everything.
The question here is what you understand under "cloning". Does it include replicating the UUIDs or does it not? If it does, just use something like blkid to backup the filesystem UUIDs (the partition UUIDs are stored in the partition table anyway) and then restore them when creating the filesystems. This is scriptable with a few lines of shell.
In case you want to keep the filesystem UUIDs really unique, I'd ask the question first where the IDs are used. And the answer is probably just fstab and the bootloader script. Create a template version of both, get the filesystem UUIDs after creating the filesystems, put these into fstab and bootscript (e.g. with sed).

User avatar
mad_ady
Posts: 5419
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by mad_ady » Sun Feb 07, 2016 4:02 am

Well, whatever works best I guess. What would be the worst thing to expect in a system where I have two partitions with the same UUID? Apart from ambiguity while mounting by UUID? I agree, it's quite scriptable. I'll try to put something togethr as soon as I need it :)

User avatar
meveric
Posts: 9762
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by meveric » Sun Feb 07, 2016 7:54 am

mad_ady wrote:dd can take forever to complete for large cards, over network and also potentially wastes a lot of space (unless you pipe it through gzip or something). The last backup I made with dd over network (from a downclocked xu3) took a few hours to complete for 32GB of mostly zeros. Other than that dd is pretty easy to use.
THAT IS why you SHRINK the partion first, as I said earlier. You can even use a easy to use graphical tool like gparted. Which makes it really easy.. and later you use dd to only backup the part where you have DATA in it and not all the zeros.. That's what I said earlier.. And that still sounds a lot easier than using dd to backup the "MBR" using a different tool to backup the files of the system and later on having to use multiple tools to restore the partition layout and file again.
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.

LiquidAcid
Posts: 1091
Joined: Fri Oct 11, 2013 11:07 pm
languages_spoken: english
ODROIDs: X2
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by LiquidAcid » Sun Feb 07, 2016 8:05 am

meveric wrote:THAT IS why you SHRINK the partion first, as I said earlier.
In this case I have to modify the clone source. If I have a system I'm using I don't want to shrink my rootfs, dd, and then resize again. That's just painful IMO.
meveric wrote:You can even use a easy to use graphical tool like gparted.
On the contrary. I can't script anything when I use a GUI based tool.
meveric wrote:Which makes it really easy.. and later you use dd to only backup the part where you have DATA in it and not all the zeros.. That's what I said earlier.. And that still sounds a lot easier than using dd to backup the "MBR" using a different tool to backup the files of the system and later on having to use multiple tools to restore the partition layout and file again.
You write the clone/restore shell script once. And what you need is part of any Linux distro anyway: fdisk or some variant, mkfs.vfat, mkfs.ext4, blkid, tar

User avatar
meveric
Posts: 9762
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by meveric » Sun Feb 07, 2016 8:12 am

might be true, but that's different to the approach with fsarchiver which is not part of the linux distro.

It's the question if the backup process should be easy or automated.

And if you want to create a "bootable" backup using dd is the easiest way it gets and it's also part of the distro.

Besides that.. I said you could "EVEN" use a graphical tool. No one said you HAVE to use it. You can shrink the partition with commands as well if that's what you want.
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.

LiquidAcid
Posts: 1091
Joined: Fri Oct 11, 2013 11:07 pm
languages_spoken: english
ODROIDs: X2
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by LiquidAcid » Sun Feb 07, 2016 8:34 am

meveric wrote:might be true, but that's different to the approach with fsarchiver which is not part of the linux distro.
I never talked about fsarchive?
meveric wrote:It's the question if the backup process should be easy or automated.
I don't see any contradictions here. If I can script something it becomes easy since all subsequent invocations are handled by that script. You make it sound like this is a XOR.
meveric wrote:And if you want to create a "bootable" backup using dd is the easiest way it gets and it's also part of the distro.
Using just dd to dump the block device was already ruled out by mad_ady. I also smell Maslow's hammer here. dd is a good tool, but you should apply it where it makes sense.
meveric wrote:Besides that.. I said you could "EVEN" use a graphical tool. No one said you HAVE to use it. You can shrink the partition with commands as well if that's what you want.
Let me repeat. This shrinking thing is bollocks if you want to keep using that system. Also it already fails when you have multiple partitions. I have rootfs, portage, home and var on different partitions. With the shrinking method I would have to shrink each partition/fs first, then also move it around to finally create an image of the block device. And don't get me started about fragmentation...

User avatar
meveric
Posts: 9762
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by meveric » Sun Feb 07, 2016 8:45 am

LiquidAcid wrote:I never talked about fsarchive?
good thing not everyone is talking about your solution either. The original poster was suggesting that tool and I was answering to that approach.
LiquidAcid wrote:I don't see any contradictions here. If I can script something it becomes easy since all subsequent invocations are handled by that script. You make it sound like this is a XOR.
The contradiction is that not everyone can do it the way you have on your mind. For you this might be "easy". If it was "that easy" for everyone, the question wouldn't have been raised in the forum.
LiquidAcid wrote:Using just dd to dump the block device was already ruled out by mad_ady.
you might want to read that statement again.
It was "ruled out" cause of the huge size of the resulting image, if you copy an entire SD card and not only the part that contains data.
Therefore reducing the the image to a size only containing the data is a lot faster, it's a difference if you have a 32 GB size backup or a 3 GB size backup (which you can compress again as well).
LiquidAcid wrote:Also it already fails when you have multiple partitions.
Images form HardKernel comes with at least 2 partitons boot and rootfs.. TWO is more than one.. more than one is called "multiple" shrinking rootfs partition, making a backup of all with dd works fine and boots every time.
I'd say your statement is invalid in that case.
LiquidAcid wrote:I have rootfs, portage, home and var on different partitions.
good for you, not everyone does.
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.

LiquidAcid
Posts: 1091
Joined: Fri Oct 11, 2013 11:07 pm
languages_spoken: english
ODROIDs: X2
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by LiquidAcid » Sun Feb 07, 2016 10:27 am

meveric wrote:good thing not everyone is talking about your solution either. The original poster was suggesting that tool and I was answering to that approach.
The discussion concerning the usage of dd is completly orthogonal to the one involving fsarchiver. It don't see why you keep bringing this up, it's irrelevant here.
meveric wrote:The contradiction is that not everyone can do it the way you have on your mind. For you this might be "easy". If it was "that easy" for everyone, the question wouldn't have been raised in the forum.
No sorry, that's not what you were saying.

Direct quote from you: "It's the question if the backup process should be easy or automated."

You were implying that you can either have easy or automated. And in particular you were referring to the "backup process". And that's exactly what I refer to. I haven't said anything about writing said script. Please stop pulling this taking-out-of-context crap, it's just annoying.
meveric wrote:It was "ruled out" cause of the huge size of the resulting image, if you copy an entire SD card and not only the part that contains data.
Therefore reducing the the image to a size only containing the data is a lot faster, it's a difference if you have a 32 GB size backup or a 3 GB size backup (which you can compress again as well).
You keep ignoring the fact that your shrink approach is flawed to begin with.
meveric wrote:Images form HardKernel comes with at least 2 partitons boot and rootfs.. TWO is more than one.. more than one is called "multiple" shrinking rootfs partition, making a backup of all with dd works fine and boots every time.
I'd say your statement is invalid in that case.
Oh, now you're already arguing trivial things, heh? Then let me rephrase it. Your method already break together when you have two partitions that would need a resize (which leaves anything "small" like boot out of the picture). Clear enough for you?
meveric wrote:good for you, not everyone does.
Good for me, and also good for everyone else, since the method is more generic. I wouldn't want to restrict myself to a single partition layout.

User avatar
mad_ady
Posts: 5419
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by mad_ady » Sun Feb 07, 2016 3:10 pm

Let's not start a flame war here guys :) The good thing about open source is that you may have multiple solutions to a problem. I'll see what's best in my case.

I would prefer an automated tool to speed up the process, and also I liked fsarchiver's idea because it doesn't backup blocks (you could restore to a different filesystem type). You can even tell it to ignore certain files or directories if you want to customize the backup.
I didn't particularly like the shrink + dd solution because you need to grow it back on the destination. I know that there's a boot script that extends it automatically, but if I'm not mistaking, it getsgets deactivated on first boot.

We'll see how it works. Thank you for your sugestions

User avatar
mad_ady
Posts: 5419
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by mad_ady » Wed Mar 23, 2016 5:18 pm

Just for further reference - here is how I backup and restore my devices. With a combination of dd (for partition table and bootloader, partimage (for FAT partitions) and fsarchiver for ext3/4 (actually I use this GUI because I'm lazy: https://sourceforge.net/projects/qt4-fsarchiver/)

Step 1: Get the offsets for the bootloader. It starts from the beginning of the device and ends where the first partition starts:

Code: Select all

$ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 14.6 GiB, 15634268160 bytes, 30535680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd3630000

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sdc1  *      2048   264191   262144  128M  c W95 FAT32 (LBA)
/dev/sdc2       264192 30535679 30271488 14.4G 83 Linux
In the output above you can see that the sector size is 512 bytes and the first partition starts on sector 2048. You'll use this information to extract the first 2048 512 bytes sectors (actually you'd need just the first 2047 sectors).

Code: Select all

$ sudo dd if=/dev/sdc of=mbr.bin bs=512 count=2048 
2048+0 records in
2048+0 records out
1048576 bytes (1.0 MB) copied, 0.0364304 s, 28.8 MB/s
Step 2: Backup FAT partitions

Code: Select all

$ sudo partimage -z1 -d --volume=2048 -M save /dev/sdc1 sdc1.img
Step 3: Use fsarchiver

Code: Select all

$ sudo fsarchiver -v -L "rootfs" -s 2048 -j 2 savefs sdc2.fsa /dev/sdc2
The restore steps are done similarly.

tjohannvr
Posts: 30
Joined: Tue Apr 12, 2016 11:29 pm
languages_spoken: english
ODROIDs: ODROID XU4 * 4
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by tjohannvr » Wed May 18, 2016 6:11 am

Hi, you said restoring is similar. What are the exact commands you use for restoring? Thanks!

User avatar
mad_ady
Posts: 5419
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Contact:

Re: Cloning EMMC/SD partitioning layout

Unread post by mad_ady » Wed May 18, 2016 3:13 pm

Sure,

Assuming that the card you want to restore to is /dev/sdb, do these:
1. Restore partition table and bootloader:

Code: Select all

sudo dd if=mbr.bin of=/dev/sdb bs=1M
2. Restore FAT partition

Code: Select all

sudo partimage -o restore /dev/sdb1 sdc1.img
3. Restore ext partitions

Code: Select all

sudo fsarchiver restfs sdc2.fsa /dev/sdb2
You can restore only one step if you don't need to restore everything. If your disk is seen as mmcblk, you may need to remove and reinsert it after restoring the partition table so that the kernel has a chance to create the partition structure.

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 2 guests