[U-Boot] [PATCH] fs: fat: handle deleted directory entries correctly
Unlink test for FAT file system seems to fail at test_unlink2. (When I added this test, I haven't seen any errors though.) for example, ===8<=== fs_obj_unlink = ['fat', '/home/akashi/tmp/uboot_sandbox_test/128MB.fat32.img']
def test_unlink2(self, u_boot_console, fs_obj_unlink): """ Test Case 2 - delete many files """ fs_type,fs_img = fs_obj_unlink with u_boot_console.log.section('Test Case 2 - unlink (many)'): output = u_boot_console.run_command('host bind 0 %s' % fs_img)
for i in range(0, 20): output = u_boot_console.run_command_list([ '%srm host 0:0 dir2/0123456789abcdef%02x' % (fs_type, i), '%sls host 0:0 dir2/0123456789abcdef%02x' % (fs_type, i)]) assert('' == ''.join(output))
output = u_boot_console.run_command( '%sls host 0:0 dir2' % fs_type)
assert('0 file(s), 2 dir(s)' in output)
E AssertionError: assert '0 file(s), 2 dir(s)' in ' ./\r\r\n ../\r\r\n 0 0123456789abcdef11\r\r\n\r\r\n1 file(s), 2 dir(s)'
test/py/tests/test_fs/test_unlink.py:52: AssertionError ===>8===
This can happen when fat_itr_next() wrongly detects an already- deleted directory entry.
File deletion, which was added in the commit f8240ce95d64 ("fs: fat: support unlink"), is implemented by marking its entry for a short name with DELETED_FLAG, but related entry slots for a long file name are kept unmodified. (So entries will never be actually deleted from media.)
To handle this case correctly, an additional check for a directory slot will be needed in fat_itr_next().
In addition, I added extra comments about long file name and short file name format in FAT file system. Although they are not directly related to the issue, I hope it will be helpful for better understandings in general.
Signed-off-by: AKASHI Takahiro takahiro.akashi@linaro.org --- fs/fat/fat.c | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)
diff --git a/fs/fat/fat.c b/fs/fat/fat.c index 9e1b842dac6b..68ce65838678 100644 --- a/fs/fat/fat.c +++ b/fs/fat/fat.c @@ -869,6 +869,14 @@ static dir_entry *extract_vfat_name(fat_itr *itr) return NULL; }
+ /* + * We are now at the short file name entry. + * If it is marked as deleted, just skip it. + */ + if (dent->name[0] == DELETED_FLAG || + dent->name[0] == aRING) + return NULL; + itr->l_name[n] = '\0';
chksum = mkcksum(dent->name, dent->ext); @@ -898,6 +906,16 @@ static int fat_itr_next(fat_itr *itr)
itr->name = NULL;
+ /* + * One logical directory entry consist of following slots: + * name[0] Attributes + * dent[N - N]: LFN[N - 1] N|0x40 ATTR_VFAT + * ... + * dent[N - 2]: LFN[1] 2 ATTR_VFAT + * dent[N - 1]: LFN[0] 1 ATTR_VFAT + * dent[N]: SFN ATTR_ARCH + */ + while (1) { dent = next_dent(itr); if (!dent) @@ -910,7 +928,17 @@ static int fat_itr_next(fat_itr *itr) if (dent->attr & ATTR_VOLUME) { if ((dent->attr & ATTR_VFAT) == ATTR_VFAT && (dent->name[0] & LAST_LONG_ENTRY_MASK)) { + /* long file name */ dent = extract_vfat_name(itr); + /* + * If succeeded, dent has a valid short file + * name entry for the current entry. + * If failed, itr points to a current bogus + * entry. So after fetching a next one, + * it may have a short file name entry + * for this bogus entry so that we can still + * check for a short name. + */ if (!dent) continue; itr->name = itr->l_name; @@ -919,8 +947,11 @@ static int fat_itr_next(fat_itr *itr) /* Volume label or VFAT entry, skip */ continue; } - } + } else if (!(dent->attr & ATTR_ARCH) && + !(dent->attr & ATTR_DIR)) + continue;
+ /* short file name */ break; }
On Tue, Nov 26, 2019 at 05:29:31PM +0900, AKASHI Takahiro wrote:
Unlink test for FAT file system seems to fail at test_unlink2. (When I added this test, I haven't seen any errors though.)
FYI, once the following patches are merged, file system pytests under /test/py/tests/test_fs will start to be invoked for sandbox build on Travis CI.
[1] https://lists.denx.de/pipermail/u-boot/2019-November/391743.html [2] https://lists.denx.de/pipermail/u-boot/2019-November/391742.html
Thanks, -Takahiro Akashi
for example, ===8<=== fs_obj_unlink = ['fat', '/home/akashi/tmp/uboot_sandbox_test/128MB.fat32.img']
def test_unlink2(self, u_boot_console, fs_obj_unlink): """ Test Case 2 - delete many files """ fs_type,fs_img = fs_obj_unlink with u_boot_console.log.section('Test Case 2 - unlink (many)'): output = u_boot_console.run_command('host bind 0 %s' % fs_img) for i in range(0, 20): output = u_boot_console.run_command_list([ '%srm host 0:0 dir2/0123456789abcdef%02x' % (fs_type, i), '%sls host 0:0 dir2/0123456789abcdef%02x' % (fs_type, i)]) assert('' == ''.join(output)) output = u_boot_console.run_command( '%sls host 0:0 dir2' % fs_type)assert('0 file(s), 2 dir(s)' in output)E AssertionError: assert '0 file(s), 2 dir(s)' in ' ./\r\r\n ../\r\r\n 0 0123456789abcdef11\r\r\n\r\r\n1 file(s), 2 dir(s)'
test/py/tests/test_fs/test_unlink.py:52: AssertionError ===>8===
This can happen when fat_itr_next() wrongly detects an already- deleted directory entry.
File deletion, which was added in the commit f8240ce95d64 ("fs: fat: support unlink"), is implemented by marking its entry for a short name with DELETED_FLAG, but related entry slots for a long file name are kept unmodified. (So entries will never be actually deleted from media.)
To handle this case correctly, an additional check for a directory slot will be needed in fat_itr_next().
In addition, I added extra comments about long file name and short file name format in FAT file system. Although they are not directly related to the issue, I hope it will be helpful for better understandings in general.
Signed-off-by: AKASHI Takahiro takahiro.akashi@linaro.org
fs/fat/fat.c | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)
diff --git a/fs/fat/fat.c b/fs/fat/fat.c index 9e1b842dac6b..68ce65838678 100644 --- a/fs/fat/fat.c +++ b/fs/fat/fat.c @@ -869,6 +869,14 @@ static dir_entry *extract_vfat_name(fat_itr *itr) return NULL; }
/*
* We are now at the short file name entry.* If it is marked as deleted, just skip it.*/if (dent->name[0] == DELETED_FLAG ||
dent->name[0] == aRING)return NULL;itr->l_name[n] = '\0';
chksum = mkcksum(dent->name, dent->ext);
@@ -898,6 +906,16 @@ static int fat_itr_next(fat_itr *itr)
itr->name = NULL;
- /*
* One logical directory entry consist of following slots:* name[0] Attributes* dent[N - N]: LFN[N - 1] N|0x40 ATTR_VFAT* ...* dent[N - 2]: LFN[1] 2 ATTR_VFAT* dent[N - 1]: LFN[0] 1 ATTR_VFAT* dent[N]: SFN ATTR_ARCH*/- while (1) { dent = next_dent(itr); if (!dent)
@@ -910,7 +928,17 @@ static int fat_itr_next(fat_itr *itr) if (dent->attr & ATTR_VOLUME) { if ((dent->attr & ATTR_VFAT) == ATTR_VFAT && (dent->name[0] & LAST_LONG_ENTRY_MASK)) {
/* long file name */ dent = extract_vfat_name(itr);/** If succeeded, dent has a valid short file* name entry for the current entry.* If failed, itr points to a current bogus* entry. So after fetching a next one,* it may have a short file name entry* for this bogus entry so that we can still* check for a short name.*/ if (!dent) continue; itr->name = itr->l_name;@@ -919,8 +947,11 @@ static int fat_itr_next(fat_itr *itr) /* Volume label or VFAT entry, skip */ continue; }
}
} else if (!(dent->attr & ATTR_ARCH) &&!(dent->attr & ATTR_DIR))continue;/* short file name */break; }
-- 2.24.0
On Tue, Nov 26, 2019 at 05:29:31PM +0900, AKASHI Takahiro wrote:
Unlink test for FAT file system seems to fail at test_unlink2. (When I added this test, I haven't seen any errors though.) for example, ===8<=== fs_obj_unlink = ['fat', '/home/akashi/tmp/uboot_sandbox_test/128MB.fat32.img']
def test_unlink2(self, u_boot_console, fs_obj_unlink): """ Test Case 2 - delete many files """ fs_type,fs_img = fs_obj_unlink with u_boot_console.log.section('Test Case 2 - unlink (many)'): output = u_boot_console.run_command('host bind 0 %s' % fs_img) for i in range(0, 20): output = u_boot_console.run_command_list([ '%srm host 0:0 dir2/0123456789abcdef%02x' % (fs_type, i), '%sls host 0:0 dir2/0123456789abcdef%02x' % (fs_type, i)]) assert('' == ''.join(output)) output = u_boot_console.run_command( '%sls host 0:0 dir2' % fs_type)assert('0 file(s), 2 dir(s)' in output)E AssertionError: assert '0 file(s), 2 dir(s)' in ' ./\r\r\n ../\r\r\n 0 0123456789abcdef11\r\r\n\r\r\n1 file(s), 2 dir(s)'
test/py/tests/test_fs/test_unlink.py:52: AssertionError ===>8===
This can happen when fat_itr_next() wrongly detects an already- deleted directory entry.
File deletion, which was added in the commit f8240ce95d64 ("fs: fat: support unlink"), is implemented by marking its entry for a short name with DELETED_FLAG, but related entry slots for a long file name are kept unmodified. (So entries will never be actually deleted from media.)
To handle this case correctly, an additional check for a directory slot will be needed in fat_itr_next().
In addition, I added extra comments about long file name and short file name format in FAT file system. Although they are not directly related to the issue, I hope it will be helpful for better understandings in general.
Signed-off-by: AKASHI Takahiro takahiro.akashi@linaro.org
Applied to u-boot/master, thanks!
Hi!
I recently stumble over FAT partitioning issue. I have a device with MMC, one partition of which has been exported as disk via USB Mass Storage to the host. When Windows 10 sees that disk it can't handle it till I format it there. So far so good. However, the same partition can't be used under U-Boot due to Windows flow, i.e. it makes a partitioning on top of the actual partition while U-Boot expects that we only have one MBR (or partitioning) on the dist. So, the commands such fatls, fatload do not recognise any file there.
In Linux it's easy to use: mount -o loop,offset=65536 /dev/mmcblk0p9 /mnt (offset can be different)
But I would like to use it in U-Boot.
Any thoughts on this?
---------- Forwarded message --------- From: Andy Shevchenko andy.shevchenko@gmail.com Date: Mon, Jan 13, 2020 at 12:52 PM Subject: fat: handle Windows formatted partition (thru USB Mass Storage) To: AKASHI Takahiro takahiro.akashi@linaro.org, Heinrich Schuchardt xypron.glpk@gmx.de Cc: Tom Rini trini@konsulko.com, U-Boot Mailing List u-boot@lists.denx.de, Simon Glass sjg@chromium.org, Bin Meng bmeng.cn@gmail.com
Hi!
I recently stumble over FAT partitioning issue. I have a device with MMC, one partition of which has been exported as disk via USB Mass Storage to the host. When Windows 10 sees that disk it can't handle it till I format it there. So far so good. However, the same partition can't be used under U-Boot due to Windows flow, i.e. it makes a partitioning on top of the actual partition while U-Boot expects that we only have one MBR (or partitioning) on the dist. So, the commands such fatls, fatload do not recognise any file there.
In Linux it's easy to use: mount -o loop,offset=65536 /dev/mmcblk0p9 /mnt (offset can be different)
But I would like to use it in U-Boot.
Any thoughts on this?
On Mon, Jan 13, 2020 at 12:52:48PM +0200, Andy Shevchenko wrote:
Hi!
I recently stumble over FAT partitioning issue. I have a device with MMC, one partition of which has been exported as disk via USB Mass Storage to the host. When Windows 10 sees that disk it can't handle it till I format it there. So far so good. However, the same partition can't be used under U-Boot due to Windows flow, i.e. it makes a partitioning on top of the actual partition while U-Boot expects that we only have one MBR (or partitioning) on the dist. So, the commands such fatls, fatload do not recognise any file there.
In Linux it's easy to use: mount -o loop,offset=65536 /dev/mmcblk0p9 /mnt (offset can be different)
But I would like to use it in U-Boot.
Any thoughts on this?
I'm not sure. If you didn't have to use offset+loop to re-expose things as a new disk and show the partition I'd say we need to support that case. But here, hmmm. We don't have "loopback". So, I'm not sure. Can't you just tell Windows to use the whole device as FAT and so mmcblk0p9 would directly be FAT and work as expected in both Linux and U-Boot?
On 1/13/20 5:34 PM, Tom Rini wrote:
On Mon, Jan 13, 2020 at 12:52:48PM +0200, Andy Shevchenko wrote:
Hi!
I recently stumble over FAT partitioning issue. I have a device with MMC, one partition of which has been exported as disk via USB Mass Storage to the host. When Windows 10 sees that disk it can't handle it till I format it there.
This I could reproduce.
So far so good. However, the same partition
can't be used under U-Boot due to Windows flow, i.e. it makes a partitioning on top of the actual partition while U-Boot expects that we only have one MBR (or partitioning) on the dist. So, the commands such fatls, fatload do not recognise any file there.
But for me the U-Boot's load command loaded a file created on Windows without problems. The size of my FAT partition was 50 MiB.
Please, provide a more detailed instruction how to create an image that is recognized by Windows but not by U-Boot.
Please, provide a gzipped test image for download.
Best regards
Heinrich
In Linux it's easy to use: mount -o loop,offset=65536 /dev/mmcblk0p9 /mnt (offset can be different)
But I would like to use it in U-Boot.
Any thoughts on this?
I'm not sure. If you didn't have to use offset+loop to re-expose things as a new disk and show the partition I'd say we need to support that case. But here, hmmm. We don't have "loopback". So, I'm not sure. Can't you just tell Windows to use the whole device as FAT and so mmcblk0p9 would directly be FAT and work as expected in both Linux and U-Boot?
On Mon, Jan 13, 2020 at 7:55 PM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/13/20 5:34 PM, Tom Rini wrote:
On Mon, Jan 13, 2020 at 12:52:48PM +0200, Andy Shevchenko wrote:
Hi!
I recently stumble over FAT partitioning issue. I have a device with MMC, one partition of which has been exported as disk via USB Mass Storage to the host. When Windows 10 sees that disk it can't handle it till I format it there.
This I could reproduce.
So far so good. However, the same partition
can't be used under U-Boot due to Windows flow, i.e. it makes a partitioning on top of the actual partition while U-Boot expects that we only have one MBR (or partitioning) on the dist. So, the commands such fatls, fatload do not recognise any file there.
But for me the U-Boot's load command loaded a file created on Windows without problems. The size of my FAT partition was 50 MiB.
Please, provide a more detailed instruction how to create an image that is recognized by Windows but not by U-Boot.
Windows GUI (Disk manager by clicking right button on the Windows icon on the bar) provides this. https://www.windowscentral.com/how-format-usb-flash-drive-windows-10
Basically I followed above (Quick or not format makes no difference -- it has the "new simple volume" available only which creates, as mentioned, volume with all structures).
Please, provide a gzipped test image for download.
What image? Disk one? Let me prepare it later when I come home.
Best regards
Heinrich
In Linux it's easy to use: mount -o loop,offset=65536 /dev/mmcblk0p9 /mnt (offset can be different)
But I would like to use it in U-Boot.
Any thoughts on this?
I'm not sure. If you didn't have to use offset+loop to re-expose things as a new disk and show the partition I'd say we need to support that case. But here, hmmm. We don't have "loopback". So, I'm not sure. Can't you just tell Windows to use the whole device as FAT and so mmcblk0p9 would directly be FAT and work as expected in both Linux and U-Boot?
On Mon, Jan 13, 2020 at 9:15 PM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Mon, Jan 13, 2020 at 7:55 PM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/13/20 5:34 PM, Tom Rini wrote:
On Mon, Jan 13, 2020 at 12:52:48PM +0200, Andy Shevchenko wrote:
...
But for me the U-Boot's load command loaded a file created on Windows without problems. The size of my FAT partition was 50 MiB.
I think you would like to know the size of mine. It's about 760MB.
P.S. I'll prepare a compressed / sparse image later as promised.
On Mon, Jan 13, 2020 at 9:22 PM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Mon, Jan 13, 2020 at 9:15 PM Andy Shevchenko andy.shevchenko@gmail.com wrote:
P.S. I'll prepare a compressed / sparse image later as promised.
https://paste.cathedral-networks.org/OverdoseSegment
I dunno how long it will be available. I created it using % dd if=/dev/mmcblk0p9 of=mmc-fat-part conv=sparse % gzip mmc-fat-part
On 1/13/20 9:58 PM, Andy Shevchenko wrote:
On Mon, Jan 13, 2020 at 9:22 PM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Mon, Jan 13, 2020 at 9:15 PM Andy Shevchenko andy.shevchenko@gmail.com wrote:
P.S. I'll prepare a compressed / sparse image later as promised.
https://paste.cathedral-networks.org/OverdoseSegment
I dunno how long it will be available. I created it using % dd if=/dev/mmcblk0p9 of=mmc-fat-part conv=sparse % gzip mmc-fat-part
Doesn't work for me:
"File Quota Exceeded"
I guess Github wouldn't give you any trouble.
Best regards
Heinrich
On Mon, Jan 13, 2020 at 11:05 PM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/13/20 9:58 PM, Andy Shevchenko wrote:
On Mon, Jan 13, 2020 at 9:22 PM Andy Shevchenko andy.shevchenko@gmail.com wrote: https://paste.cathedral-networks.org/OverdoseSegment
I dunno how long it will be available. I created it using % dd if=/dev/mmcblk0p9 of=mmc-fat-part conv=sparse % gzip mmc-fat-part
Doesn't work for me:
"File Quota Exceeded"
I guess Github wouldn't give you any trouble.
https://gist.github.com/andy-shev/469aef8dfcd8f5605cb8992cf5958769
On 1/13/20 10:52 PM, Andy Shevchenko wrote:
On Mon, Jan 13, 2020 at 11:05 PM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/13/20 9:58 PM, Andy Shevchenko wrote:
On Mon, Jan 13, 2020 at 9:22 PM Andy Shevchenko andy.shevchenko@gmail.com wrote: https://paste.cathedral-networks.org/OverdoseSegment
I dunno how long it will be available. I created it using % dd if=/dev/mmcblk0p9 of=mmc-fat-part conv=sparse % gzip mmc-fat-part
Doesn't work for me:
"File Quota Exceeded"
I guess Github wouldn't give you any trouble.
https://gist.github.com/andy-shev/469aef8dfcd8f5605cb8992cf5958769
This image loads fine on current U-Boot, see below.
So if you have trouble loading the same image on your machine from MMC the problem might be with MMC and not with FAT.
Best regards
Heinrich
make qemu_arm64_defconfig export CROSS_COMPILE=aarch64-linux-gnu- make
qemu-system-aarch64 -machine virt -m 1G -smp cores=2 \ -bios u-boot.bin -cpu cortex-a53 -nographic -gdb tcp::1234 \ -netdev user,id=eth0,tftp=tftp -device e1000,netdev=eth0 \ -drive if=none,file=foo.img,format=raw,id=mydisk \ -device virtio-rng-pci \ -device ich9-ahci,id=ahci -device ide-drive,drive=mydisk,bus=ahci.0
U-Boot 2020.01-00268-ge936504885-dirty (Jan 12 2020 - 20:27:36 +0100)
DRAM: 1 GiB Flash: 128 MiB *** Warning - bad CRC, using default environment
In: pl011@9000000 Out: pl011@9000000 Err: pl011@9000000 Net: No ethernet found. Hit any key to stop autoboot: 0 => scsi scan scanning bus for devices... Target spinup took 0 ms. SATA link 1 timeout. SATA link 2 timeout. SATA link 3 timeout. SATA link 4 timeout. SATA link 5 timeout. AHCI 0001.0000 32 slots 6 ports 1.5 Gbps 0x3f impl SATA mode flags: 64bit ncq only Device 0: (0:0) Vendor: ATA Prod.: QEMU HARDDISK Rev: 2.5+ Type: Hard Disk Capacity: 768.0 MB = 0.7 GB (1572864 x 512) => ls scsi 0:1 System Volume Information/ 23 New Text Document.txt $RECYCLE.BIN/
1 file(s), 2 dir(s)
=> load scsi 0:1 0x40200000 'New Text Document.txt' 23 bytes read in 11 ms (2 KiB/s) => md.b 40200000 20 40200000: 46 69 6c 65 20 6f 6e 20 46 41 54 33 32 20 70 61 File on FAT32 pa 40200010: 72 74 69 74 69 6f 6e 00 00 00 00 00 00 00 00 00 rtition......... =>
On Tue, Jan 14, 2020 at 1:14 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/13/20 10:52 PM, Andy Shevchenko wrote:
On Mon, Jan 13, 2020 at 11:05 PM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/13/20 9:58 PM, Andy Shevchenko wrote:
On Mon, Jan 13, 2020 at 9:22 PM Andy Shevchenko andy.shevchenko@gmail.com wrote: https://paste.cathedral-networks.org/OverdoseSegment
I dunno how long it will be available. I created it using % dd if=/dev/mmcblk0p9 of=mmc-fat-part conv=sparse % gzip mmc-fat-part
Doesn't work for me:
"File Quota Exceeded"
I guess Github wouldn't give you any trouble.
https://gist.github.com/andy-shev/469aef8dfcd8f5605cb8992cf5958769
This image loads fine on current U-Boot, see below.
Of course it does *in the test case you have done*. I'm describing different one. The provided image must be a *partition* on the real disk.
So, before use it the preparatory steps must be made.
Something like
% dd if=/dev/zero of=image-file bs=1M count=1000 % fdisk image-file ...create a partition table, where one partition has a (similar) size of the image I provided % mount -o loop,offset=... image-file /mnt # use *partition* as a disk! % dd --sparse if=mmc-fat-part of=/mnt % umount /mnt
And use image-file instead.
On Tue, Jan 14, 2020 at 10:21 AM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Tue, Jan 14, 2020 at 1:14 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/13/20 10:52 PM, Andy Shevchenko wrote:
...
This image loads fine on current U-Boot, see below.
Of course it does *in the test case you have done*. I'm describing different one. The provided image must be a *partition* on the real disk.
So, before use it the preparatory steps must be made.
Something like
% dd if=/dev/zero of=image-file bs=1M count=1000 % fdisk image-file ...create a partition table, where one partition has a (similar) size of the image I provided % mount -o loop,offset=... image-file /mnt # use *partition* as a disk! % dd --sparse if=mmc-fat-part of=/mnt % umount /mnt
And use image-file instead.
Should I prepare it for you or you can do it yourself?
-- With Best Regards, Andy Shevchenko
On Tue, Jan 14, 2020 at 10:23 AM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Tue, Jan 14, 2020 at 10:21 AM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Tue, Jan 14, 2020 at 1:14 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/13/20 10:52 PM, Andy Shevchenko wrote:
...
This image loads fine on current U-Boot, see below.
Of course it does *in the test case you have done*. I'm describing different one. The provided image must be a *partition* on the real disk.
So, before use it the preparatory steps must be made.
Something like
% dd if=/dev/zero of=image-file bs=1M count=1000 % fdisk image-file ...create a partition table, where one partition has a (similar) size of the image I provided % mount -o loop,offset=... image-file /mnt # use *partition* as a disk! % dd --sparse if=mmc-fat-part of=/mnt % umount /mnt
And use image-file instead.
Should I prepare it for you or you can do it yourself?
It's there under name image-file.gz
Commit messages have the commands I performed to get this image cooked.
On Tue, Jan 14, 2020 at 2:43 PM Andy Shevchenko andy.shevchenko@gmail.com wrote:
$ qemu-system-i386 -m 1G -bios u-boot.rom -nographic -drive if=none,file=/tmp/for- andy/image-file,format=raw,id=mydisk -no-reboot -device ich9-ahci,id=ahci -device ide-drive,drive=mydisk,bus=ahci. 0 qemu-system-i386: -device ide-drive,drive=mydisk,bus=ahci.0: warning: 'ide-drive' is deprecated, please use 'ide-h d' or 'ide-cd' instead
U-Boot 2020.01-00012-gb4daeb7f02 (Jan 14 2020 - 15:06:52 +0200)
CPU: QEMU Virtual CPU version 2.5+ DRAM: 1 GiB Video: 1024x768x32 Model: QEMU x86 (I440FX) Net: e1000: 52:54:00:12:34:56
Warning: e1000#0 using MAC address from ROM eth0: e1000#0 Hit any key to stop autoboot: 0 => scsi scan scanning bus for devices... Target spinup took 0 ms. SATA link 1 timeout. SATA link 2 timeout. SATA link 3 timeout. SATA link 4 timeout. SATA link 5 timeout. AHCI 0001.0000 32 slots 6 ports 1.5 Gbps 0x3f impl SATA mode flags: 64bit ncq only Device 0: (0:0) Vendor: ATA Prod.: QEMU HARDDISK Rev: 2.5+ Type: Hard Disk Capacity: 1024.0 MB = 1.0 GB (2097152 x 512) => ls scsi 0:0 ** Unrecognized filesystem type ** => ls scsi 0:1
0 file(s), 0 dir(s)
=> ls scsi 0:2 ** Invalid partition 2 ** =>
On Tue, Jan 14, 2020 at 02:43:43PM +0200, Andy Shevchenko wrote:
On Tue, Jan 14, 2020 at 10:23 AM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Tue, Jan 14, 2020 at 10:21 AM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Tue, Jan 14, 2020 at 1:14 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/13/20 10:52 PM, Andy Shevchenko wrote:
...
This image loads fine on current U-Boot, see below.
Of course it does *in the test case you have done*. I'm describing different one. The provided image must be a *partition* on the real disk.
So, before use it the preparatory steps must be made.
Something like
% dd if=/dev/zero of=image-file bs=1M count=1000 % fdisk image-file ...create a partition table, where one partition has a (similar) size of the image I provided % mount -o loop,offset=... image-file /mnt # use *partition* as a disk! % dd --sparse if=mmc-fat-part of=/mnt % umount /mnt
And use image-file instead.
Should I prepare it for you or you can do it yourself?
It's there under name image-file.gz
===8<=== $ hd image-file 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001b0 00 00 00 00 00 00 00 00 59 02 8c 89 00 00 00 20 |........Y...... | 000001c0 21 00 0c 08 27 62 00 08 00 00 01 00 18 00 00 00 |!...'b..........| 000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * (snip) 00100000 eb 3c 90 6d 6b 64 6f 73 66 73 00 00 02 20 01 00 |.<.mkdosfs... ..| 00100010 02 00 02 00 00 f8 c0 00 10 00 04 00 00 00 00 00 |................| 00100020 00 00 18 00 80 00 29 ea 36 23 57 20 20 20 20 20 |......).6#W | 00100030 20 20 20 20 20 20 46 41 54 31 36 20 20 20 0e 1f | FAT16 ..| (snip) 001001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 001001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.| 00100200 f8 ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00100210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00110000 eb 58 90 4d 53 44 4f 53 35 2e 30 00 02 08 1a 14 |.X.MSDOS5.0.....| 00110010 02 00 00 00 00 f8 00 00 3f 00 ff 00 80 00 00 00 |........?.......| 00110020 00 e8 17 00 f3 05 00 00 00 00 00 00 02 00 00 00 |................| 00110030 01 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00110040 80 00 29 f8 a9 74 d0 4e 4f 20 4e 41 4d 45 20 20 |..)..t.NO NAME | 00110050 20 20 46 41 54 33 32 20 20 20 33 c9 8e d1 bc f4 | FAT32 3.....| (snip) ===>8===
[0x100000-0x100200) looks to be PBR. [0x110000-0x110050) looks to be MBR. But I don't know what is [0x0-0x100000). Obviously U-Boot's fat code cannot handle it.
Thanks, -Takahiro Akashi
Commit messages have the commands I performed to get this image cooked.
-- With Best Regards, Andy Shevchenko
On Wed, Jan 15, 2020 at 09:12:59AM +0900, AKASHI Takahiro wrote:
On Tue, Jan 14, 2020 at 02:43:43PM +0200, Andy Shevchenko wrote:
On Tue, Jan 14, 2020 at 10:23 AM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Tue, Jan 14, 2020 at 10:21 AM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Tue, Jan 14, 2020 at 1:14 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/13/20 10:52 PM, Andy Shevchenko wrote:
...
This image loads fine on current U-Boot, see below.
Of course it does *in the test case you have done*. I'm describing different one. The provided image must be a *partition* on the real disk.
So, before use it the preparatory steps must be made.
Something like
% dd if=/dev/zero of=image-file bs=1M count=1000 % fdisk image-file ...create a partition table, where one partition has a (similar) size of the image I provided % mount -o loop,offset=... image-file /mnt # use *partition* as a disk! % dd --sparse if=mmc-fat-part of=/mnt % umount /mnt
And use image-file instead.
Should I prepare it for you or you can do it yourself?
It's there under name image-file.gz
===8<=== $ hd image-file 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001b0 00 00 00 00 00 00 00 00 59 02 8c 89 00 00 00 20 |........Y...... | 000001c0 21 00 0c 08 27 62 00 08 00 00 01 00 18 00 00 00 |!...'b..........| 000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
(snip) 00100000 eb 3c 90 6d 6b 64 6f 73 66 73 00 00 02 20 01 00 |.<.mkdosfs... ..| 00100010 02 00 02 00 00 f8 c0 00 10 00 04 00 00 00 00 00 |................| 00100020 00 00 18 00 80 00 29 ea 36 23 57 20 20 20 20 20 |......).6#W | 00100030 20 20 20 20 20 20 46 41 54 31 36 20 20 20 0e 1f | FAT16 ..| (snip) 001001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
001001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.| 00100200 f8 ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00100210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00110000 eb 58 90 4d 53 44 4f 53 35 2e 30 00 02 08 1a 14 |.X.MSDOS5.0.....| 00110010 02 00 00 00 00 f8 00 00 3f 00 ff 00 80 00 00 00 |........?.......| 00110020 00 e8 17 00 f3 05 00 00 00 00 00 00 02 00 00 00 |................| 00110030 01 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00110040 80 00 29 f8 a9 74 d0 4e 4f 20 4e 41 4d 45 20 20 |..)..t.NO NAME | 00110050 20 20 46 41 54 33 32 20 20 20 33 c9 8e d1 bc f4 | FAT32 3.....| (snip) ===>8===
[0x100000-0x100200) looks to be PBR. [0x110000-0x110050) looks to be MBR. But I don't know what is [0x0-0x100000).
(Correction) [0x0-0x200) is actually a partition table:
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001b0 00 00 00 00 00 00 00 00 59 02 8c 89 00 00/00/20 |........Y...... | ^^:boot_ind boot_ind: not ACTIVE 000001c0 21 00 0c 08 27 62/00 08 00 00/01 00 18 00/00 00 |!...'b..........| p1's start p1's size start: 0x00000800 sector (= 0x100000) size: 0x00180000 sectors 000001d0 00 00/00 00 00 00/00 00 00 00/00 00 00 00/00 00 |................| * 000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.| ^^^^^:magic number
And [0x100000-0x100200) is a PBR, which then points to a next-level partition:
00100000 eb 3c 90 6d 6b 64 6f 73 66 73 00 00 02 20 01 00 |.<.mkdosfs... ..| (snip) 001001b0 00 00 00 00 00 00 00 00 8c 60 88 d5 00 00/00/02/ |.........`......| 001001c0 03 00 0c fe 3f 60/80 00 00 00/00 e8 17 00/00 00 |....?`..........| start size start: 0x00000080 size: 0x0017e800
Obviously U-Boot's fat code cannot handle it.
So precisely, U-Boot cannot handle nested partition( table)s?
-Takahiro Akashi
Thanks, -Takahiro Akashi
Commit messages have the commands I performed to get this image cooked.
-- With Best Regards, Andy Shevchenko
On Thu, Jan 16, 2020 at 4:01 AM AKASHI Takahiro takahiro.akashi@linaro.org wrote:
On Wed, Jan 15, 2020 at 09:12:59AM +0900, AKASHI Takahiro wrote:
On Tue, Jan 14, 2020 at 02:43:43PM +0200, Andy Shevchenko wrote:
On Tue, Jan 14, 2020 at 10:23 AM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Tue, Jan 14, 2020 at 10:21 AM Andy Shevchenko andy.shevchenko@gmail.com wrote:
On Tue, Jan 14, 2020 at 1:14 AM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/13/20 10:52 PM, Andy Shevchenko wrote:
...
This image loads fine on current U-Boot, see below.
Of course it does *in the test case you have done*. I'm describing different one. The provided image must be a *partition* on the real disk.
So, before use it the preparatory steps must be made.
Something like
% dd if=/dev/zero of=image-file bs=1M count=1000 % fdisk image-file ...create a partition table, where one partition has a (similar) size of the image I provided % mount -o loop,offset=... image-file /mnt # use *partition* as a disk! % dd --sparse if=mmc-fat-part of=/mnt % umount /mnt
And use image-file instead.
Should I prepare it for you or you can do it yourself?
It's there under name image-file.gz
===8<=== $ hd image-file 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001b0 00 00 00 00 00 00 00 00 59 02 8c 89 00 00 00 20 |........Y...... | 000001c0 21 00 0c 08 27 62 00 08 00 00 01 00 18 00 00 00 |!...'b..........| 000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
(snip) 00100000 eb 3c 90 6d 6b 64 6f 73 66 73 00 00 02 20 01 00 |.<.mkdosfs... ..| 00100010 02 00 02 00 00 f8 c0 00 10 00 04 00 00 00 00 00 |................| 00100020 00 00 18 00 80 00 29 ea 36 23 57 20 20 20 20 20 |......).6#W | 00100030 20 20 20 20 20 20 46 41 54 31 36 20 20 20 0e 1f | FAT16 ..| (snip) 001001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
001001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.| 00100200 f8 ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00100210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00110000 eb 58 90 4d 53 44 4f 53 35 2e 30 00 02 08 1a 14 |.X.MSDOS5.0.....| 00110010 02 00 00 00 00 f8 00 00 3f 00 ff 00 80 00 00 00 |........?.......| 00110020 00 e8 17 00 f3 05 00 00 00 00 00 00 02 00 00 00 |................| 00110030 01 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00110040 80 00 29 f8 a9 74 d0 4e 4f 20 4e 41 4d 45 20 20 |..)..t.NO NAME | 00110050 20 20 46 41 54 33 32 20 20 20 33 c9 8e d1 bc f4 | FAT32 3.....| (snip) ===>8===
[0x100000-0x100200) looks to be PBR. [0x110000-0x110050) looks to be MBR. But I don't know what is [0x0-0x100000).
(Correction) [0x0-0x200) is actually a partition table:
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001b0 00 00 00 00 00 00 00 00 59 02 8c 89 00 00/00/20 |........Y...... | ^^:boot_ind boot_ind: not ACTIVE 000001c0 21 00 0c 08 27 62/00 08 00 00/01 00 18 00/00 00 |!...'b..........| p1's start p1's size start: 0x00000800 sector (= 0x100000) size: 0x00180000 sectors 000001d0 00 00/00 00 00 00/00 00 00 00/00 00 00 00/00 00 |................|
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.| ^^^^^:magic number
And [0x100000-0x100200) is a PBR, which then points to a next-level partition:
00100000 eb 3c 90 6d 6b 64 6f 73 66 73 00 00 02 20 01 00 |.<.mkdosfs... ..| (snip) 001001b0 00 00 00 00 00 00 00 00 8c 60 88 d5 00 00/00/02/ |.........`......| 001001c0 03 00 0c fe 3f 60/80 00 00 00/00 e8 17 00/00 00 |....?`..........| start size start: 0x00000080 size: 0x0017e800
Thank you for detailed explanation.
Obviously U-Boot's fat code cannot handle it.
So precisely, U-Boot cannot handle nested partition( table)s?
Seems so. We need to be able to supply the partition number we would like to open, something like cmd <interface> [<dev>[:<partition>[:<nested partition>]]] otherwise it will require some (error prone) heuristics to understand which one user would like to use.
On 1/16/20 11:39 AM, Andy Shevchenko wrote:
Obviously U-Boot's fat code cannot handle it.
So precisely, U-Boot cannot handle nested partition( table)s?
Seems so. We need to be able to supply the partition number we would like to open, something like cmd <interface> [<dev>[:<partition>[:<nested partition>]]] otherwise it will require some (error prone) heuristics to understand which one user would like to use.
I have formatted a USB stick Windows 10.
When I display the partition table with fdisk it shows:
Device Boot Start End Sectors Size Id Type /dev/sdb1 1935758368 3615603091 1679844724 801G 61 SpeedStor /dev/sdb2 0 0 0 0B 65 Novell Netware 386 /dev/sdb4 28049408 28049850 443 221.5K 0 Empty
But there is no partition table at all. The FAT filesystem starts at sector 0. I can mount the disk with
sudo mount /dev/sdb /mnt
In U-Boot I can read the file system as partition 0:
=> ls scsi 0:0 System Volume Information/ 11 test.txt.txt
1 file(s), 1 dir(s)
=> load scsi 0:0 0x40200000 test.txt.txt 11 bytes read in 10 ms (1000 Bytes/s) => md.b 0x40200000 b 40200000: 48 65 6c 6c 6f 20 77 6f 72 6c 64 Hello world =>
The UEFI shell started from U-Boot sees the file system:
Mapping table FS0: Alias(s):F0a0:;BLK0: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/Scsi(0,0)
In the UEFI shell I created a new text file. I could read it in Windows.
Now I formatted the USB in Linux with gdisk to give it a GPT partition table:
Number Start (sector) End (sector) Size Code Name 1 2048 33556479 16.0 GiB 0700 Microsoft basic data 2 33556480 62521310 13.8 GiB 0700 Microsoft basic data
and formatted the partitions with mkfs.vfat as FAT32.
Now Windows sees this USB stick as two partions and can read files created in Linux.
So this is what I learnt:
Windows 10 does not support MBR partition tables on USB sticks. Windows 10 supports GPT partition tables on USB sticks. Windows formats empty USB sticks without any partition table at all. There is no obvious compatibility issue in U-Boot.
Best regards
Heinrich
On Thu, Jan 16, 2020 at 9:20 PM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/16/20 11:39 AM, Andy Shevchenko wrote:
Obviously U-Boot's fat code cannot handle it.
So precisely, U-Boot cannot handle nested partition( table)s?
Seems so. We need to be able to supply the partition number we would like to open, something like cmd <interface> [<dev>[:<partition>[:<nested partition>]]] otherwise it will require some (error prone) heuristics to understand which one user would like to use.
I have formatted a USB stick Windows 10.
That is *not* the case I'm describing.
Maybe I described it wrong. Let me try again.
Prerequisites: - the board with U-Boot and installed Linux OS on eMMC - g_multi module in Linux OS that shares *one of the eMMC partitions* (pay attention here) as a disk to Windows host
Now, when you format that exposed disk (which is actually a partition on eMMC!) in Windows, you will get nested partitioning.
P.S. I can easily reproduce this on real device with latest U-Boot. U-Boot has obvious issue with recognizing such disks.
On Thu, Jan 16, 2020 at 10:31:49PM +0200, Andy Shevchenko wrote:
On Thu, Jan 16, 2020 at 9:20 PM Heinrich Schuchardt xypron.glpk@gmx.de wrote:
On 1/16/20 11:39 AM, Andy Shevchenko wrote:
Obviously U-Boot's fat code cannot handle it.
So precisely, U-Boot cannot handle nested partition( table)s?
Seems so. We need to be able to supply the partition number we would like to open, something like cmd <interface> [<dev>[:<partition>[:<nested partition>]]] otherwise it will require some (error prone) heuristics to understand which one user would like to use.
I have formatted a USB stick Windows 10.
That is *not* the case I'm describing.
Maybe I described it wrong. Let me try again.
Prerequisites:
- the board with U-Boot and installed Linux OS on eMMC
- g_multi module in Linux OS that shares *one of the eMMC partitions*
(pay attention here) as a disk to Windows host
I also misunderstood your assumption above; You are developing a linux-based USB gadget for Windows (10)?
Now, when you format that exposed disk (which is actually a partition on eMMC!) in Windows, you will get nested partitioning.
So why do you want to access *that* partition from U-Boot on the board? I don't think it is a common case.
-Takahiro Akashi
P.S. I can easily reproduce this on real device with latest U-Boot. U-Boot has obvious issue with recognizing such disks.
-- With Best Regards, Andy Shevchenko
On Fri, Jan 17, 2020 at 8:13 AM AKASHI Takahiro takahiro.akashi@linaro.org wrote:
On Thu, Jan 16, 2020 at 10:31:49PM +0200, Andy Shevchenko wrote:
...
Prerequisites:
- the board with U-Boot and installed Linux OS on eMMC
- g_multi module in Linux OS that shares *one of the eMMC partitions*
(pay attention here) as a disk to Windows host
I also misunderstood your assumption above; You are developing a linux-based USB gadget for Windows (10)?
No.
Now, when you format that exposed disk (which is actually a partition on eMMC!) in Windows, you will get nested partitioning.
So why do you want to access *that* partition from U-Boot on the board? I don't think it is a common case.
What I'm trying to do is to copy some bootables to that partition from Windows machine in order to boot them via U-Boot. That's allow me not to disturb stock image on that board.
If U-Boot is not going to support nested partitioning, perhaps I can submit a documentation fix to advertise this explicitly.
On Fri, Jan 17, 2020 at 11:47:03AM +0200, Andy Shevchenko wrote:
On Fri, Jan 17, 2020 at 8:13 AM AKASHI Takahiro takahiro.akashi@linaro.org wrote:
On Thu, Jan 16, 2020 at 10:31:49PM +0200, Andy Shevchenko wrote:
...
Prerequisites:
- the board with U-Boot and installed Linux OS on eMMC
- g_multi module in Linux OS that shares *one of the eMMC partitions*
(pay attention here) as a disk to Windows host
I also misunderstood your assumption above; You are developing a linux-based USB gadget for Windows (10)?
No.
Now, when you format that exposed disk (which is actually a partition on eMMC!) in Windows, you will get nested partitioning.
So why do you want to access *that* partition from U-Boot on the board? I don't think it is a common case.
What I'm trying to do is to copy some bootables to that partition from Windows machine in order to boot them via U-Boot. That's allow me not to disturb stock image on that board.
If U-Boot is not going to support nested partitioning, perhaps I can submit a documentation fix to advertise this explicitly.
I'm confused here, sorry. I thought one of the described examples was how to have Windows use the whole device it was given, without partition table, as FAT and thus it would be shareable. Is that not possible?
On Fri, Jan 17, 2020 at 11:47:03AM +0200, Andy Shevchenko wrote:
On Fri, Jan 17, 2020 at 8:13 AM AKASHI Takahiro takahiro.akashi@linaro.org wrote:
On Thu, Jan 16, 2020 at 10:31:49PM +0200, Andy Shevchenko wrote:
...
Prerequisites:
- the board with U-Boot and installed Linux OS on eMMC
- g_multi module in Linux OS that shares *one of the eMMC partitions*
(pay attention here) as a disk to Windows host
I also misunderstood your assumption above; You are developing a linux-based USB gadget for Windows (10)?
No.
Really? I've got messed up now. - Why do you need to use g_multi to access the disk? - Where is a disk attached to? - Are Linux and Windows running on the same board?
-Takahiro Akashi
Now, when you format that exposed disk (which is actually a partition on eMMC!) in Windows, you will get nested partitioning.
So why do you want to access *that* partition from U-Boot on the board? I don't think it is a common case.
What I'm trying to do is to copy some bootables to that partition from Windows machine in order to boot them via U-Boot. That's allow me not to disturb stock image on that board.
If U-Boot is not going to support nested partitioning, perhaps I can submit a documentation fix to advertise this explicitly.
-- With Best Regards, Andy Shevchenko
On Tue, Jan 21, 2020 at 2:39 AM AKASHI Takahiro takahiro.akashi@linaro.org wrote:
On Fri, Jan 17, 2020 at 11:47:03AM +0200, Andy Shevchenko wrote:
On Fri, Jan 17, 2020 at 8:13 AM AKASHI Takahiro takahiro.akashi@linaro.org wrote:
...
Really? I've got messed up now.
- Why do you need to use g_multi to access the disk?
How else you may provide a way to access it from remote host via USB? Actually it's not me it's a stock image on that board. But it doesn't matter.
- Where is a disk attached to?
To my desktop system. Or you are asked how it's connected to the embedded board? It's eMMC chip soldered on it.
- Are Linux and Windows running on the same board?
No, it's not the case.
[ embedded board + U-Boot + stock OS] ---USB--- [Windows 10 desktop] | v [ eMMC soldered on embedded board, *partition* is shared as a disk]
participants (4)
-
AKASHI Takahiro -
Andy Shevchenko -
Heinrich Schuchardt -
Tom Rini