[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS
Hi u-booters,
I have a short script in my u-boot environment which chooses which of two ubifs partitions to boot by attempting to read a release file in each one.
Unfortunately, after an unclean shutdown sometimes the ubifsmount fails. (By "unclean shutdown" I mean that the board was power cycled while doing some low bandwidth logging.)
The strange thing is that Linux has no problem mounting the partition as its root filesystem. This is very confusing because it looks like the ubifs implementation in u-boot is just a copy of the one in Linux.
Has anyone else seen this problem?
Regards,
Alex
PS My kernel is linux-3.0.0/armv5tel and the full u-boot trace is below:
U-Boot 2011.06 (Feb 10 2012 - 12:29:06) OpenRD-Base SoC: Kirkwood 88F6281_A1 DRAM: 128 MiB NAND: 512 MiB *** Warning - bad CRC, using default environment In: serial Out: serial Err: serial Net: egiga0 88E6351 Initialized on egiga0 Hit any key to stop autoboot: 0 Creating 1 MTD partitions on "nand0": 0x000001000000-0x000010000000 : "mtd=2" UBI: attaching mtd1 to ubi0 UBI: physical eraseblock size: 131072 bytes (128 KiB) UBI: logical eraseblock size: 126976 bytes UBI: smallest flash I/O unit: 2048 UBI: sub-page size: 512 UBI: VID header offset: 2048 (aligned 2048) UBI: data offset: 4096 UBI: attached mtd1 to ubi0 UBI: MTD device name: "mtd=2" UBI: MTD device size: 240 MiB UBI: number of good PEBs: 1913 UBI: number of bad PEBs: 7 UBI: max. allowed volumes: 128 UBI: wear-leveling threshold: 4096 UBI: number of internal volumes: 1 UBI: number of user volumes: 1 UBI: available PEBs: 0 UBI: total number of reserved PEBs: 1913 UBI: number of PEBs reserved for bad PEB handling: 19 UBI: max/mean erase counter: 7/1 UBIFS: recovery needed Error reading superblock on volume 'ubi:rootfs'! UBI: mtd1 is detached from ubi0
On 7 March 2012 14:42, Alex Zeffertt azeffertt@cambridgesys.com wrote:
Hi u-booters,
I have a short script in my u-boot environment which chooses which of two ubifs partitions to boot by attempting to read a release file in each one.
Unfortunately, after an unclean shutdown sometimes the ubifsmount fails. (By "unclean shutdown" I mean that the board was power cycled while doing some low bandwidth logging.)
The strange thing is that Linux has no problem mounting the partition as its root filesystem. This is very confusing because it looks like the ubifs implementation in u-boot is just a copy of the one in Linux.
Has anyone else seen this problem?
Regards,
Alex
PS My kernel is linux-3.0.0/armv5tel and the full u-boot trace is below:
U-Boot 2011.06 (Feb 10 2012 - 12:29:06) OpenRD-Base SoC: Kirkwood 88F6281_A1 DRAM: 128 MiB NAND: 512 MiB *** Warning - bad CRC, using default environment In: serial Out: serial Err: serial Net: egiga0 88E6351 Initialized on egiga0 Hit any key to stop autoboot: 0 Creating 1 MTD partitions on "nand0": 0x000001000000-0x000010000000 : "mtd=2" UBI: attaching mtd1 to ubi0 UBI: physical eraseblock size: 131072 bytes (128 KiB) UBI: logical eraseblock size: 126976 bytes UBI: smallest flash I/O unit: 2048 UBI: sub-page size: 512 UBI: VID header offset: 2048 (aligned 2048) UBI: data offset: 4096 UBI: attached mtd1 to ubi0 UBI: MTD device name: "mtd=2" UBI: MTD device size: 240 MiB UBI: number of good PEBs: 1913 UBI: number of bad PEBs: 7 UBI: max. allowed volumes: 128 UBI: wear-leveling threshold: 4096 UBI: number of internal volumes: 1 UBI: number of user volumes: 1 UBI: available PEBs: 0 UBI: total number of reserved PEBs: 1913 UBI: number of PEBs reserved for bad PEB handling: 19 UBI: max/mean erase counter: 7/1 UBIFS: recovery needed Error reading superblock on volume 'ubi:rootfs'! UBI: mtd1 is detached from ubi0
I've been comparing the linux and u-boot implementations, and it looks like the following fix is in the kernel but not in u-boot. I don't really understand it, but it looks like a candidate. Might porting this change to u-boot fix the issue?
- Alex
commit 2ef13294d29bcfb306e0d360f1b97f37b647b0c0 Author: Artem Bityutskiy Artem.Bityutskiy@nokia.com Date: Sun Sep 19 18:34:26 2010 +0300
UBIFS: introduce new flags for RO mounts
Commit 2fde99cb55fb9d9b88180512a5e8a5d939d27fec "UBIFS: mark VFS SB RO too" introduced regression. This commit made UBIFS set the 'MS_RDONLY' flag in the VFS superblock when it switches to R/O mode due to an error. This was done to make VFS show the R/O UBIFS flag in /proc/mounts.
However, several places in UBIFS relied on the 'MS_RDONLY' flag and assume this flag can only change when we re-mount. For example, 'ubifs_put_super()'.
This patch introduces new UBIFS flag - 'c->ro_mount' which changes only when we re-mount, and preserves the way UBIFS was originally mounted (R/W or R/O). This allows us to de-initialize UBIFS cleanly in 'ubifs_put_super()'.
This patch also changes all 'ubifs_assert(!c->ro_media)' assertions to 'ubifs_assert(!c->ro_media && !c->ro_mount)', because we never should write anything if the FS was mounter R/O.
All the places where we test for 'MS_RDONLY' flag in the VFS SB were changed and now we test the 'c->ro_mount' flag instead, because it preserves the original UBIFS mount type, unlike the 'MS_RDONLY' flag.
Signed-off-by: Artem Bityutskiy Artem.Bityutskiy@nokia.com
Hi Alex,
On Wednesday 07 March 2012 16:14:28 Alex Zeffertt wrote:
On 7 March 2012 14:42, Alex Zeffertt azeffertt@cambridgesys.com wrote:
Hi u-booters,
I have a short script in my u-boot environment which chooses which of two ubifs partitions to boot by attempting to read a release file in each one.
Unfortunately, after an unclean shutdown sometimes the ubifsmount fails. (By "unclean shutdown" I mean that the board was power cycled while doing some low bandwidth logging.)
The strange thing is that Linux has no problem mounting the partition as its root filesystem. This is very confusing because it looks like the ubifs implementation in u-boot is just a copy of the one in Linux.
Correct. The U-Boot version is mainly a copy of the Linux version. With minor changes. But its based on an older Linux version. Many changes have gone into the UBI Linux code.
Has anyone else seen this problem?
Regards,
Alex
PS My kernel is linux-3.0.0/armv5tel and the full u-boot trace is below:
U-Boot 2011.06 (Feb 10 2012 - 12:29:06) OpenRD-Base SoC: Kirkwood 88F6281_A1 DRAM: 128 MiB NAND: 512 MiB *** Warning - bad CRC, using default environment In: serial Out: serial Err: serial Net: egiga0 88E6351 Initialized on egiga0 Hit any key to stop autoboot: 0 Creating 1 MTD partitions on "nand0": 0x000001000000-0x000010000000 : "mtd=2" UBI: attaching mtd1 to ubi0 UBI: physical eraseblock size: 131072 bytes (128 KiB) UBI: logical eraseblock size: 126976 bytes UBI: smallest flash I/O unit: 2048 UBI: sub-page size: 512 UBI: VID header offset: 2048 (aligned 2048) UBI: data offset: 4096 UBI: attached mtd1 to ubi0 UBI: MTD device name: "mtd=2" UBI: MTD device size: 240 MiB UBI: number of good PEBs: 1913 UBI: number of bad PEBs: 7 UBI: max. allowed volumes: 128 UBI: wear-leveling threshold: 4096 UBI: number of internal volumes: 1 UBI: number of user volumes: 1 UBI: available PEBs: 0 UBI: total number of reserved PEBs: 1913 UBI: number of PEBs reserved for bad PEB handling: 19 UBI: max/mean erase counter: 7/1 UBIFS: recovery needed Error reading superblock on volume 'ubi:rootfs'! UBI: mtd1 is detached from ubi0
I've been comparing the linux and u-boot implementations, and it looks like the following fix is in the kernel but not in u-boot. I don't really understand it, but it looks like a candidate. Might porting this change to u-boot fix the issue?
Hard to tell. Might be worth a try, if its not too complicated. It would be great if you could report the outcome of this. And best to send this patch to the list as well.
Thanks, Stefan
-- DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-0 Fax: (+49)-8142-66989-80 Email: office@denx.de
I've been comparing the linux and u-boot implementations, and it looks like the following fix is in the kernel but not in u-boot. I don't really understand it, but it looks like a candidate. Might porting this change to u-boot fix the issue?
Hard to tell. Might be worth a try, if its not too complicated. It would be great if you could report the outcome of this. And best to send this patch to the list as well.
Thanks, I've ported that one changeset to u-boot. (I first tried updating u-boot/fs/ubifs to the latest kernel code but this was too difficult without any real understanding.)
Unfortunately I've got to wait for a repro before I can tell if it fixes the issue... so I may go quiet for a few weeks.
Regards,
Alex
Hi Alex,
On Thursday 08 March 2012 10:49:33 Alex Zeffertt wrote:
I've been comparing the linux and u-boot implementations, and it looks like the following fix is in the kernel but not in u-boot. I don't really understand it, but it looks like a candidate. Might porting this change to u-boot fix the issue?
Hard to tell. Might be worth a try, if its not too complicated. It would be great if you could report the outcome of this. And best to send this patch to the list as well.
Thanks, I've ported that one changeset to u-boot. (I first tried updating u-boot/fs/ubifs to the latest kernel code but this was too difficult without any real understanding.)
Unfortunately I've got to wait for a repro before I can tell if it fixes the issue... so I may go quiet for a few weeks.
Understood. Thanks for the status update so far.
Thanks, Stefan
-- DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-0 Fax: (+49)-8142-66989-80 Email: office@denx.de
On 7 March 2012 14:42, Alex Zeffertt azeffertt@cambridgesys.com wrote:
Hi u-booters,
I have a short script in my u-boot environment which chooses which of two ubifs partitions to boot by attempting to read a release file in each one.
Unfortunately, after an unclean shutdown sometimes the ubifsmount fails. (By "unclean shutdown" I mean that the board was power cycled while doing some low bandwidth logging.)
The strange thing is that Linux has no problem mounting the partition as its root filesystem. This is very confusing because it looks like the ubifs implementation in u-boot is just a copy of the one in Linux.
Has anyone else seen this problem?
I've now managed to repro this problem and add some debug. It looks like u-boot is simply running out of memory whilst trying to mount a filesystem that "needs recovery". (Error -12 is -ENOMEM.) The partition it is mounting is 240MB, but only about 40MB full.
The debug output is below after the "ubifsmount" command:
U-Boot 2011.06-00009-g3b6754e-dirty (Mar 08 2012 - 16:30:13) OpenRD-Base
SoC: Kirkwood 88F6281_A1 DRAM: 128 MiB NAND: 512 MiB In: serial Out: serial Err: serial Net: egiga0 88E6351 Initialized on egiga0 Marvell>> ubi part rootfs 2048 Creating 1 MTD partitions on "nand0": 0x000001000000-0x000010000000 : "mtd=2" UBI: attaching mtd1 to ubi0 UBI: physical eraseblock size: 131072 bytes (128 KiB) UBI: logical eraseblock size: 126976 bytes UBI: smallest flash I/O unit: 2048 UBI: sub-page size: 512 UBI: VID header offset: 2048 (aligned 2048) UBI: data offset: 4096 UBI: attached mtd1 to ubi0 UBI: MTD device name: "mtd=2" UBI: MTD device size: 240 MiB UBI: number of good PEBs: 1913 UBI: number of bad PEBs: 7 UBI: max. allowed volumes: 128 UBI: wear-leveling threshold: 4096 UBI: number of internal volumes: 1 UBI: number of user volumes: 1 UBI: available PEBs: 0 UBI: total number of reserved PEBs: 1913 UBI: number of PEBs reserved for bad PEB handling: 19 UBI: max/mean erase counter: 7/1 Marvell>> ubifsmount rootfs UBIFS: recovery needed UBIFS error (pid 0): replay_bud: insert_node failed: err=-12 UBIFS error (pid 0): replay_buds: replay_bud failed: err=-12 UBIFS error (pid 0): ubifs_replay_journal: replay_buds failed: err=-12 UBIFS error (pid 0): mount_ubifs: ubifs_replay_journal failed: err=-12 UBIFS error (pid 0): ubifs_fill_super: mount_ubifs failed: err=-12 UBIFS error (pid 0): ubifs_get_sb: ubifs_fill_super failed: err=-12 Error reading superblock on volume 'ubi:rootfs'! Marvell>>
Does this help explain the issue?
Regards,
Alex
Dear Alex,
In message CANNiwJ+91o+A5Ef6w9V0g=rvj1yA939+ZzTSnKA4DmR67oFe0w@mail.gmail.com you wrote:
I've now managed to repro this problem and add some debug. It looks like u-boot is simply running out of memory whilst trying to mount a filesystem that "needs recovery". (Error -12 is -ENOMEM.) The partition it is mounting is 240MB, but only about 40MB full.
Can you please try out and test what happens when you increase the size of the malloc arena? "include/configs/openrd.h" includes "include/configs/mv-common.h" which sets CONFIG_SYS_MALLOC_LEN to 1 MiB - try changing it to 4 MiB.
BTW: nice to "seeing" you again :-) Hope all is well for you.
Best regards,
Wolfgang Denk
On 8 March 2012 18:18, Wolfgang Denk wd@denx.de wrote:
In message CANNiwJ+91o+A5Ef6w9V0g=rvj1yA939+ZzTSnKA4DmR67oFe0w@mail.gmail.com you wrote:
I've now managed to repro this problem and add some debug. It looks like u-boot is simply running out of memory whilst trying to mount a filesystem that "needs recovery". (Error -12 is -ENOMEM.) The partition it is mounting is 240MB, but only about 40MB full.
Can you please try out and test what happens when you increase the size of the malloc arena? "include/configs/openrd.h" includes "include/configs/mv-common.h" which sets CONFIG_SYS_MALLOC_LEN to 1 MiB - try changing it to 4 MiB.
Thanks Wolfgang, that worked! Is there any reason why you would not set this variable to the maximum possible. For example, is this memory available to the kernel after it is booted? How do you know how big this might need to be in order to mount a filesystem that "needs recovery"?
Regards,
Alex
Dear Alex,
In message CANNiwJKREXiOBfiA_1v_-QGMoV_A59Ce0ADcVfK1xLMFkReFgw@mail.gmail.com you wrote:
Can you please try out and test what happens when you increase the size of the malloc arena? "include/configs/openrd.h" includes "include/configs/mv-common.h" which sets CONFIG_SYS_MALLOC_LEN to 1 MiB - try changing it to 4 MiB.
Thanks Wolfgang, that worked! Is there any reason why you would not set this variable to the maximum possible. ...
Define "maximum possible". The more memory we reserve here, the less will be available for loading Linux and root filesystem images and such. So it's always a compromize: make it as big as needed, but not (much) bigger.
... For example, is thismemory available to the kernel after it is booted? ...
When the kernel boots, all of U-Boot is gone. The kernel has no idea of how U-Boot used the memory; all such information is gone, too.
... How do you knowhow big this might need to be in order to mount a filesystem that "needs recovery"?
I don't know of ways to determine this in advance. Eventually all we can do is thry what you did: increase the malloc arena when it turns out to be too small.
If you are willing to invest a little more time, it would be interesting to test how much memory is actually needed on your system.
Best regards,
Wolfgang Denk
On 9 March 2012 14:02, Wolfgang Denk wd@denx.de wrote:
In message CANNiwJKREXiOBfiA_1v_-QGMoV_A59Ce0ADcVfK1xLMFkReFgw@mail.gmail.com you wrote:
... How do you know how big this might need to be in order to mount a filesystem that "needs recovery"?
I don't know of ways to determine this in advance. Eventually all we can do is thry what you did: increase the malloc arena when it turns out to be too small.
If you are willing to invest a little more time, it would be interesting to test how much memory is actually needed on your system.
I know that 1MiB was too little and 4MiB was enough. However, I don't know whether a filesystem corruption could occur that caused ubifsmount to require more than 4MiB. For that reason I made the malloc area 64MiB. This is half my total RAM, which leaves the other half for downloading images.
Regards,
Alex
participants (3)
-
Alex Zeffertt -
Stefan Roese -
Wolfgang Denk