[U-Boot] incremental environment updating
currently the env code will erase the entire env storage before writing back out the current env, even if the env storage has enough empty space to store the current env. for example, if CONFIG_ENV_SIZE is declared as 0x2000 but the current env only takes up ~0x300 bytes, the whole 0x2000 is erased and then the ~0x300 gets written out. seems like we can get a pretty good return for fairly low effort if we appended env updates rather than erasing/writing every time ? it'd certainly be faster. while systems with a dedicated sector this isnt so bad, but for people who have to embed the env in the middle of a large sector, this would be much faster most of the time.
has there been previous discussion along these lines that i havent seen ? -mike
Dear Mike,
In message 200904130609.08060.vapier@gentoo.org you wrote:
currently the env code will erase the entire env storage before writing back out the current env, even if the env storage has enough empty space to store the current env. for example, if CONFIG_ENV_SIZE is declared as 0x2000 but the current env only takes up ~0x300 bytes, the whole 0x2000 is erased and then the ~0x300 gets written out. seems like we can get a pretty good return for fairly low effort if we appended env updates rather than erasing/writing every time ? it'd certainly be faster. while systems with a dedicated sector this isnt so bad, but for people who have to embed the env in the middle of a large sector, this would be much faster most of the time.
has there been previous discussion along these lines that i havent seen ?
This hasn't been discussed before. Interesting idea. However, I fail to see how this could be implemented without changing the environment format?
Best regards,
Wolfgang Denk
On Monday 13 April 2009 06:15:24 Wolfgang Denk wrote:
In message Mike wrote:
currently the env code will erase the entire env storage before writing back out the current env, even if the env storage has enough empty space to store the current env. for example, if CONFIG_ENV_SIZE is declared as 0x2000 but the current env only takes up ~0x300 bytes, the whole 0x2000 is erased and then the ~0x300 gets written out. seems like we can get a pretty good return for fairly low effort if we appended env updates rather than erasing/writing every time ? it'd certainly be faster. while systems with a dedicated sector this isnt so bad, but for people who have to embed the env in the middle of a large sector, this would be much faster most of the time.
has there been previous discussion along these lines that i havent seen ?
This hasn't been discussed before. Interesting idea. However, I fail to see how this could be implemented without changing the environment format?
that depends on how you want the compatibility to go. being able to read old environments by newer u-boots is reasonable, but i dont think having old u- boots read newer environments makes realistic sense ?
in terms of actual changes, i had a couple of ideas ... the current env format is: <crc><env><NUL>[undefined]. so if we logically extend the format where [undefined] is <crc><env><NUL>[...], then all existing env storage would be automatically imported. considering most env storage out there uses a bit value of "0" to mean programmed and "1' to mean unprogrammed, it should be pretty easy to quickly detect where the appended envs stop. -mike
Mike Frysinger wrote:
On Monday 13 April 2009 06:15:24 Wolfgang Denk wrote:
In message Mike wrote:
currently the env code will erase the entire env storage before writing back out the current env, even if the env storage has enough empty space to store the current env. for example, if CONFIG_ENV_SIZE is declared as 0x2000 but the current env only takes up ~0x300 bytes, the whole 0x2000 is erased and then the ~0x300 gets written out. seems like we can get a pretty good return for fairly low effort if we appended env updates rather than erasing/writing every time ? it'd certainly be faster. while systems with a dedicated sector this isnt so bad, but for people who have to embed the env in the middle of a large sector, this would be much faster most of the time.
has there been previous discussion along these lines that i havent seen ?
This hasn't been discussed before. Interesting idea. However, I fail to see how this could be implemented without changing the environment format?
that depends on how you want the compatibility to go. being able to read old environments by newer u-boots is reasonable, but i dont think having old u- boots read newer environments makes realistic sense ?
in terms of actual changes, i had a couple of ideas ... the current env format is: <crc><env><NUL>[undefined]. so if we logically extend the format where [undefined] is <crc><env><NUL>[...], then all existing env storage would be automatically imported. considering most env storage out there uses a bit value of "0" to mean programmed and "1' to mean unprogrammed, it should be pretty easy to quickly detect where the appended envs stop. -mike
Hi Mike,
Another concept is to append a complete new env on every write. This is less efficient than just a delta, but has a substantial advantage in that it gives the user a way to erase env variables as well as change them. I suspect it would also be simpler to implement.
Since flash can change 1s to 0s, but cannot change them back, you could redefine the env storage to be <next><crc><env><NUL>[undefined] (note the added <next> at the start). By definition, the valid env's <next> == 0xFFFFFFFF. To write a new env, simply write the offset of [undefined] into the <next> location and then write a new env lump. <next><crc><env><NUL> <next><crc><env><NUL>[undefined] ^^^^ offset of -----> ^^^^ 0xFFFFFFFF
To find the valid env, you simply start at the start of the env area and keep adding <next> to the env start address until <next> == 0xFFFFFFFF. Add four and your env address is correct. No other change needed to the env reading code.
Either way, we would need a new command, say "compactenv" (kind of a long command :-(, maybe there is a shorter German word ;-). This would erase the current env sector and rewrite it with one env, resetting the clock. This could also be done automatically if there is no room left in the env storage area (both options would probably be good to implement).
Another note: this would make it safer to append the env to u-boot rather than keeping it in a separate sector, except when the env storage area needs to be cleared. I don't know if that is worth bragging about, however ("Our New Improved[tm] Env storage handling is 99 times safer: it now only risks bricking your board one time in a hundred!").
Best regards, gvb
Dear Jerry Van Baren,
In message 49E32C3C.8090404@ge.com you wrote:
Since flash can change 1s to 0s, but cannot change them back, you could redefine the env storage to be <next><crc><env><NUL>[undefined] (note the added <next> at the start). By definition, the valid env's <next> == 0xFFFFFFFF. To write a new env, simply write the offset of [undefined] into the <next> location and then write a new env lump.
I see problems with this:
- it's incompatible with the current format, i. e. detection of existing old formats needs a special case which is something I always consider ugly.
- it works only on NOR flash, not on any media that can be accessed only in blocks
- handling of redundant environment becomes a lot more complicated
BTW: what was the exact problem we were trying to solve? Saving the time it takes to erase one flash sector?
If we have to pay for this with introducing a new, incomptible environment format, and probably having to handle NOR flash and other storage media differently, I doubt if this is worth the effort.
Best regards,
Wolfgang Denk
Wolfgang Denk wrote:
Dear Jerry Van Baren,
In message 49E32C3C.8090404@ge.com you wrote:
Since flash can change 1s to 0s, but cannot change them back, you could redefine the env storage to be <next><crc><env><NUL>[undefined] (note the added <next> at the start). By definition, the valid env's <next> == 0xFFFFFFFF. To write a new env, simply write the offset of [undefined] into the <next> location and then write a new env lump.
I see problems with this:
- it's incompatible with the current format, i. e. detection of existing old formats needs a special case which is something I always consider ugly.
True, although that is an issue only for upgrades to boards that have an existing env and that you want to use the new env method on. I would expect the special casing would not be excessively ugly and may not be necessary, depending on how important backward compatibility (board upgrading) is.
- it works only on NOR flash, not on any media that can be accessed only in blocks
Not true: the offset merely needs to be aligned to the next block in that case: <next><crc><env><NUL>FFFFFFFFFFFF <next><crc><env><NUL>[undefined] ^^^^ offset of ------------------> ^^^^ 0xFFFFFFFF
All flash that I can recall seeing allows at least a limited number of rewrites to a block (the limited number getting smaller on successive generations), so two writes to the env start block (once to write the new env, a second time to rewrite the 0xFFFFFFFF to point to the superceeding env). If the flash didn't support two writes, that would be a show-stopper. Due to how people use flash, I don't think the number will ever go to one write (IIRC, 4 is typical of current generation NAND).
- handling of redundant environment becomes a lot more complicated
I don't think so: it would be handled the same way as the primary env in a separate sector as is done currently. If you step through all of the primary env lumps and the last one has an invalid CRC, you fall back to the backup env and repeat the search.
BTW: what was the exact problem we were trying to solve? Saving the time it takes to erase one flash sector?
If we have to pay for this with introducing a new, incompatible environment format, and probably having to handle NOR flash and other storage media differently, I doubt if this is worth the effort.
That could very well be true.
Best regards,
Wolfgang Denk
Ditto, gvb
Dear Jerry Van Baren,
In message 49E335BB.4050609@ge.com you wrote:
- it works only on NOR flash, not on any media that can be accessed only in blocks
Not true: the offset merely needs to be aligned to the next block in that case: <next><crc><env><NUL>FFFFFFFFFFFF <next><crc><env><NUL>[undefined] ^^^^ offset of ------------------> ^^^^ 0xFFFFFFFF
All flash that I can recall seeing allows at least a limited number of rewrites to a block (the limited number getting smaller on successive generations), so two writes to the env start block (once to write the new env, a second time to rewrite the 0xFFFFFFFF to point to the superceeding env). If the flash didn't support two writes, that would be a show-stopper. Due to how people use flash, I don't think the number will ever go to one write (IIRC, 4 is typical of current generation NAND).
So what do we save, then? Instead of a single write we now need actually two...
BTW: what was the exact problem we were trying to solve? Saving the time it takes to erase one flash sector?
If we have to pay for this with introducing a new, incompatible environment format, and probably having to handle NOR flash and other storage media differently, I doubt if this is worth the effort.
That could very well be true.
;-)
Best regards,
Wolfgang Denk
Wolfgang Denk wrote:
Dear Jerry Van Baren,
In message 49E335BB.4050609@ge.com you wrote:
- it works only on NOR flash, not on any media that can be accessed only in blocks
Not true: the offset merely needs to be aligned to the next block in that case: <next><crc><env><NUL>FFFFFFFFFFFF <next><crc><env><NUL>[undefined] ^^^^ offset of ------------------> ^^^^ 0xFFFFFFFF
All flash that I can recall seeing allows at least a limited number of rewrites to a block (the limited number getting smaller on successive generations), so two writes to the env start block (once to write the new env, a second time to rewrite the 0xFFFFFFFF to point to the superceeding env). If the flash didn't support two writes, that would be a show-stopper. Due to how people use flash, I don't think the number will ever go to one write (IIRC, 4 is typical of current generation NAND).
So what do we save, then? Instead of a single write we now need actually two...
Well, I have a (NOR) flash with a 128Kbyte sector size. If my env is ~2K, I can do ~64 updates before I would need to erase the sector and start over.
From a practical point of view, that probably isn't a big deal. From a "bugs me" POV, it bugs me that I'm "wasting" 126K of that sector. :-P
IIRC, redundant env lumps take two sectors so the redundant env isn't erased (need to check the code). Depending on your acceptable risk tolerance, the "redundant" env lump would simply be lump(n-1).
[snip]
Best regards, gvb
On Monday 13 April 2009 08:12:44 Jerry Van Baren wrote:
Mike Frysinger wrote:
On Monday 13 April 2009 06:15:24 Wolfgang Denk wrote:
In message Mike wrote:
currently the env code will erase the entire env storage before writing back out the current env, even if the env storage has enough empty space to store the current env. for example, if CONFIG_ENV_SIZE is declared as 0x2000 but the current env only takes up ~0x300 bytes, the whole 0x2000 is erased and then the ~0x300 gets written out. seems like we can get a pretty good return for fairly low effort if we appended env updates rather than erasing/writing every time ? it'd certainly be faster. while systems with a dedicated sector this isnt so bad, but for people who have to embed the env in the middle of a large sector, this would be much faster most of the time.
has there been previous discussion along these lines that i havent seen ?
This hasn't been discussed before. Interesting idea. However, I fail to see how this could be implemented without changing the environment format?
that depends on how you want the compatibility to go. being able to read old environments by newer u-boots is reasonable, but i dont think having old u- boots read newer environments makes realistic sense ?
in terms of actual changes, i had a couple of ideas ... the current env format is: <crc><env><NUL>[undefined]. so if we logically extend the format where [undefined] is <crc><env><NUL>[...], then all existing env storage would be automatically imported. considering most env storage out there uses a bit value of "0" to mean programmed and "1' to mean unprogrammed, it should be pretty easy to quickly detect where the appended envs stop.
Another concept is to append a complete new env on every write. This is less efficient than just a delta, but has a substantial advantage in that it gives the user a way to erase env variables as well as change them. I suspect it would also be simpler to implement.
this is actually what i was talking about. doing a delta would require the env code to compare the old env and "whiteout" variables that were set but are now unset, and it would take longer to boot up as the env code would have to rebuild using the deltas.
Since flash can change 1s to 0s, but cannot change them back, you could redefine the env storage to be <next><crc><env><NUL>[undefined] (note the added <next> at the start). By definition, the valid env's <next> == 0xFFFFFFFF. To write a new env, simply write the offset of [undefined] into the <next> location and then write a new env lump. <next><crc><env><NUL> <next><crc><env><NUL>[undefined] ^^^^ offset of -----> ^^^^ 0xFFFFFFFF
using a crc of 0x00000000 accomplishes the same thing and retains env format
Either way, we would need a new command, say "compactenv" (kind of a long command :-(, maybe there is a shorter German word ;-). This would erase the current env sector and rewrite it with one env, resetting the clock. This could also be done automatically if there is no room left in the env storage area (both options would probably be good to implement).
i didnt intend for any external behavior change. the env_*.c would take care of the appending and/or reset step. every thing else would continue to operate the same way. along those lines, i dont really see the need for "compactenv" beyond a helper command when debugging implementation. there would be no need to manually run this step. -mike
Dear Mike,
In message 200904130834.35359.vapier@gentoo.org you wrote:
Since flash can change 1s to 0s, but cannot change them back, you could redefine the env storage to be <next><crc><env><NUL>[undefined] (note the added <next> at the start). By definition, the valid env's <next> =3D=3D 0xFFFFFFFF. To write a new env, simply write the offset of [undefined] into the <next> location and then write a new env lump. <next><crc><env><NUL> <next><crc><env><NUL>[undefined] ^^^^ offset of -----> ^^^^ 0xFFFFFFFF
using a crc of 0x00000000 accomplishes the same thing and retains env format
No, this is not the same. With the offset, you can jump directly to the next block; with just a "valid" flag you have to scan invalid copy sequentially to find the end - especially on EEPROM based storage this is painfully slow (see http://www.denx.de/wiki/DULG/AN2004_11_BootTimeOptimization).
Best regards,
Wolfgang Denk
Mike Frysinger wrote:
On Monday 13 April 2009 08:12:44 Jerry Van Baren wrote:
Mike Frysinger wrote:
On Monday 13 April 2009 06:15:24 Wolfgang Denk wrote:
In message Mike wrote:
currently the env code will erase the entire env storage before writing back out the current env, even if the env storage has enough empty space to store the current env. for example, if CONFIG_ENV_SIZE is declared as 0x2000 but the current env only takes up ~0x300 bytes, the whole 0x2000 is erased and then the ~0x300 gets written out. seems like we can get a pretty good return for fairly low effort if we appended env updates rather than erasing/writing every time ? it'd certainly be faster. while systems with a dedicated sector this isnt so bad, but for people who have to embed the env in the middle of a large sector, this would be much faster most of the time.
Hmmm, rereading this, if the issue is simply the time to erase a sector, I don't think this is that big of a deal. It takes ~2 seconds? to erase a sector and env variables are rarely changed in my experience. Does your mileage vary?
has there been previous discussion along these lines that i havent seen ?
This hasn't been discussed before. Interesting idea. However, I fail to see how this could be implemented without changing the environment format?
that depends on how you want the compatibility to go. being able to read old environments by newer u-boots is reasonable, but i dont think having old u- boots read newer environments makes realistic sense ?
in terms of actual changes, i had a couple of ideas ... the current env format is: <crc><env><NUL>[undefined]. so if we logically extend the format where [undefined] is <crc><env><NUL>[...], then all existing env storage would be automatically imported. considering most env storage out there uses a bit value of "0" to mean programmed and "1' to mean unprogrammed, it should be pretty easy to quickly detect where the appended envs stop.
Another concept is to append a complete new env on every write. This is less efficient than just a delta, but has a substantial advantage in that it gives the user a way to erase env variables as well as change them. I suspect it would also be simpler to implement.
this is actually what i was talking about. doing a delta would require the env code to compare the old env and "whiteout" variables that were set but are now unset, and it would take longer to boot up as the env code would have to rebuild using the deltas.
I'm confused, are we actually proposing the same thing? When you say "delta" and "whiteout" I picture additional definitions appended as a delta env lump to the original env lump, not a whole new env lump.
My counterproposal eliminates the delta determination software and totally eliminates having to invent a new way to "whiteout" to indicate removed variables (IIRC, it is valid to set a variable with no value).
Further, once you find "the" env lump, the env processing is 100% the same so the implementation of my counterproposal simply wraps that existing env handling code with a "find the latest lump" loop.
Since flash can change 1s to 0s, but cannot change them back, you could redefine the env storage to be <next><crc><env><NUL>[undefined] (note the added <next> at the start). By definition, the valid env's <next> == 0xFFFFFFFF. To write a new env, simply write the offset of [undefined] into the <next> location and then write a new env lump. <next><crc><env><NUL> <next><crc><env><NUL>[undefined] ^^^^ offset of -----> ^^^^ 0xFFFFFFFF
using a crc of 0x00000000 accomplishes the same thing and retains env format
Three problems: 1) 0x00000000 can be a valid CRC. You either have a risk of failure (granted, very small) or you have to special case that instance. Note that, if you don't handle the 0x00000000 case properly, you will probably brick the board.
2) Handling a redundant env may be more difficult. OK, that probably isn't too bad, assuming you can handle the 0x00000000 CRC value properly since a bad env would have a non-zero invalid CRC.
3) My <next> offset is much faster since you don't have to step through the old env to find where the next one starts. It also handles Wolfgang's point with NAND flashes needing to be aligned to blocks. (What do you do with a non-zero bad CRC? Fall back to the redundant env?)
Either way, we would need a new command, say "compactenv" (kind of a long command :-(, maybe there is a shorter German word ;-). This would erase the current env sector and rewrite it with one env, resetting the clock. This could also be done automatically if there is no room left in the env storage area (both options would probably be good to implement).
i didnt intend for any external behavior change. the env_*.c would take care of the appending and/or reset step. every thing else would continue to operate the same way. along those lines, i dont really see the need for "compactenv" beyond a helper command when debugging implementation. there would be no need to manually run this step. -mike
OK.
gvb
On Monday 13 April 2009 09:11:59 Jerry Van Baren wrote:
Mike Frysinger wrote:
On Monday 13 April 2009 08:12:44 Jerry Van Baren wrote:
Mike Frysinger wrote:
On Monday 13 April 2009 06:15:24 Wolfgang Denk wrote:
In message Mike wrote:
currently the env code will erase the entire env storage before writing back out the current env, even if the env storage has enough empty space to store the current env. for example, if CONFIG_ENV_SIZE is declared as 0x2000 but the current env only takes up ~0x300 bytes, the whole 0x2000 is erased and then the ~0x300 gets written out. seems like we can get a pretty good return for fairly low effort if we appended env updates rather than erasing/writing every time ? it'd certainly be faster. while systems with a dedicated sector this isnt so bad, but for people who have to embed the env in the middle of a large sector, this would be much faster most of the time.
Hmmm, rereading this, if the issue is simply the time to erase a sector, I don't think this is that big of a deal. It takes ~2 seconds? to erase a sector and env variables are rarely changed in my experience. Does your mileage vary?
speed, reliability, and wear
has there been previous discussion along these lines that i havent seen ?
This hasn't been discussed before. Interesting idea. However, I fail to see how this could be implemented without changing the environment format?
that depends on how you want the compatibility to go. being able to read old environments by newer u-boots is reasonable, but i dont think having old u- boots read newer environments makes realistic sense ?
in terms of actual changes, i had a couple of ideas ... the current env format is: <crc><env><NUL>[undefined]. so if we logically extend the format where [undefined] is <crc><env><NUL>[...], then all existing env storage would be automatically imported. considering most env storage out there uses a bit value of "0" to mean programmed and "1' to mean unprogrammed, it should be pretty easy to quickly detect where the appended envs stop.
Another concept is to append a complete new env on every write. This is less efficient than just a delta, but has a substantial advantage in that it gives the user a way to erase env variables as well as change them. I suspect it would also be simpler to implement.
this is actually what i was talking about. doing a delta would require the env code to compare the old env and "whiteout" variables that were set but are now unset, and it would take longer to boot up as the env code would have to rebuild using the deltas.
I'm confused, are we actually proposing the same thing? When you say "delta" and "whiteout" I picture additional definitions appended as a delta env lump to the original env lump, not a whole new env lump.
My counterproposal eliminates the delta determination software and totally eliminates having to invent a new way to "whiteout" to indicate removed variables (IIRC, it is valid to set a variable with no value).
Further, once you find "the" env lump, the env processing is 100% the same so the implementation of my counterproposal simply wraps that existing env handling code with a "find the latest lump" loop.
i was saying what deltas would require. i didnt say that is what i wanted to do. i want to append the entire env blob.
Since flash can change 1s to 0s, but cannot change them back, you could redefine the env storage to be <next><crc><env><NUL>[undefined] (note the added <next> at the start). By definition, the valid env's <next> == 0xFFFFFFFF. To write a new env, simply write the offset of [undefined] into the <next> location and then write a new env lump. <next><crc><env><NUL> <next><crc><env><NUL>[undefined] ^^^^ offset of -----> ^^^^ 0xFFFFFFFF
using a crc of 0x00000000 accomplishes the same thing and retains env format
Three problems:
- 0x00000000 can be a valid CRC. You either have a risk of failure
(granted, very small) or you have to special case that instance. Note that, if you don't handle the 0x00000000 case properly, you will probably brick the board.
i know 0x00000000 is a valid CRC. the implied statement is that the next two bytes can tell you with clarity whether this is the environment or skipping it. either way, i see it as a micro optimization that really isnt necessary for the larger idea.
- My <next> offset is much faster since you don't have to step through
the old env to find where the next one starts. It also handles Wolfgang's point with NAND flashes needing to be aligned to blocks. (What do you do with a non-zero bad CRC? Fall back to the redundant env?)
how much faster <next> is depends on the flash in use. i dont think NAND can deal with this idea in general anyways since it has ECC checks in the OOB page. unless the OOB can be erased independent of the data page ? i thought erasing of NAND/OOB pages had to be done together. -mike
participants (3)
-
Jerry Van Baren -
Mike Frysinger -
Wolfgang Denk