[U-Boot] Fix for a build ?

Hi everybody,
We have recently installed an automated build system in our company and occasionally there is a problem when U-Boot is built. One build will work the other will not without any changes to the sources (it is not: 1 work, then other fail... it is more like 3 may work the 4 will not but the 5 will). Our build system guy says that the problem is probably a race condition with a dependency in the U-Boot make.
Here is the error that trigger the failure. *[09:23:58]:* */home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boot-1.3.4/cpu/mpc5xxx/.depend:73: *** multiple target patterns. Stop.* *[09:23:58]:* *make: *** [/home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boot-1.3.4/cpu/mpc5xxx/start.o] Error 2* *[09:23:58]:* */home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boot-1.3.4/cpu/mpc5xxx/.depend:73: *** multiple target patterns. Stop.* *[09:23:58]:* *make: *** Waiting for unfinished jobs....* *[09:23:58]:* *make: *** [/home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boot-1.3.4/cpu/mpc5xxx/libmpc5xxx.a] Error 2* *[09:24:01]:* *Process exited with code 2*
It is not extremely problematic because everything is automated and next build after the failing one will work (when it is this error), but it's only that we get a notification email when a build fail and we are afraid that our developers will take the failure as a "Boy Who Cried Wolf"... if you know what I mean.
Is there a way to fix this ? It could be putting some "sleep" somewhere...
Any suggestion ?
Sylvain
* Also, upgrading U-Boot is not an option... since we don't have the need nor resources to port a new version of U-Boot to this product. So please don't suggest that.

On Monday 03 May 2010 09:51:29 Sylvain Lamontagne wrote:
We have recently installed an automated build system in our company and occasionally there is a problem when U-Boot is built. One build will work the other will not without any changes to the sources (it is not: 1 work, then other fail... it is more like 3 may work the 4 will not but the 5 will). Our build system guy says that the problem is probably a race condition with a dependency in the U-Boot make.
Here is the error that trigger the failure. *[09:23:58]:* */home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boo t-1.3.4/cpu/mpc5xxx/.depend:73: *** multiple target patterns. Stop.*
this should already be fixed in the latest version -mike

And do you know what was the fix so that I could apply it to our version ?
On Mon, May 3, 2010 at 2:52 PM, Mike Frysinger vapier@gentoo.org wrote:
On Monday 03 May 2010 09:51:29 Sylvain Lamontagne wrote:
We have recently installed an automated build system in our company and occasionally there is a problem when U-Boot is built. One build will work the other will not without any changes to the sources (it is not: 1 work, then other fail... it is more like 3 may work the 4 will not but the 5 will). Our build system guy says that the problem is probably a race condition with a dependency in the U-Boot make.
Here is the error that trigger the failure. *[09:23:58]:*
*/home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boo
t-1.3.4/cpu/mpc5xxx/.depend:73: *** multiple target patterns. Stop.*
this should already be fixed in the latest version -mike

Dear Sylvain Lamontagne,
please do not top-post / full quote. Please read http://www.netmeister.org/news/learn2quote.html
In message t2x3e45db431005031314s2a2ef407oa2eb9eb256c7b2df@mail.gmail.com you wrote:
And do you know what was the fix so that I could apply it to our version ?
You can try and run git bisect on the tree to find out yourself. But I guess the fix is buried in a ton of other changes, so you will find a backport is not as trivial a thing as it might seem.
Best regards,
Wolfgang Denk

Dear Sylvain Lamontagne,
In message h2y3e45db431005030651sacba80cz14740e8d79c4d178@mail.gmail.com you wrote:
We have recently installed an automated build system in our company and occasionally there is a problem when U-Boot is built. One build will work the other will not without any changes to the sources (it is not: 1 work, then other fail... it is more like 3 may work the 4 will not but the 5 will). Our build system guy says that the problem is probably a race condition with a dependency in the U-Boot make.
...
*[09:23:58]:* *make: *** [/home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boot-1.3.4/cpu/mpc5xxx/start.o]
-----------------------------------------------------------------------^^^^^^^^^^^^
Ah, U-Boot release 1.3.4.
That's nearly two years old, which means it's ancient.
Yes, there may be race conditions during building, especially on SMP machines.
Is there a way to fix this ? It could be putting some "sleep" somewhere...
Please upgrade to a recent version.
- Also, upgrading U-Boot is not an option... since we don't have the need
nor resources to port a new version of U-Boot to this product. So please don't suggest that.
Heh. So what do you want? A solution, but no changes?
We will not fix for free any bugs in old versions of the code. You have the following options:
- stick with the obsolete code you have and stop complaining - update to the current release and ask again if there should still be problems (which I doubt) - fix the problems yourself - hire somebody else to fix the problems (and most probably the most efficient way to fix the problem will be an update to the current release)
For your next project you can try and learn a lesson from this issue: the amount of effort saved by not pushing your port upstream into mainline is often smaller than the added maintenance cost for an out-of-tree port.
Best regards,
Wolfgang Denk

Is there a way to fix this ? It could be putting some "sleep" somewhere...
Please upgrade to a recent version.
Thank you for suggesting ...
- Also, upgrading U-Boot is not an option... since we don't have the need
nor resources to port a new version of U-Boot to this product. So please don't suggest that.
Heh. So what do you want? A solution, but no changes?
I was asking in case someone remembered and could have been an easy fix. Something like: Yes, Sylvain If I remember there was a race condition with MakefileX in Y specially on an SMP machine. That would have been a perfectly correct answer.
We will not fix for free any bugs in old versions of the code.
That was absolutely not what I was asking for ... I did not want anybody of you to fix this for me. I was asking for pointers and it was a quick question that should have get a quick answer, nothing more.
- stick with the obsolete code you have and stop complaining
Again, I was not complaining. Reading the archive of this mailing list, it seems that you, Wolfgang, tend to often take what people ask as a complaint and I suggest that you should work on this aspect if you want people to contribute by their free will.
- update to the current release and ask again if there should still be
problems (which I doubt)
- fix the problems yourself
- hire somebody else to fix the problems (and most probably the most
efficient way to fix the problem will be an update to the current release)
You may have never been in this situation, but I work for a company that is big enough to not let me change something that work fine only for some failure from time to time related to a race condition on a new automated build server. I did try to update U-Boot in the past few month but I always end-up with problem related to variables that changes names (CFG vs CONFIG) or to bizarre memory config created by my predecessor that make the new version unworkable. I am not an electronician and there is still aspect of embedded development that I don't yet understand. I have an IT degree and I'm currently learning embedded "in the field" since my predecessor was laid off. If giving chance to people is not in your nature, then I'm sorry to have been taking your time.
Make it easy to upgrade and people may then upgrade more often. The Linux kernel 2.4 is still supported in some way for people that have very old hardware that work only with it, so why wouldn't it be legitimate to keep a tested version of U-Boot on a running product only because it's 2 year old ?
Anyway thx for answering even if the answer was a bit rude...
Sylvain

On Monday 03 May 2010 18:02:26 Sylvain Lamontagne wrote:
Make it easy to upgrade and people may then upgrade more often.
as Wolfgang said, this is your fault. integrated board ports get updated automatically by other people. so if you had integrated your board port, the CFG vs CONFIG and related changes wouldnt have been an issue as other people would have fixed it for you. -mike

Dear Sylvain Lamontagne,
In message s2j3e45db431005031502zb199821fndee7e8f220853489@mail.gmail.com you wrote:
Heh. So what do you want? A solution, but no changes?
I was asking in case someone remembered and could have been an easy fix. Something like: Yes, Sylvain If I remember there was a race condition with MakefileX in Y specially on an SMP machine. That would have been a perfectly correct answer.
We have about 650 Makefiles in the current U-Boot source tree; the top level Makefile alone has seen 371 commits since v1.3.4, including major reworks like changes to the directory structure, architecture renames, etc. There have been a number of plain bug fixes to the U-Boot build system, and there have been a number of work-arounds for broken tool chains like the broken GNU make 3.80 or for some broken ARM compilers.
In total there must be way over 1000 commits to Makefiles, plus several hundred to config.mk and other make related files.
I am not in a position to remember each and every of these changes, or their potential impact on building in SMP configurations.
That was absolutely not what I was asking for ... I did not want anybody of you to fix this for me. I was asking for pointers and it was a quick question that should have get a quick answer, nothing more.
I gave you quick answer: use git bisect to find out which commit makes a difference for your build system. Of course this requires that you have some build target that is supported in mainline, and that shows the same problem like your out-of-tree port.
You may have never been in this situation, but I work for a company that is big enough to not let me change something that work fine only for some failure from time to time related to a race condition on a new automated build server.
I have. More than once. And of course we have customers in such situations, too.
I did try to update U-Boot in the past few month but I always end-up with problem related to variables that changes names (CFG vs CONFIG) or to bizarre memory config created by my predecessor that make the new version unworkable. I am not an electronician and there is still
As Mike already pointed out this is a well known problem, and it is important that you understand that it's a completely self-made purgatory. U-Boot develops continuously and quickly, so any out-of-tree port is obsolete as soon as you release it, and maintaining it over more than a few months costs way more effort than pushing your changes upstream into the mainline repositories.
You can now waste efforts on trying to back-port fix after fix from mainline to your old code, or you can invest the effort in updating your code base and pushing it upstream. This is your (or your manager's) decision. We don't actually care about this, but it seems only fair to mention that from our experience the one-time effort to push things upstream is smaller than the repeating costs of maintaining an out-of-tree port.
aspect of embedded development that I don't yet understand. I have an IT degree and I'm currently learning embedded "in the field" since my predecessor was laid off. If giving chance to people is not in your nature, then I'm sorry to have been taking your time. Make it easy to upgrade and people may then upgrade more often. The
You seem to fail to understand how a free software project like U-Boot works. U-Boot is very easy to upgrade - we take great care not to break support for any of the supported boards, and we accept even exotic boards and include these into the mainline tree, even if there is most likely no other user ever.
So if you want to be able to upgrade easily just make sure your code is part of the mainline distribution.
By not pushing your code upstream you may save a certain amount of effort, but you also miss the chance of getting a free code review by a number of highly qualified experts that may improve the quality of your code considerably, and you accept that you will have to spend maintenance efforts all yourself.
This is *your* choice. It is nothing we can do for you, or refrain from doing for you.
Linux kernel 2.4 is still supported in some way for people that have very old hardware that work only with it, so why wouldn't it be legitimate to keep a tested version of U-Boot on a running product only because it's 2 year old ?
U-Boot takes really, really lots of efforts to keep even ancient boards working - in mainline, that it.
The responsibility for any out-of-tree port like yours is completely in your own hands.
Anyway thx for answering even if the answer was a bit rude...
I did not intend to be rude, but I have to admit that your attitude is not exactly in line how community projects like this work.
I don't have the time to answer requests like yours in long prosa - the build system is a pretty complex issue and there are tons of factors that have impact. I already mentioned tool chain issues like broken ARM compilers or things like the white-space madness in GNU make 3.80, but there are tons more. Is your source directory mounted over NFS? Depending on the NFS version you may see problems (like https://bugzilla.redhat.com/show_bug.cgi?id=530625 resp. http://bugs.gentoo.org/show_bug.cgi?id=289022). Are you using any sort of caching? Fscache has it's onw collection of bugs, ccache has some known issues, and so on and on. You did not reveal any information about you build environment, so you should not expect too much useful feedback either. Maybe you want to read http://catb.org/esr/faqs/smart-questions.html
Best regards,
Wolfgang Denk

Dear Wolfgang Denk,
We have about 650 Makefiles in the current U-Boot source tree; the top level Makefile alone has seen 371 commits since v1.3.4
[snip]
I am not in a position to remember each and every of these changes, or their potential impact on building in SMP configurations.
I understand that one person cannot remember the awesome lot of work that have been done since 1.3.4 and that many fundamental changes occurred during these two years.
I gave you quick answer: use git bisect to find out which commit makes a difference for your build system. Of course this requires that you have some build target that is supported in mainline, and that shows the same problem like your out-of-tree port.
I'm sorry I did not see the email related to this before sending the other one. This is a perfect suggestion and I'll probably setup a build configuration for the lite5200 board that is on my desk to see if my problem show itself.
[snip]
You seem to fail to understand how a free software project like U-Boot works. U-Boot is very easy to upgrade - we take great care not to break support for any of the supported boards, and we accept even exotic boards and include these into the mainline tree, even if there is most likely no other user ever.
So if you want to be able to upgrade easily just make sure your code is part of the mainline distribution.
Ok, I am now in the seat of someone that could push the idea of putting the next platform we develop into the mainline. But other MBA/Finance/Manager persons that really take decisions will surely ask my department why I would like to give internal informations of our product in a way that any of our competitors could get it easily. How could I give them an answer that would be aligned with the philosophy ?
I understand how free software project works, I have participate in some and even worked to revive an old dead project (http://datavibe.net/~essiene/ale/) into something up-to-date while I was at University. (http://sonia.etsmtl.ca/index.php?id=553) I've also participate in some kernel janitor task in my free time because I care about open source and I'm passionate by technology. I was a quite active member in the french forum of gentoo to help new comers and find solutions to problems. I care about people and I want the people around me to improve as best as they can.
I did not intend to be rude, but I have to admit that your attitude is not exactly in line how community projects like this work.
Then I'm sorry, but for me this project sounds like it is directed by a bunch of elitist who would not accept someone that don't already know how the project work. Criticizing and judging questions asked by people that may never have work on or with something like U-Boot before. If you are afraid of getting to much dumb questions then this means that the documentation and the FAQ from the U-Boot website could be improve in a way that new comers would find easily the informations they need.
I don't have the time to answer requests like yours in long prosa -
Then you can simply skip the question, this mailing list have surely more than hundred persons that can answer if they think they could help. I don't think anybody ask that you answer to all emails, this is surely an extremely huge time consuming task.
Maybe you want to read http://catb.org/esr/faqs/smart-questions.html
And maybe you want to read http://www.reddit.com/r/linux/comments/9y9de/eric_raymonds_famous_how_to_ask...
Have a nice day
Sylvain

Dear Sylvain Lamontagne,
In message n2m3e45db431005040739m7ff0eb77x712f3e39681f44f8@mail.gmail.com you wrote:
Ok, I am now in the seat of someone that could push the idea of putting the next platform we develop into the mainline. But other MBA/Finance/Manager persons that really take decisions will surely ask my department why I would like to give internal informations of our product in a way that any of our competitors could get it easily. How could I give them an answer that would be aligned with the philosophy ?
First, U-Boot is covered by the GPL, so you will have to give the full sources anyway to any of your customers who asks for it - even if it's one of your direct competitors.
Second, do you really think there is any intellectual property in a port of U-Boot that is worth such protection? If so, you better write your own proprietary boot loader from scratch.
Third: according to Ohloh (http://www.ohloh.net/p/u-boot) it would cost you (or your company) some 270+ man-years or $_15,000,000 to develop something like U-Boot from scratch. You got all this code completely for free, so it seems only just and equitable that you return the little you added to the community, too.
I did not intend to be rude, but I have to admit that your attitude is not exactly in line how community projects like this work.
Then I'm sorry, but for me this project sounds like it is directed by a bunch of elitist who would not accept someone that don't already know how the project work. Criticizing and judging questions asked by people that may never have work on or with something like U-Boot before. If you are afraid of getting to much dumb questions then this means that the documentation and the FAQ from the U-Boot website could be improve in a way that new comers would find easily the informations they need.
All the documentation is in a wiki - please feel free to add any parts you consider missing.
Wolfgang Denk
participants (3)
-
Mike Frysinger
-
Sylvain Lamontagne
-
Wolfgang Denk