Hi everyone,
On 11/12/2019 09.25, Heiko Schocher wrote:
Hello Harald,
Am 10.12.2019 um 18:18 schrieb Harald Seiler:
Hi Simon,
On Tue, 2019-12-10 at 05:48 -0700, Simon Glass wrote:
Hi,
I sometimes find that gitlab.denx.de is very slow. Just now is is worse than usual - it has taken almost 10 minutes to push a branch and still not finished. I am getting a 503 error on the web site.
Any ideas?
Yes, we noticed that as well but so far have failed to find out the exact reason. Server-side we see horrible load spikes while the CPU is essentially in idle and no notable IO is happening. I personally believe it might have something to do with the virtualization technology but we don't have a certain answer yet.
I added Claudius in CC, he might know more ...
May dummy thought ... could the gitlab runners be the reason? If we push to a u-boot repo on gitlab, we automatically start now a gitlab runner... so if there "a lot of" pushs may a lot of jobs are pending?
No probably not. I don't think the problem is anything under our control.
A bit of background information: Our gitlab is hosted on a virtual server from the provider Strato. Strato uses Virtuozzo/OpenVZ as a virtualization platform, which is sort of container based with a kernel that is shared by all guests. So as a customer we have access to the root file system, but most privileged access to the running kernel is prohibited. (no dmesg, etc.)
With our monitoring tools, we can see the load sometimes spikes go over 100 while we have about 10GiB free memory and not much going on disk or network. The cpu mostly idles but there are a lot of tasks in D state. When the load is high it also seems to be that the cache/buffer of the kernel is at about 200MiB. When the load is low then the cache/buffer is about 5GiB. So there seems to be something off with the cache and/or IO, which both is not under our control. (We don't even have access to block level statistics)
Strato apparently took our gitlab instance at about 23:52 to 01:04 CET offline last night and the load now seems more stable. Hopefully they resolved this issue now. *knocking on wood*
regards, Claudius