status / neptune.cloudns.io
Last updated 16 seconds ago. neptune.cloudns.io

Incidents

January 20th, 2021

Cloudlinux Kernel Bug
Identified

Our senior team have been working closely with the development team at Cloudlinux, to investigate a kernel bug in the latest version(s) of the CL software.

The issue relates to the kswapd0 process which is responsible for memory management in the Linux kernel. We had found through an internal analysis, that the kswapd0 process was consuming significantly more CPU than expected, in times where memory was readily available for use.

We have requested escalation to Cloudlinux senior support, then directly to the development team of the kernel, who have now confirmed this to be a present bug based on the following...

There are several issues related to high-order allocations, which resulted in high CPU usage in kswapd:

  1. Intense Transparent Huge Page usage
  2. high order allocations in setsid() system call
  3. high order allocs in KMODLVE due to larger volumes of LVE

The machines we use here are more than capable of handling significantly more than we throw at them, with enterprise grade hardware throughout. The confirmation that this is a bug does mean many servers are currently affected.

We have had the assistance of Igor, CEO of Cloudlinux also provide some assistance in terms of diagnosing these issues.

During the next week or so, small tweaks and modifications may be made (if possible, out of hours), which may cause intermittent but small outages, or performance degradation.

We are working on our own patch internally, however we are restricted with the access we have in terms of the kernel itself (as Cloudlinux is a commercial product).

Some tweaks made so far have been promising, whilst we await a permanent solution.

We appreciate your patience.

Dennis - Brixly