CPU-intensive Ruby/Python code runs slower on default-configured Docker

Summary

Docker enables security mechanism for Spectre vulnerability by default. This degrades the performance of almost all CPU-intensive programs, especially, interpreters like Ruby and Python. The execution time becomes about twice at worst.

Problem

This is a simple benchmark program that runs an empty loop 100M times.

i=0
while i < 100_000_000
  i+=1
end

It takes 1.3 sec. on the host, but it does 2.5 sec. on a Docker container.

On the host:

$ ruby -ve 't = Time.now; i=0;while i<100_000_000;i+=1;end; puts "#{ Time.now - t } sec"'
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
1.321703922 sec

On a Docker container:

$ docker run -it --rm ruby:2.7 ruby -ve 't = Time.now; i=0;while i<100_000_000;i+=1;end; puts "#{ Time.now - t } sec"'
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
2.452876383 sec

If you specify an option "--security-opt seccomp=unconfined" for docker run command, it runs as fast as the host.

$ docker run --security-opt seccomp=unconfined -it --rm ruby:2.7 ruby -ve 't = Time.now; i=0;while i<100_000_000;i+=1;end; puts "#{ Time.now - t } sec"'
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
1.333669449 sec

The above example uses Ruby, but I confirmed the problem with Python too. (Two-line code "i=0 / while i<100000000; i+=1" took 7.0 sec. on the host and 11 sec. on Docker.)

Note that this code does not access file system nor network. In fact, it issues no syscall at all in the main loop. Thus, in this case, there is no virtualization overhead of Docker.

You may not reproduce this issue depending upon the kernel configuration as described later.

Why

The recent Linux kernel implements some mitigation options against Spectre vulnerability).

One of them suppresses indirect branch prediction (called STIBP). This makes CPU-intensive code 2x slower, so it is disabled by default. Docker, however, runs a container with the option enabled.

(Koichi Sasada says it may depend the kernel configuration of each distribution. See Spectre Side Channels — The Linux Kernel documentation.)

This option makes almost all programs slower. According to this article, it reduced the performance of Java, Node.js, memcached, PHP, and so on. Interpreters like Ruby and Python are especially affected because they heavily depends upon indirect branches, e.g., switch/case, direct threading, and so on. They run faster on the host because the option is off, and slower on Docker because the option is on.

Using "perf stat", I measured branch miss count: 522,663 on the host, and 199,260,442 on Docker.

With --security-opt seccomp=unconfined (vulnerable against Spectre):

 Performance counter stats for process id '153095':
          1,235.67 msec task-clock                #    0.618 CPUs utilized
                 8      context-switches          #    0.006 K/sec
                 0      cpu-migrations            #    0.000 K/sec
                 2      page-faults               #    0.002 K/sec
     4,284,307,990      cycles                    #    3.467 GHz
    13,903,977,890      instructions              #    3.25  insn per cycle
     1,700,742,230      branches                  # 1376.378 M/sec
           522,663      branch-misses             #    0.03% of all branches
       2.000223507 seconds time elapsed

Without --security-opt seccomp=unconfined (not vulnerable against Spectre):

 Performance counter stats for process id '152556':
          3,300.42 msec task-clock                #    0.550 CPUs utilized
                16      context-switches          #    0.005 K/sec
                 2      cpu-migrations            #    0.001 K/sec
                 2      page-faults               #    0.001 K/sec
    11,912,594,779      cycles                    #    3.609 GHz
    13,906,818,105      instructions              #    1.17  insn per cycle
     1,701,237,677      branches                  #  515.460 M/sec
       199,260,442      branch-misses             #   11.71% of all branches
       6.000985834 seconds time elapsed

This issue is reproduced only when STIBP is "conditional", which means off by default but it can be enabled by seccomp. If STIBP is "disabled", the issue does not occur; both the host and Docker run faster (but vulnerable). If STIBP is "forced", the issue does not occur; both the host and Docker run slower. You can see the configuration of your system in /sys/devices/system/cpu/vulnerabilities/spectre_v2.

In addition to STIBP, the security mitigation "spec_store_bypass_disable", which is against side-channel attacks of speculative store bypass, also degrades the performance. The option --security-opt seccomp=unconfined seems to suppress the measure too. According to Koichi Sasada's investigation, the slowdown was removed by kernel options spectre_v2_user=off spec_store_bypass_disable=off.

Solution

Unfortunately, there is no recommended solution for this issue. The option --security-opt seccomp=unconfined (or --privileged) solves the issue, but in general, I cannot recommend the usage because it is vulnerable against Spectre attacks.

Fortunately, this problem is only for CPU-intensive programs. Since almost all (Ruby on Rails) Web applications are IO-intensive or memory-intensive, you will see no significant performance improvement even if you specify the options, perhaps. Thus, I recommend you don't care the problem for a while.

In a long term: if CPUs adderss the Spectre attacks, this issue will be fundamentally solved. (But you must wait for a decade.) Or, a new VM approach based on contest threading proposed by Koichi Sasada, does not depend on indirect branches, so it may solve this issue. (But you must wait for a few years at least.)

Acknowledgment

• Those who replied to my tweet (Especially, @ryotarai found --security-opt seccomp=unconfined.)
• Those who joined the discussion on #container channel in ruby-jp Slack (Mainly, Koichi Sasada investigated how the option works.)
• Ruby committers

Postscript

My benchmark program called optcarrot, which is one of the targets for Ruby 3x3 project, is heavily affected by this issue: it runs 33 fps on the host, 14 fps on Docker.