2020-05-23

CPU-intensive Ruby/Python code runs slower on default-configured Docker

Summary

Docker enables security mechanism for Spectre vulnerability by default. This degrades the performance of almost all CPU-intensive programs, especially, interpreters like Ruby and Python. The execution time becomes about twice at worst.

Problem

This is a simple benchmark program that runs an empty loop 100M times.

i=0
while i < 100_000_000
  i+=1
end

It takes 1.3 sec. on the host, but it does 2.5 sec. on a Docker container.

On the host:

$ ruby -ve 't = Time.now; i=0;while i<100_000_000;i+=1;end; puts "#{ Time.now - t } sec"'
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
1.321703922 sec

On a Docker container:

$ docker run -it --rm ruby:2.7 ruby -ve 't = Time.now; i=0;while i<100_000_000;i+=1;end; puts "#{ Time.now - t } sec"'
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
2.452876383 sec

If you specify an option "--security-opt seccomp=unconfined" for docker run command, it runs as fast as the host.

$ docker run --security-opt seccomp=unconfined -it --rm ruby:2.7 ruby -ve 't = Time.now; i=0;while i<100_000_000;i+=1;end; puts "#{ Time.now - t } sec"'
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
1.333669449 sec

The above example uses Ruby, but I confirmed the problem with Python too. (Two-line code "i=0 / while i<100000000; i+=1" took 7.0 sec. on the host and 11 sec. on Docker.)

Note that this code does not access file system nor network. In fact, it issues no syscall at all in the main loop. Thus, in this case, there is no virtualization overhead of Docker.

You may not reproduce this issue depending upon the kernel configuration as described later.

Why

The recent Linux kernel implements some mitigation options against Spectre vulnerability).

One of them suppresses indirect branch prediction (called STIBP). This makes CPU-intensive code 2x slower, so it is disabled by default. Docker, however, runs a container with the option enabled.

(Koichi Sasada says it may depend the kernel configuration of each distribution. See Spectre Side Channels — The Linux Kernel documentation.)

This option makes almost all programs slower. According to this article, it reduced the performance of Java, Node.js, memcached, PHP, and so on. Interpreters like Ruby and Python are especially affected because they heavily depends upon indirect branches, e.g., switch/case, direct threading, and so on. They run faster on the host because the option is off, and slower on Docker because the option is on.

Using "perf stat", I measured branch miss count: 522,663 on the host, and 199,260,442 on Docker.

With --security-opt seccomp=unconfined (vulnerable against Spectre):

 Performance counter stats for process id '153095':
          1,235.67 msec task-clock                #    0.618 CPUs utilized
                 8      context-switches          #    0.006 K/sec
                 0      cpu-migrations            #    0.000 K/sec
                 2      page-faults               #    0.002 K/sec
     4,284,307,990      cycles                    #    3.467 GHz
    13,903,977,890      instructions              #    3.25  insn per cycle
     1,700,742,230      branches                  # 1376.378 M/sec
           522,663      branch-misses             #    0.03% of all branches
       2.000223507 seconds time elapsed

Without --security-opt seccomp=unconfined (not vulnerable against Spectre):

 Performance counter stats for process id '152556':
          3,300.42 msec task-clock                #    0.550 CPUs utilized
                16      context-switches          #    0.005 K/sec
                 2      cpu-migrations            #    0.001 K/sec
                 2      page-faults               #    0.001 K/sec
    11,912,594,779      cycles                    #    3.609 GHz
    13,906,818,105      instructions              #    1.17  insn per cycle
     1,701,237,677      branches                  #  515.460 M/sec
       199,260,442      branch-misses             #   11.71% of all branches
       6.000985834 seconds time elapsed

This issue is reproduced only when STIBP is "conditional", which means off by default but it can be enabled by seccomp. If STIBP is "disabled", the issue does not occur; both the host and Docker run faster (but vulnerable). If STIBP is "forced", the issue does not occur; both the host and Docker run slower. You can see the configuration of your system in /sys/devices/system/cpu/vulnerabilities/spectre_v2.

In addition to STIBP, the security mitigation "spec_store_bypass_disable", which is against side-channel attacks of speculative store bypass, also degrades the performance. The option --security-opt seccomp=unconfined seems to suppress the measure too. According to Koichi Sasada's investigation, the slowdown was removed by kernel options spectre_v2_user=off spec_store_bypass_disable=off.

Solution

Unfortunately, there is no recommended solution for this issue. The option --security-opt seccomp=unconfined (or --privileged) solves the issue, but in general, I cannot recommend the usage because it is vulnerable against Spectre attacks.

Fortunately, this problem is only for CPU-intensive programs. Since almost all (Ruby on Rails) Web applications are IO-intensive or memory-intensive, you will see no significant performance improvement even if you specify the options, perhaps. Thus, I recommend you don't care the problem for a while.

In a long term: if CPUs adderss the Spectre attacks, this issue will be fundamentally solved. (But you must wait for a decade.) Or, a new VM approach based on contest threading proposed by Koichi Sasada, does not depend on indirect branches, so it may solve this issue. (But you must wait for a few years at least.)

Acknowledgment

  • Those who replied to my tweet (Especially, @ryotarai found --security-opt seccomp=unconfined.)
  • Those who joined the discussion on #container channel in ruby-jp Slack (Mainly, Koichi Sasada investigated how the option works.)
  • Ruby committers

Postscript

My benchmark program called optcarrot, which is one of the targets for Ruby 3x3 project, is heavily affected by this issue: it runs 33 fps on the host, 14 fps on Docker.

2019-06-16

A Brief History of Pipeline Operator

Pipeline operator is experimentally introduced to the bleeding edge of Ruby. For some reason, this seems to lead a lot of people to leave their comments on the ticket of the bug tracker.

I investigated the history of the pipeline operator. I’m not an expert on the operator, so let me know if this article contains any wrong description.

What is pipeline operator?

This is what pipeline operator is from users’ perspective:

x |> f |> g |> h  # identical to h(g(f(x)))

In the function-application expression h(g(f(x))), the order of the function call (fgh) is different from the literal order in the program (hgf). This problem becomes more serious when the function names are longer:

wonderful_process_h(
  marvelous_process_g(
    fantastic_process_f(
      astonishing_argument_x
    )
  )
)

It also suffers from indentation hell.

The pipeline operator |> solves both problems:

astonishing_argument_x
|> fantastic_process_f
|> marvelous_process_g
|> wonderful_process_h

There are neither “order” problem nor indent problem.

Isabelle/ML

According to The Early History of F#, the first language that introduced pipeline operator is Isabelle/ML. We can read the discussion at 1994.

According to the discussion, its purpose was to avoid the above two problems: programmers want to write function names as their execution order. For example, we write the following code in ML to make AAA, add BBB to it, add CCC to it, and then add DDD to it:

add DDD (add CCC (add BBB (make AAA ())))

By using |>, we can write the same program as follows:

make AAA
|> add BBB
|> add CCC
|> add DDD

The operator was called “also” in the original proposal. In fact, MLs let programmers define infix operators using even alphabets, and can write make AAA add BBB add CCC add DDD by defining add infix operator based on add function. So, you actually don’t need another feature like “also”, but it would avoid messing the program with a bunch of add_XXX infix declarations.

Here are some commits involving also and |> from Isabelle/ML repository.

F

F# is a language for .NET Framework. It is based on OCaml.

Pipeline operator has been popular because it was introduced by F#. “Expert F#”, a book written by Don Syme who is the designer of F#, says “The |> forward pipe operator is perhaps the most important operator in F# programming.”

According to The Early History of F#, pipeline operator was added to the F# standard library in 2003. The reason why |> is particularly important is not only because the function order is good, but also because it is easy for type inference. The type-inference algorithm of F# is left-to-right, based on information available earlier in the program. Consider the following program:

let data = ["one"; "three"]
data |> List.map (fun s -> s.Length)

In the above case, the type inference knows that data is string list, so no type annotation is required. However, if we write it in a traditional style without |>,

List.map (fun (s: string) -> s.Length) data

An annotation s: string is required because data follows the anonymous function. (This example is attributed to “The Early History of F#”.)

Pipeline operator in F# was introduced back to OCaml in 2013. F# is based on OCaml, but with regards to pipeline operator, F# was prior to OCaml.

Three “facts” about pipeline operator in ML

I digress from history. I’d like to summarize some facts of pipeline operator in ML family.

In ML, |> is not special. It is not a built-in operator but is merely a normal function. This is a cool hack that functional programming lovers like. There are three facts to make the hack possible.

  1. Users can define their own infix operator.
  2. Currying is built-in by default.
  3. Most functions accept a “primary” argument as the final argument.

1 is cool, but 2 and 3 look particularly important.

By “primary”, I mean the subjective argument of a function; for example, a list for list manipulation function, an array for array manipulation function, etc. In F#, most functions are carefully designed to accept their “primary” argument at the last. For example, List.map accepts a closure as a first argument, a list as a second argument, and then returns a new list.

List.map   : ('a -> 'b) -> 'a list -> 'b list
Array.map  : ('a -> 'b) -> 'a[] -> 'b[]
Option.map : ('a -> 'b) -> 'a option -> 'b option
Seq.map    : ('a -> 'b) -> #seq<'a> -> seq<'b>

Thanks to these facts, we can define |> just as follow. Great!

let (|>) x f = f x

Pipeline operator and method chain

As I said, |> in F# is used for chaining functions in the execution order: x |> f |> g |> h.

By the way, a method chain in object-oriented programs is also used for chaining methods in the execution order: x.f().g().h().

I’m not sure whether F# intentionally introduced |> being on aware of the above fact, or it was just a coincidence. Anyway, some people think that the two things are related.

Elixir

Returning to history.

Elixir introduced pipeline operator. According to José Valim, the author of Elixir, it came from F#. Its appearance is really similar to F#.

x |> f |> g |> h

However, it is quite different in terms of the language specifications. In Elixir, |> is not a normal operator, but a built-in language construction.

x |> f(args..) # identical to f(x, args...)

It first evaluates the left side (x), and passes it to the right call as the first (not last) argument.

If |> is a normal operator, the left and right expression are independent. Consider an addition expression expr1 + expr2. The two exprs are both self-contained expressions. In expr1 |> expr2 of F#, expr2 is independent. If expr2 is a function that accepts two or more arguments, we can exploit currying and partial application. The “primary” argument that is typically passed in a pipeline is final, so the hack shines!

The right side of |> in Elixir is not an independent expression. It is an incomplete function call that lacks the first argument. So, Elixir’s |> is not a normal operator; rather it is considered a language construction.

The rationale of the Elixir’s design is, that the three facts are all unsatisfied in Elixir. It cannot define infix operator freely; there is no currying by default; most functions accept its “primary” argument at the first. Therefore, in my opinion, Elixir’s |> is completely different from F#’s, though it looks similar.

When considering a new feature on a “practical” programming language, I think we have to care much about what users see on its appearance, rather than what some language maniacs say. That is, if a new operator in a language A has the same syntax as an operator in some other language B, and they work differently, it might not so important from the user of the language A.

Ruby

Finally, the development version of Ruby introduced pipeline operator only a few days ago.

x |> f |> g |> h # identical to x.f().g().h()
x |> f(args...) # identical to x.f(args...)

It first evaluates the left side. And then, it calls a method on that result. The method name and arguments are written on the right side.

Like Elixir, Ruby does not satisfy the three facts. So, Ruby introduced |> as a new language construction, not as a simple operator.

|> in Elixir passes the left expression as the first argument of the right function call. On the other hand, |> in Ruby passes it as a receiver of the right method call. In Ruby, the “primary” argument is typically a receiver. So this is a natural choice, in a sense.

What is this feature practically good for? It is arguable. But I’d say that it is useful to explicitly write a multi-line method chain. And we can omit parentheses.

(1..)
.take(10)
.each {|x| p x }

1..
|> take 10
|> each {|x| p x }

You might feel more comfortable with the later style. Mind you, either is fine for me.

Anyway, the feature has some problems.

  • It is different from Elixir’s.
  • It is almost the same to the existing method call syntax (.).

My opinion for the first point: As far as I know, this is the first case to add pipeline operator to object-oriented language. Just saying “it is different from Elixir!” is not helpful at all because Ruby is not Elixir and Elixir is not object-oriented. We need to find a good design suitable for Ruby. Besides, most of the normal methods in Ruby accept the “primary” argument as a receiver. Function-style methods that accept the “primary” one in the first argument are relatively few. (Of course, there are some exceptions like File.join, Math.log, etc.) So, Elixir-style pipeline operator might not be very useful in Ruby.

My opinion for the second point: we don’t have to hurry. If matz admits that there is no use case, he will remove the feature before it is released. The next release is planned in December.

Good news is that Matz also realizes those problems and is now considering another candidate of its name and symbol. Then, Ruby’s new “pipeline” still has a chance to be a vaporware soon? I’m not sure. Keep your eyes on the development of Ruby.

Conclusion

I briefly explained the history of pipeline operator. I just investigated all on one night, so let me know if I was wrong. (I’m a non-native speaker, so don’t get me wrong if the expression is not appropriate.)

Acknowledgment

Keiichiro Shikano kindly reviewed the draft of this article.

2010-12-23

Merry Quine-mas 2010

Tonight is the eve of the anniversary of W. V. Quine's death. (in Japanese time zone :-)

 %;';;X=   %q(x=1;                                                     z="%c"%
 32;eval   %w(w=N=                   Merry                             =128;r=
 z*79+$/; ;p=open"                  /                                  dev/dsp
  ","w"rescue$_;/                  /                                   ;eval"s
   rand(%s)"%$*[    d=0];a=[];s=(r+z+%    (%;';;X=#{z*    3}%q(#{X});e val(X))
     +r*=23).l      ines.map{|l|l.chop+  r};t=proc{|y|s  [y][x-1,w].sca n(/\S
   /){s[y+1][m=x    -1+$`  .size  ,1]==           z&&a<  <[$&,m,        y,s[y
  ][m]=$/]};y};t[   23];f  =(0..  1071)   .map{|i|c,x,y   =a.shuffle!.  pop;s
 [y][x]=z ;[794-46  .03*i  **0.4  +y/k=  (3+ra    nd*w=          3)/9,[t
 [t[y]-1   ],x,c],  (0..y  /k).m  ap{[y  -=k,0.5+x+=k*r  and*2-k,?*]}]} ;"|ii
 xCG40_|   CK8&d9B  ix0,B  J?B?G  ^K9CG   CJ|ii>Ci?9CV;   =*-*.sEA.2gA  w>dc=
  :     `:z:J>      s      9QvA:>      J      taaA>`      :      @AP;Fa     c
  y   =Anm|?Bdqr    h    ?<CDJCD<IU    <    DuCJ>:d_C?    h    ?<CWC:'.'+   r
  9  A.*. '>| =*-   *   .:9i 9A> :JC   s   :Q> :>9 CJA=   g   `;C i9A >M".  b
 yt  es{|c|a+=([%  [+.  5$2-'30&)"/:  (7%  !*][c%19].or  d]*  [47,23,11][4  -c
 /1   9]<     <3   3).   *c/     95   +1}   ;$     ><<   (T=   "%     q\x   7e
  '    #{e="\x1    b [    "}H#{e}J    # {    e}?25")+    " l    ";863.do    w
   n to(0){|i|b=r +   [ "'"*25]*2*"_ (   c )_2010_@mame t   t er,@hirekoke "
    ;f.map{ |j,x,v     |i<j&&( y,x,c=     v.pop| |x;x<0|     |x>78| |b[x+y.
   to_i*80]=0<i&&i<   40??#:c)};($><<(   i>0?e+"H":T+"h")   +b).flush;(p)?4.
   times{N. times{v   =8;4.tim es{|l|t   =a[l*N* 9+d/N%11   52];t>3 3&&v+=(d
    *0.11*4**(t/24     .0-l%3))%8<=>4     };d+=1;p<<[16*     v].pack("C*")}
      ;p.flush}:         sleep(0.1)         };puts)*""         ;);eval(X)
'''''''''''''''''''''''''_(c)_2010_@mametter,@hirekoke'''''''''''''''''''''''''

Save the code as "quinemas2010.rb" and run it under an terminal bigger than 80 x 24. You can use cygwin if you're a windows user. You may need to use "aoss" or "padsp" if you're using Un*x. I tested it with ruby 1.8.7p299 and 1.9.2p0.

A screencast for you guys who are lazy:

FAQ

  • Q. Is this a pseudo-Quine because its output is not strictly equal to the code?
    • A. Yes. This is a Quine generator. Run it as follows. You'll find tmp1.rb is certainly a Quine.
$ ruby quinemas2010.rb 0 > tmp1.rb
$ ruby tmp1.rb 0 > tmp2.rb
$ diff tmp1.rb tmp2.rb
  • Q. What does the command line argument mean?

    • A. A random seed which affects how to snow.
  • Q. How does it animate?

    • A. It just prints each frame with the aid of escape sequence. Because it uses only ANSI escape code, it works on almost all terminal emulator (but not cmd.exe).
  • Q. How does it play a sound?

translated from mamememo in Japanese (2010-12-24).

2010-09-07

inline-ook: a library that allows you to embed Ook! program in Ruby script

In RubyKaigi 2010, Chad Fowler introduced Ook! which is a programming language designed for orang-utans.

Ook! program is just like Ruby program, isn't it? Ook. Ook! Ook. Ook? can be interpreted as Ook.Ook!(Ook.Ook?) in Ruby. Thus I tried to embed Ook! program into Ruby program, and succeeded:

require "inline-ook"

puts "Hello, Ruby!"  #=> Hello, Ruby!

ook begin
  Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
  Ook. Ook. Ook. Ook. Ook. Ook. Ook! Ook? Ook. Ook? Ook. Ook.
  Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
  Ook. Ook. Ook. Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
  Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
  Ook. Ook. Ook. Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
  Ook. Ook. Ook. Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
  Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook? Ook.
  Ook? Ook. Ook? Ook. Ook? Ook. Ook! Ook! Ook? Ook! Ook. Ook?
  Ook! Ook. Ook. Ook? Ook. Ook. Ook. Ook. Ook! Ook. Ook. Ook.
  Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
  Ook! Ook. Ook! Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook! Ook.
  Ook. Ook? Ook! Ook! Ook! Ook. Ook! Ook! Ook! Ook! Ook! Ook!
  Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook!
  Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook. Ook. Ook? Ook! Ook!
  Ook! Ook! Ook! Ook. Ook? Ook. Ook? Ook. Ook! Ook. Ook! Ook!
  Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook. Ook. Ook? Ook. Ook.
  Ook! Ook. Ook? Ook. Ook? Ook. Ook? Ook. Ook. Ook. Ook. Ook.
  Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.
  Ook. Ook. Ook. Ook. Ook! Ook.
ook end  #=> Hello, Ook!

puts "Hello, Ruby again!"  #=> Hello, Ruby again!

You can write any Ook! program in Ruby program just by surrounding ook begin and ook end. You don't have to use here document. You don't have to apply any patch to ruby interpreter.

$ ruby example/hello.ook.rb
Hello, Ruby!
Hello, Ook!
Hello, Ruby again!

You can install inline-ook as follows. I tested it with 1.8.7-p72 and 1.9.2-p0.

$ gem install inline-ook

translated from mamememo in Japanese (2010-09-07).

2010-09-05

The Qlobe

My Quines are sometimes specialized for Japanese. But to attend RubyConf, I should write "global" Quines.

v=0000;eval$s=%q~d=%!^Lcf<LK8,                  _@7gj*LJ=c5nM)Tp1g0%Xv.,S[<>YoP
4ZojjV)O>qIH1/n[|2yE[>:ieC       "%.#%  :::##"       97N-A&Kj_K_><wS5rtWk@*a+Y5
yH?b[F^e7C/56j|pmRe+:)B     "##%      ::##########"     O98(Zh)'Iof*nm.,$C5Nyt=
PPu01Avw^<IiQ=5$'D-y?    "##:         ###############"    g6`YT+qLw9k^ch|K'),tc
6ygIL8xI#LNz3v}T=4W    "#            #.   .####:#######"    lL27FZ0ij)7TQCI)P7u
}RT5-iJbbG5P-DHB<.   "              ##### # :############"   R,YvZ_rnv6ky-G+4U'
$*are@b4U351Q-ug5   "              #######################"   00x8RR%`Om7VDp4M5
PFixrPvl&<p[]1IJ   "              ############:####  %#####"   EGgDt8Lm#;bc4zS^
y]0`_PstfUxOC(q   "              .#############:##%   .##  ."   /,}.YOIFj(k&q_V
zcaAi?]^lCVYp!;  " %%            .################.     #.   "  ;s="v=%04o;ev"%
(;v=(v-($*+[45,  ":####:          :##############%       :   "  ])[n=0].to_i;)%
360)+"al$s=%q#{  "%######.              #########            "  ;;"%c"%126+$s<<
126}";d.gsub!(/  "##########.           #######%             "  |\s|".*"/,"");;
require"zlib"||  "###########           :######.             "  ;d=d.unpack"C*"
d.map{|c|n=(n||  ":#########:           .######: .           "  )*90+(c-2)%91};
e=["%x"%n].pack   " :#######%           :###### #:          "   &&"H*";e=Zlib::
Inflate.inflate(   "  ######%           .####% ::          "   &&e).unpack("b*"
)[0];22.times{|y|   "  ####%             %###             "   ;w=(Math.sqrt(1-(
(y*2.0-21)/22)**(;   " .###:             .#%             "   ;2))*23).floor;(w*
2-1).times{|x|u=(e+    " %##                           "    )[y*z=360,z]*2;u=u[
90*x/w+v+90,90/w];s[(    " #.                        "    ;y*80)+120-w+x]=(""<<
32<<".:%#")[4*u.count((     " .                   "     ;"0"))/u.size]}};;puts\
s+";_ The Qlobe#{" "*18+ (       "#  :#######"       ;"Copyright(C).Yusuke End\
oh, 2010")}";exit~;_ The Qlobe                  Copyright(C).Yusuke Endoh, 2010

This quine turns 45 degrees when executed; thus, it will come around when executed eight times (see below). You can make it any-degree turn by giving an integer as an command-line argument.

v=0416;eval$s=%q~d=%!^Lcf<LK8,                  _@7gj*LJ=c5nM)Tp1g0%Xv.,S[<>YoP
4ZojjV)O>qIH1/n[|2yE[>:ieC       ".#####%.#% "       97N-A&Kj_K_><wS5rtWk@*a+Y5
yH?b[F^e7C/56j|pmRe+:)B     "    .#####%##%      :"     O98(Zh)'Iof*nm.,$C5Nyt=
PPu01Avw^<IiQ=5$'D-y?    "       #####%###:         #"    g6`YT+qLw9k^ch|K'),tc
6ygIL8xI#LNz3v}T=4W    "        %########            #."    lL27FZ0ij)7TQCI)P7u
}RT5-iJbbG5P-DHB<.   "          :#######              ###"   R,YvZ_rnv6ky-G+4U'
$*are@b4U351Q-ug5   "            ###:  :              ####"   00x8RR%`Om7VDp4M5
PFixrPvl&<p[]1IJ   "              %#...               #####"   EGgDt8Lm#;bc4zS^
y]0`_PstfUxOC(q   "                %###               .#####"   /,}.YOIFj(k&q_V
zcaAi?]^lCVYp!;  "                    :. %%            .#####"  ;s="v=%04o;ev"%
(;v=(v-($*+[45,  "                      :####:          :####"  ])[n=0].to_i;)%
360)+"al$s=%q#{  "                      %######.             "  ;;"%c"%126+$s<<
126}";d.gsub!(/  "                      ##########.          "  |\s|".*"/,"");;
require"zlib"||  "                      ###########          "  ;d=d.unpack"C*"
d.map{|c|n=(n||  "                      :#########:          "  )*90+(c-2)%91};
e=["%x"%n].pack   "                      :#######%          "   &&"H*";e=Zlib::
Inflate.inflate(   "                      ######%          "   &&e).unpack("b*"
)[0];22.times{|y|   "                     ####%           "   ;w=(Math.sqrt(1-(
(y*2.0-21)/22)**(;   "                   .###:           "   ;2))*23).floor;(w*
2-1).times{|x|u=(e+    "                 %##           "    )[y*z=360,z]*2;u=u[
90*x/w+v+90,90/w];s[(    "               #.          "    ;y*80)+120-w+x]=(""<<
32<<".:%#")[4*u.count((     "            .        "     ;"0"))/u.size]}};;puts\
s+";_ The Qlobe#{" "*18+ (       " .#####  :#"       ;"Copyright(C).Yusuke End\
oh, 2010")}";exit~;_ The Qlobe                  Copyright(C).Yusuke Endoh, 2010

v=0341;eval$s=%q~d=%!^Lcf<LK8,                  _@7gj*LJ=c5nM)Tp1g0%Xv.,S[<>YoP
4ZojjV)O>qIH1/n[|2yE[>:ieC       "#% .#####%."       97N-A&Kj_K_><wS5rtWk@*a+Y5
yH?b[F^e7C/56j|pmRe+:)B     " :        %#####%##. "     O98(Zh)'Iof*nm.,$C5Nyt=
PPu01Avw^<IiQ=5$'D-y?    "              #####%###:   "    g6`YT+qLw9k^ch|K'),tc
6ygIL8xI#LNz3v}T=4W    "                %########      "    lL27FZ0ij)7TQCI)P7u
}RT5-iJbbG5P-DHB<.   "                   :#######        "   R,YvZ_rnv6ky-G+4U'
$*are@b4U351Q-ug5   "                     :###   :        "   00x8RR%`Om7VDp4M5
PFixrPvl&<p[]1IJ   "                        %#...          "   EGgDt8Lm#;bc4zS^
y]0`_PstfUxOC(q   "                          .###:          "   /,}.YOIFj(k&q_V
zcaAi?]^lCVYp!;  "                               :. %%       "  ;s="v=%04o;ev"%
(;v=(v-($*+[45,  "                                 :####:    "  ])[n=0].to_i;)%
360)+"al$s=%q#{  "                                 %######.  "  ;;"%c"%126+$s<<
126}";d.gsub!(/  "                                 ##########"  |\s|".*"/,"");;
require"zlib"||  "%                                ##########"  ;d=d.unpack"C*"
d.map{|c|n=(n||  "                                 :#########"  )*90+(c-2)%91};
e=["%x"%n].pack   "                                 ########"   &&"H*";e=Zlib::
Inflate.inflate(   "#.                              ######%"   &&e).unpack("b*"
)[0];22.times{|y|   "##                            :####. "   ;w=(Math.sqrt(1-(
(y*2.0-21)/22)**(;   "#.                          .###:  "   ;2))*23).floor;(w*
2-1).times{|x|u=(e+    "%    .                   %##   "    )[y*z=360,z]*2;u=u[
90*x/w+v+90,90/w];s[(    "   .                  #.   "    ;y*80)+120-w+x]=(""<<
32<<".:%#")[4*u.count((     "                 :   "     ;"0"))/u.size]}};;puts\
s+";_ The Qlobe#{" "*18+ (       "#.  .##### "       ;"Copyright(C).Yusuke End\
oh, 2010")}";exit~;_ The Qlobe                  Copyright(C).Yusuke Endoh, 2010

v=0264;eval$s=%q~d=%!^Lcf<LK8,                  _@7gj*LJ=c5nM)Tp1g0%Xv.,S[<>YoP
4ZojjV)O>qIH1/n[|2yE[>:ieC       "####% .####"       97N-A&Kj_K_><wS5rtWk@*a+Y5
yH?b[F^e7C/56j|pmRe+:)B     "#####  :       .#####"     O98(Zh)'Iof*nm.,$C5Nyt=
PPu01Avw^<IiQ=5$'D-y?    "######.              #####%"    g6`YT+qLw9k^ch|K'),tc
6ygIL8xI#LNz3v}T=4W    "####:  .                %######"    lL27FZ0ij)7TQCI)P7u
}RT5-iJbbG5P-DHB<.   "####  .                     :######"   R,YvZ_rnv6ky-G+4U'
$*are@b4U351Q-ug5   "####%                          ###:  "   00x8RR%`Om7VDp4M5
PFixrPvl&<p[]1IJ   "###                               %#..."   EGgDt8Lm#;bc4zS^
y]0`_PstfUxOC(q   "##  ..                               %###"   /,}.YOIFj(k&q_V
zcaAi?]^lCVYp!;  ":#.  .                                    :"  ;s="v=%04o;ev"%
(;v=(v-($*+[45,  ":   : :                                    "  ])[n=0].to_i;)%
360)+"al$s=%q#{  "% %#:                                      "  ;;"%c"%126+$s<<
126}";d.gsub!(/  "#::#.:  %#%                                "  |\s|".*"/,"");;
require"zlib"||  "  :      :#%                               "  ;d=d.unpack"C*"
d.map{|c|n=(n||  "       :% :                                "  )*90+(c-2)%91};
e=["%x"%n].pack   "     %###%%                              "   &&"H*";e=Zlib::
Inflate.inflate(   "   ########.                           "   &&e).unpack("b*"
)[0];22.times{|y|   "   ########.                         "   ;w=(Math.sqrt(1-(
(y*2.0-21)/22)**(;   "   ##:%###.                        "   ;2))*23).floor;(w*
2-1).times{|x|u=(e+    "       %%    .                 "    )[y*z=360,z]*2;u=u[
90*x/w+v+90,90/w];s[(    "          .                "    ;y*80)+120-w+x]=(""<<
32<<".:%#")[4*u.count((     "                     "     ;"0"))/u.size]}};;puts\
s+";_ The Qlobe#{" "*18+ (       "####.  .###"       ;"Copyright(C).Yusuke End\
oh, 2010")}";exit~;_ The Qlobe                  Copyright(C).Yusuke Endoh, 2010

v=0207;eval$s=%q~d=%!^Lcf<LK8,                  _@7gj*LJ=c5nM)Tp1g0%Xv.,S[<>YoP
4ZojjV)O>qIH1/n[|2yE[>:ieC       "#######% .#"       97N-A&Kj_K_><wS5rtWk@*a+Y5
yH?b[F^e7C/56j|pmRe+:)B     "##########: :        "     O98(Zh)'Iof*nm.,$C5Nyt=
PPu01Avw^<IiQ=5$'D-y?    "#############.             "    g6`YT+qLw9k^ch|K'),tc
6ygIL8xI#LNz3v}T=4W    "############:  .               "    lL27FZ0ij)7TQCI)P7u
}RT5-iJbbG5P-DHB<.   "#############  .                   "   R,YvZ_rnv6ky-G+4U'
$*are@b4U351Q-ug5   "##############                       "   00x8RR%`Om7VDp4M5
PFixrPvl&<p[]1IJ   "#  %#########                          "   EGgDt8Lm#;bc4zS^
y]0`_PstfUxOC(q   "    %#:  %##:  :                         "   /,}.YOIFj(k&q_V
zcaAi?]^lCVYp!;  "     #.   .:#.  .                          "  ;s="v=%04o;ev"%
(;v=(v-($*+[45,  "      :    :   : :                         "  ])[n=0].to_i;)%
360)+"al$s=%q#{  "          :% %#:                           "  ;;"%c"%126+$s<<
126}";d.gsub!(/  "           #::#.:  %#%                     "  |\s|".*"/,"");;
require"zlib"||  "             :      :#%                    "  ;d=d.unpack"C*"
d.map{|c|n=(n||  "                  :% :                     "  )*90+(c-2)%91};
e=["%x"%n].pack   "               .###%#                    "   &&"H*";e=Zlib::
Inflate.inflate(   "             ########.                 "   &&e).unpack("b*"
)[0];22.times{|y|   "            :########                "   ;w=(Math.sqrt(1-(
(y*2.0-21)/22)**(;   "            ##:%###.               "   ;2))*23).floor;(w*
2-1).times{|x|u=(e+    "               %%    .         "    )[y*z=360,z]*2;u=u[
90*x/w+v+90,90/w];s[(    "                 .         "    ;y*80)+120-w+x]=(""<<
32<<".:%#")[4*u.count((     "                     "     ;"0"))/u.size]}};;puts\
s+";_ The Qlobe#{" "*18+ (       "#######.  ."       ;"Copyright(C).Yusuke End\
oh, 2010")}";exit~;_ The Qlobe                  Copyright(C).Yusuke Endoh, 2010

v=0132;eval$s=%q~d=%!^Lcf<LK8,                  _@7gj*LJ=c5nM)Tp1g0%Xv.,S[<>YoP
4ZojjV)O>qIH1/n[|2yE[>:ieC       ":::#######%"       97N-A&Kj_K_><wS5rtWk@*a+Y5
yH?b[F^e7C/56j|pmRe+:)B     "################  :  "     O98(Zh)'Iof*nm.,$C5Nyt=
PPu01Avw^<IiQ=5$'D-y?    "####################.      "    g6`YT+qLw9k^ch|K'),tc
6ygIL8xI#LNz3v}T=4W    "  .####:############:  .       "    lL27FZ0ij)7TQCI)P7u
}RT5-iJbbG5P-DHB<.   "# # :#################  .          "   R,YvZ_rnv6ky-G+4U'
$*are@b4U351Q-ug5   "#######################%             "   00x8RR%`Om7VDp4M5
PFixrPvl&<p[]1IJ   "######:####  %#########                "   EGgDt8Lm#;bc4zS^
y]0`_PstfUxOC(q   "#######:##%   .##  .###  ..              "   /,}.YOIFj(k&q_V
zcaAi?]^lCVYp!;  "##########.     #.   .:#.  .               "  ;s="v=%04o;ev"%
(;v=(v-($*+[45,  "#########%       :    :   : :              "  ])[n=0].to_i;)%
360)+"al$s=%q#{  "#########            :% %#:                "  ;;"%c"%126+$s<<
126}";d.gsub!(/  "#######%              #::#.:  %#%          "  |\s|".*"/,"");;
require"zlib"||  ":######.                :      :#%         "  ;d=d.unpack"C*"
d.map{|c|n=(n||  ".######: .                   :% :          "  )*90+(c-2)%91};
e=["%x"%n].pack   ":###### #:                %###%%         "   &&"H*";e=Zlib::
Inflate.inflate(   ".####% ::              ########.       "   &&e).unpack("b*"
)[0];22.times{|y|   " %###                 ########.      "   ;w=(Math.sqrt(1-(
(y*2.0-21)/22)**(;   " .#%                 ##:%###.      "   ;2))*23).floor;(w*
2-1).times{|x|u=(e+    "                       %%    . "    )[y*z=360,z]*2;u=u[
90*x/w+v+90,90/w];s[(    "                        .  "    ;y*80)+120-w+x]=(""<<
32<<".:%#")[4*u.count((     "                     "     ;"0"))/u.size]}};;puts\
s+";_ The Qlobe#{" "*18+ (       "##########."       ;"Copyright(C).Yusuke End\
oh, 2010")}";exit~;_ The Qlobe                  Copyright(C).Yusuke Endoh, 2010

v=0055;eval$s=%q~d=%!^Lcf<LK8,                  _@7gj*LJ=c5nM)Tp1g0%Xv.,S[<>YoP
4ZojjV)O>qIH1/n[|2yE[>:ieC       "%  :::#####"       97N-A&Kj_K_><wS5rtWk@*a+Y5
yH?b[F^e7C/56j|pmRe+:)B     "   .:################"     O98(Zh)'Iof*nm.,$C5Nyt=
PPu01Avw^<IiQ=5$'D-y?    "     ######################"    g6`YT+qLw9k^ch|K'),tc
6ygIL8xI#LNz3v}T=4W    "     #.   .####:############:  "    lL27FZ0ij)7TQCI)P7u
}RT5-iJbbG5P-DHB<.   "     ##### # :#################  . "   R,YvZ_rnv6ky-G+4U'
$*are@b4U351Q-ug5   "    :############################    "   00x8RR%`Om7VDp4M5
PFixrPvl&<p[]1IJ   "    ############:####  %#########      "   EGgDt8Lm#;bc4zS^
y]0`_PstfUxOC(q   "    %############:###    %#:  %##:  :    "   /,}.YOIFj(k&q_V
zcaAi?]^lCVYp!;  "    .################.     #.   .:#.  .    "  ;s="v=%04o;ev"%
(;v=(v-($*+[45,  "     :##############%       :    :   : :   "  ])[n=0].to_i;)%
360)+"al$s=%q#{  "           #########            :% %#:     "  ;;"%c"%126+$s<<
126}";d.gsub!(/  "           #######%              #::#.:  %#"  |\s|".*"/,"");;
require"zlib"||  "           :######.                :      :"  ;d=d.unpack"C*"
d.map{|c|n=(n||  "           .######: .                   :% "  )*90+(c-2)%91};
e=["%x"%n].pack   "           ######%:#                .###%"   &&"H*";e=Zlib::
Inflate.inflate(   "          .####% ::              ######"   &&e).unpack("b*"
)[0];22.times{|y|   "          .###:                :#####"   ;w=(Math.sqrt(1-(
(y*2.0-21)/22)**(;   "          .#%                 ##:%#"   ;2))*23).floor;(w*
2-1).times{|x|u=(e+    "                               "    )[y*z=360,z]*2;u=u[
90*x/w+v+90,90/w];s[(    "                           "    ;y*80)+120-w+x]=(""<<
32<<".:%#")[4*u.count((     "                     "     ;"0"))/u.size]}};;puts\
s+";_ The Qlobe#{" "*18+ (       ":##########"       ;"Copyright(C).Yusuke End\
oh, 2010")}";exit~;_ The Qlobe                  Copyright(C).Yusuke Endoh, 2010

v=0000;eval$s=%q~d=%!^Lcf<LK8,                  _@7gj*LJ=c5nM)Tp1g0%Xv.,S[<>YoP
4ZojjV)O>qIH1/n[|2yE[>:ieC       "%.#%  :::##"       97N-A&Kj_K_><wS5rtWk@*a+Y5
yH?b[F^e7C/56j|pmRe+:)B     "##%      ::##########"     O98(Zh)'Iof*nm.,$C5Nyt=
PPu01Avw^<IiQ=5$'D-y?    "##:         ###############"    g6`YT+qLw9k^ch|K'),tc
6ygIL8xI#LNz3v}T=4W    "#            #.   .####:#######"    lL27FZ0ij)7TQCI)P7u
}RT5-iJbbG5P-DHB<.   "              ##### # :############"   R,YvZ_rnv6ky-G+4U'
$*are@b4U351Q-ug5   "              #######################"   00x8RR%`Om7VDp4M5
PFixrPvl&<p[]1IJ   "              ############:####  %#####"   EGgDt8Lm#;bc4zS^
y]0`_PstfUxOC(q   "              .#############:##%   .##  ."   /,}.YOIFj(k&q_V
zcaAi?]^lCVYp!;  " %%            .################.     #.   "  ;s="v=%04o;ev"%
(;v=(v-($*+[45,  ":####:          :##############%       :   "  ])[n=0].to_i;)%
360)+"al$s=%q#{  "%######.              #########            "  ;;"%c"%126+$s<<
126}";d.gsub!(/  "##########.           #######%             "  |\s|".*"/,"");;
require"zlib"||  "###########           :######.             "  ;d=d.unpack"C*"
d.map{|c|n=(n||  ":#########:           .######: .           "  )*90+(c-2)%91};
e=["%x"%n].pack   " :#######%           :###### #:          "   &&"H*";e=Zlib::
Inflate.inflate(   "  ######%           .####% ::          "   &&e).unpack("b*"
)[0];22.times{|y|   "  ####%             %###             "   ;w=(Math.sqrt(1-(
(y*2.0-21)/22)**(;   " .###:             .#%             "   ;2))*23).floor;(w*
2-1).times{|x|u=(e+    " %##                           "    )[y*z=360,z]*2;u=u[
90*x/w+v+90,90/w];s[(    " #.                        "    ;y*80)+120-w+x]=(""<<
32<<".:%#")[4*u.count((     " .                   "     ;"0"))/u.size]}};;puts\
s+";_ The Qlobe#{" "*18+ (       "#  :#######"       ;"Copyright(C).Yusuke End\
oh, 2010")}";exit~;_ The Qlobe                  Copyright(C).Yusuke Endoh, 2010

translated from mamememo in Japanese (2010-09-05).

2010-08-30

RubyKaigi 2010

Long time no update!

I attended RubyKaigi 2010 at Tsukuba, Japan. It was really exciting!

I made a presentation, "Esoteric, Obfuscated Ruby Programming", which introduces esoteric programming with Ruby. Here is the presentation material:

The slide is written in both Japanese and (weird) English.

Some programs will work on any platform, but some depends on Linux, gcc and its sound driver (/dev/dsp). If you cannot run them yourself, you can look my presentation movie which includes demos (spoken in Japanese, but all the programs are written in Ruby, of course!).

I appeared on stage three times in this RubyKaigi.

  • committer Q&A, as a 1.9.2 assistant release manager. By questioning to the audience, I clarified the fact that many people wants core team to backport only security patches rather than new feature patches.
  • Matz's keynote, because I won "Ruby 1.9.2 Award" for contribution to ruby 1.9.2 release! I'm really proud of that.
  • my presentation, "Esoteric, Obfuscated Ruby Programming". It seemed to get a favorable reception, from some geeks :-)

translated from mamememo in Japanese (2010-08-29).

2009-12-24

Merry Quine-mas

open("/dev/dsp","wb"){|h|s=%q{d=["-KQW[UZbgu*HNT+]TNOOOTZ+cZTUUUUUZbagmssUZbagm
ss+wmpgja+KQW[dfnu","-KEKOINV[W*HBH+QHBCCCHN+WNHIIIIINVU[aUUINVU[aUU+YOR[^I+KEK
OXZbW","-W[acg vsc*TZ`+eaaaaa--vucavuca+eadsvs+W[dgvrtc","-K991LIL77777dIIIII--
LKKILKKI+Mad[   ^U+K991LHJK"].map{|l|l.unpack("C*").map{|c|[(c/6-4)*12/7-8,"012
35b"[c%6,1].     hex]}*4};y=32.chr;l="@"+[(m="Jnx4sn3sgd1")+"vnqkc!6sgd2Lnqc4gz
r4bnld;6/Ld       s2dzqsg6qd2@bdhud6gdq2Khmf;77/Lds2du4@dqx4gdzqs6oqd2@ozqd4ghl
4qnnl,+Amc         2gdz++2   @udm   4z        mc      2gdz      4@+u   dm   2zm
c2mz+@stqd+r     hmf",m+"E    zq    sg  !6sgd2Sz  u4@h  nt  q4qd  hfm  r;  6/Ld
s2ldm6sgdhq       2rnmfr6d  l    2  @o       knx      ;77/      Wghkd2    ehdkc
r4zmc4eknn         cr,6qnb  jr  ,2  gh  kkr,4zmc  4okz  hm  r+Rd  2@odz  s+2+,6
qd2@odzs6           +sgd2r  ntmc+@  hm        f+  inx"  ,"  Nn4l  nqd3k  ds1rhm
r6zmc2rn             4@qqnvr4fqnv,6/Nnq2sgnqmr6hm2@edrs6sgd2fqntmc;77/Hd2bnldr4
sn4lzjd               6Hhr2ak    d4@rrh   mfr4eknv+Fzq2zr+2+,6ezq2zr,6+sgd2btqr
d+hr+entmc         ","Hd4qtkdr3s  gd1   vnqkc6vhsg2sqtsg4zmc4fqzbd,6/Amc2lzjdr6
sgd2mz6@s           hnmr2oqnud77/ T   gd2fk   n4@q   hdr4    ne6Hh       r2qhfg
s4@dntr4             @mdrr,+Amc2v   nm+2+6@    cd    qr,  2v  nm6  +@cdqr2ne+Hh
r+knud"               ].map{|a|   a ,b,c,e,  *    f  =a.  sp  lit"      +";[a,c
=b+c+f                 *"2",c   ,b+  e+f*"4  "+  ".  8/        /"]*"6/"  }.join
.tr("                   a-z   /","b-    za\  n").gs  ub  (/^/  ,"       #"<<32)
;c=0;640.tim     es{|w|c<1&&(l=~/(@)?(.*?)(\d)/m;($>.<<$1?$2:y+$2).flush;l=$';c
=eval$3);c-=     1;a=[128]*z=1200;4.times{|j|t,n=d[j].pop;f=0.3456*2**(t/12.0-j
/2);(n>0?(d[     j]<<[t,n-1];z):800).times{|i|t>-3&&a[i]+=16*Math.sin(f.*w*z+i)
}};(h.<<a.pack"C*").flush};puts(s.gsub(/./){"!">$&?"#":y}+%(\nopen("/dev/dsp#{"
(c)Y.Endoh2009";'",'}"wb"){|h|s=%q{#{s}};eval(s.split*"")}))};eval(s.split*"")}

Run it under environment where /dev/dsp is available. If you are windows user, you can use cygwin. It often abends on cygwin. Please re-run it until it works.

The structure of the program:

  • Lines 1-4: encoded musical score data
  • Lines 4-5: score decoder
  • Lines 5-17: crypted compressed lyrics data (with timing data)
  • Lines 17-19: lyrics decompressor
  • Lines 20-21: lyrics player
  • Lines 21-23: wave synthesizer
  • Lines 23-24: quine

translated from mamememo in Japanese (2009-12-24).