Erlang Kung Fu in Three Hours

OSCON 2012, Portland

Garrett Smith, CloudBees

@gart1

Presenter Notes

Three Hours, Really?

About Erlang
Eh, forget Erlang - solve a problem!
Code 'til we drop!
Okay, learn Erlang :)

Presenter Notes

About Erlang

Presenter Notes

History (Broad Strokes)

1985 - 1986 Erlang born
1987 - 1991 Erlang's early years, lots of change
1992 - 1998 Erlang commericalized, ongoing development and use
1998 AXD301 shipped, Erlang banned in Ericsson, open sourced
2007 - 2008 Popularity explosion (books, OSS, adoption)
Today Widepsread but niche (Amazon, Facebook, T-Mobile, Nokia, Yahoo, Ericsson, CloudBees)

Presenter Notes

Key Points

OS independent virtual machine
Massive fine grained concurrency
Asynchronous message passing
Reliability over performance
Functional, but not purely functional

Presenter Notes

Upcoming: Watch For This

Erlang is an operating system for your code
Erlang is a pragmatic functional language

Presenter Notes

Solving Problems

More Fun Than Programming!

Presenter Notes

Our Problem

Presenter Notes

Our Solution

Presenter Notes

First: Monitor something

Presenter Notes

Sample Web App

python -m SimpleHTTPServer
200 response is Good
Everything else is Bad

Presenter Notes

Next: Monitor with code

Presenter Notes

Erlang Has Batteries Included

Huge, high quality core library
Excellent documentation
Useful links

Presenter Notes

Use httpc to check site

Presenter Notes

httpc:request/1

Eshell V5.9.1  (abort with ^G)
1> httpc:request("http://localhost:8000").
** exception exit: {noproc,
                    {gen_server,call,
                     [httpc_manager,
                      {request,
                       {request,undefined,<0.32.0>,0,http,
                        {"localhost",8000},
                        "/",[],get,
                        {http_request_h,undefined,"keep-alive",undefined,
                         undefined,undefined,undefined,undefined,undefined,
                         undefined,...},
                        {[],[]},
                        {http_options,"HTTP/1.1",infinity,true,
                         {essl,[]},
                         undefined,false,infinity,...},
                        "http://localhost:8000",[],none,[],1340149470691,
                        undefined,undefined,false}},
                      infinity]}}
     in function  gen_server:call/3 (gen_server.erl, line 188)
     in call from httpc:handle_request/9 (httpc.erl, line 562)

Presenter Notes

WTF Erlang

Presenter Notes

Recall

Erlang is an operating system for your code

Presenter Notes

The Erlang VM

Presenter Notes

Some Erlang Apps

Presenter Notes

Revisit the Carnage

    Eshell V5.9.1  (abort with ^G)
    1> httpc:request("http://localhost:8000").
    ** exception exit: {noproc,
                        {gen_server,call,
    ...
        in function  gen_server:call/3 (gen_server.erl, line 188)
        in call from httpc:handle_request/9 (httpc.erl, line 562)

noproc means someone expected a process to be running, it wasn't
Often means "we need to start an app"
In this case httpc is a part of the inets app

Presenter Notes

Start `inets`

2> appmon:start().
{ok,<0.36.0>}
3> application:start(inets).
ok
4> httpc:request("http://localhost:8000").
{ok,{{"HTTP/1.0",200,"OK"},
     [{"date","Wed, 20 Jun 2012 00:28:46 GMT"},
      {"server","SimpleHTTP/0.6 Python/2.7.3"},
      {"content-length","178"},
      {"content-type","text/html; charset=UTF-8"}],
     "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\"><html>\n<title>Directory listing for /</title>\n<body>\n<h2>Directory listing for /</h2>\n<hr>\n<ul>\n</ul>\n<hr>\n</body>\n</html>\n"}}

Presenter Notes

App Monitor

Presenter Notes

Basic Erlang Data Types

Integers and Floats: 1, 2, 45.5
Atoms: hello
Strings: "hello"
Binaries: <<"hello">>
Tuples: {color, "red"}
Lists: ["red", "green", "blue"]

Presenter Notes

Variables

Assigned:

    X = 1.

Bad match:

    X = 2.
    ** exception error: no match of right hand side value 2

Assigned:

    {Y, Z} = {1, 2}.
    Y = 1.
    Z = 2.

Bad match:

    {ok, Val} = {error, "oops"}.
    ** exception error: no match of right hand side value {error, "oops"}

Presenter Notes

Lists

Define a list:

L = [1, 2, 3].

Cons to construct a new list by appending head:

[0,1,2,3] = [0|L].

Cons pattern match to separate head and tail:

[H|T] = L,
H = 1,
T = [2,3].

lists module provides useful higher order functions:

Increment = fun(X) -> X + 1 end,
[2,3,4] = lists:map(Increment, L).

Presenter Notes

Next: Run check in a loop

Presenter Notes

Erlang Modules

Contain code
Compile to "beam" bytecode
Provide a remedial but useful namespace
For public access, function must be exported
Compile using erlc

Presenter Notes

`monitor.erl`

-module(monitor).

-export([run/0]).

run() ->
    Result = httpc:request("http://localhost:8000"),
    io:format("~n~p~n", [Result]),
    timer:sleep(5000),
    run().

run is a function of "arity" 0 (no args)
Runs the check and prints the result
Waits using timer:sleep/1
Makes a "tail call" to itself (recursion)

Presenter Notes

Yaaaah, we have a monitor!

Presenter Notes

Recall

Erlang is a pragmatic functional language

Presenter Notes

Embarrassingly Obvious Problems

Reduce problems to the utterly trivial
Keep doing that until it's all obvious
Skip the tests - you're done! [1]

[1] Slight exaggeration

Presenter Notes

Make It Obvious

-module(monitor).

-export([run/0]).

-define(URL, "http://localhost:8000").
-define(CHECK_DELAY, 5000).

run() ->
    check(),
    run_again().

check() ->
    handle_result(request(?URL)).

request(URL) ->
    httpc:request(URL).

handle_result(Result) ->
    io:format("~n~p~n", [Result]).

run_again() ->
    timer:sleep(Milliseconds),
    run().

Presenter Notes

Next: Fix result formatting

Presenter Notes

Function Clauses, Pattern Matching

handle_result({ok, {{_Protocol, 200, _Msg}, _Headers, _Body}}) ->
    io:format("~nSite is UP~n");
handle_result({ok, {{_Protocol, Code, Msg}, _Headers, _Body}}) ->
    io:format("~nSite is DOWN: ~p~n", [{Code, Msg}]);
handle_result({error, Err}) ->
    io:format("~nSite is DOWN: ~p~n", [Err]).

Function clauses separated with semi-colons
Headers are matched from top to bottom
Matched variables are bound (assigned) to the value
Btw, this code sucks - just look at it!

Presenter Notes

True Statement

Lazy people blame Erlang for hideous syntax

Presenter Notes

Much Better!

check() ->
    handle_status(url_status(?URL)).

url_status(URL) ->
    result_status(httpc:request(URL)).

result_status({ok, {{_Protocol, Code, Msg}, _Headers, _Body}}) ->
    {ok, {Code, Msg}};
result_status({error, Err}) ->
    {error, Err}.

handle_status({ok, {200, _Msg}}) ->
    handle_site_up();
handle_status({ok, {Code, Msg}}) ->
    handle_site_down({Code, Msg});
handle_status({error, Err}) ->
    handle_site_down(Err).

handle_site_up() ->
    e2_log:info("Site is UP").

handle_site_down(Err) ->
    e2_log:error("Site is DOWN: ~p", [Err]).

Presenter Notes

Next: Run check in background

Presenter Notes

Erlang Processes

Start with spawn or spawn_link
Isolated from other processes
- Separate heap
- Separate garbage collection life cycle
- All communication via messages
Monitor / trap exit
Similar semantically to OS process, much lighter

Presenter Notes

Simple change to `monitor.erl`

    -export([start/0]).

    start() -> spawn(fun run/0).

Spawn runs a function of arity 0 in a separate process
Returns a process ID (Pid)
fun run/0 is a reference to to run/0

Presenter Notes

Fun Facts About Erlang Processes

Readily available stats (current function, status, message queue, memory)
Traceable
Profile using etop

Presenter Notes

Next: Real notification

Presenter Notes

Third Party Erlang Libraries

Not as mature as other ecosystems (Python, Ruby, Perl, etc.)
Improving steadily
github (source code) emerging as a de facto distribution channel
But, you don't always need a library!

Presenter Notes

An Email Client

SMTP is simple! So I wanted something simple!
Poked around, nothing seemed simple
Then found this

http://21ccw.blogspot.com/2009/05/how-to-send-email-via-gmail-using.html

Presenter Notes

Super Simple SMTP

connect() ->
   {ok, Socket} = ssl:connect("smtp.gmail.com", 465,
                              [{active, false}], 1000),
   recv(Socket),
   send(Socket, "HELO localhost"),
   send(Socket, "AUTH LOGIN"),
   send(Socket, binary_to_list(base64:encode("me@gmail.com"))),
   send(Socket, binary_to_list(base64:encode("letmein"))),
   send(Socket, "MAIL FROM: <me@gmail.com>"),
   send(Socket, "RCPT TO: <you@mail.com>"),
   send(Socket, "DATA"),
   send_no_receive(Socket, "From: <me@gmail.com>"),
   send_no_receive(Socket, "To: <you@mail.com>"),
   send_no_receive(Socket, "Date: Tue, 20 Jun 2012 20:34:43 +0000"),
   send_no_receive(Socket, "Subject: Hi!"),
   send_no_receive(Socket, ""),
   send_no_receive(Socket, "This was sent from Erlang. So simple!"),
   send_no_receive(Socket, ""),
   send(Socket, "."),
   send(Socket, "QUIT"),
   ssl:close(Socket).

Presenter Notes

Next: Send email on DOWN

Presenter Notes

Yay - Everyone Gets A Sticker!

Presenter Notes

Real World Scenario

Running great for weeks!
Suddenly, stops working
You manually restart and it goes again, for a while...

Presenter Notes

Lurking Evil

Presenter Notes

When Evil Strikes!

The bug:

check() ->
    lurking_evil(),
    handle_status(url_status("http://localhost:8000/foo")).

lurking_evil() ->
    crash_if_true(randomly_true()).

The result:

2> monitor:start().
<0.57.0>

Site is UP

Site is UP

Site is UP
8>
=ERROR REPORT==== 5-Jul-2012::11:34:50 ===
Error in process <0.57.0> with exit value: {lurking_evil,[{monitor,crash_if_true,1},{monitor,check,0},{monitor,run,0}]}

Presenter Notes

Next: Fault tolerance

Presenter Notes

Recall, Erlang "System"

Presenter Notes

Basic App Structure

Presenter Notes

Process Crash

Presenter Notes

Process Restart

Presenter Notes

e2: Overview

http://e2project.org

Presenter Notes

Creating An App

$ cd $E2_HOME
$ make new-project appid=monitor appdir=~/monitor-app
...
$ cd ~/monitor-app
$ make
...
$ make shell
...
1> application:start(monitor).

=INFO REPORT==== 20-Jun-2012::01:12:18 ===
TODO: configure top-level processes for your app

e2 uses make to perform various commands
new-project creates a basic Erlang application
make builds the project
make shell runs the project in a shell

Presenter Notes

e2: Services and Tasks

Use a service to provide long running services
Use a task to run something, either once or repeatedly
Monitor check process is a classic task

Presenter Notes

monitor_check.erl

-module(monitor_check).

-behavior(e2_task).

-export([start_link/0, handle_task/1]).

start_link() ->
    e2_task:start_link(?MODULE, [], [{repeat, ?REPEAT_INTERVAL}]).

handle_task(State) ->
    handle_status(url_status(?URL)),
    {repeat, State}.

...

start_link/0 called by supervisor to start the task
handle_task/1 callback performs the work
Task repeats until terminated by supervisor

Presenter Notes

Erlang Behaviors

Declare that module fulfills a behavior contract
Compiler warns of any unexported behavior functions
That's it!

Presenter Notes

Using monitor_check in our app

-module(monitor_app).

-behavior(e2_application).

-export([init/0]).

init() ->
    {ok, [monitor_check]}.

init/0 returns a list of app process specs
Task started by calling monitor_check:start_link/0
By default, app processes are always restarted if they terminate

Presenter Notes

Start 'er up!

1> application:start(monitor).
ok
2>
=ERROR REPORT==== 20-Jun-2012::01:46:37 ===
** Generic server <0.40.0> terminating
** Last message in was timeout
** When Server state == {state,e2_task,
                           {state,monitor_check,[],false,undefined,5000},
                           '$task'}
** Reason for termination ==
** {noproc,
       {gen_server,call,
           [httpc_manager,
...

Presenter Notes

Whaaaa??

Presenter Notes

Recall, App Dependencies

Presenter Notes

We Require `inets`

src/monitor.app.src:

{application, monitor,
 [{description, "monitor app"},
  {vsn, "1"},
  {modules, []},
  {registered, []},
  {mod, {e2_application, [monitor_app]}},
  {env, []},
  {applications, [kernel, stdlib, inets]}]}.

Requires applications specified in applications list (last line)
Erlang won't start your app unless requires apps are started!

Presenter Notes

Dependency Fixed!

1> application:start(monitor).
{error,{not_started,inets}}
2> monitor:start().

=INFO REPORT==== 20-Jun-2012::02:00:47 ===
Started inets
ok
3>
Site is UP

Site is UP

monitor:start/0 is a short-cut for "start monitor along with all its dependencies"
When you start monitor, check task is started

Presenter Notes

Monitor App Process Hierarchy

Last process is our monitor_check task

Presenter Notes

Re-introduce our bug!

Site is UP

Site is UP

=ERROR REPORT==== 20-Jun-2012::02:04:57 ===
** Generic server <0.99.0> terminating
** Last message in was '$task'
** When Server state == {state,e2_task,
                               {state,monitor_check,[],false,1340175847478,
                                      5000},
                               undefined}
** Reason for termination ==
** {lurking_evil,[{monitor_check,crash_if_true,1,
...

Site is UP

Site is UP

Presenter Notes

Holy #@$! Erlang!

Presenter Notes

Next: Fix hard coding

Presenter Notes

We Hard Coded

Monitored URL
Check Interval
Email sender info
Email recipient info

Presenter Notes

App Config

Sample monitor config

[{monitor,
  [{smtp_account, "ceug.monitor@gmail.com"},
   {smtp_pwd, "sesame0620"},
   {check_interval, 10},
   {check_url, "http://localhost:8000/"},
   {notify_email, ["ceug.monitor@gmail.com"]}]}].

One file contains config for all apps, each in its own section
File named "xxxx.config"
Specify at VM startup using -config option

Presenter Notes

Specify config for make shell

$ make opts="-config priv/dev" shell
...
1> monitor:start().

INFO REPORT==== 20-Jun-2012::02:38:51 ===
Started inets
ok
2> application:get_all_env(monitor).
[{check_interval,10},
 {check_url,"http://localhost:8000/"},
 {smtp_pwd,"sesame0620"},
 {included_applications,[]},
 {notify_email,["ceug.monitor@gmail.com"]},
 {smtp_account,"ceug.monitor@gmail.com"}]

Presenter Notes

Using App Config

...

handle_task(State) ->
    handle_status(url_status(url())),
    {repeat, State}.

url() ->
    required_env(application:get_env(check_url), check_url).

required_env({ok, Val}, _Name) -> Val;
required_env(undefined, Name) -> error({required_env, Name}).

...

Replace hard-coded values with functions
Implement functions using application:get_value/1
Helpers to support optional and required values

Presenter Notes

Using Process State

...

init([]) ->
    {ok, #state{url=url()}}.

handle_task(#state{url=URL}=State) ->
    handle_status(url_status(URL)),
    {repeat, State}.
...

State is initialized in init/1
State is always provided to callbacks
Next state is returned by callbacks
By convention, represented by local state record

Presenter Notes

Records

Syntactic ~~sugar~~ sanity for tuples-as-data-structure
Easily the worst part of Erlang syntax
Most tameable with some effort

Presenter Notes

Using Records

-module(beer).
-export([new/1, rate/2, to_list/1]).

-record(beer, {name, rating}).

new(Name) -> #beer{name=Name}.

rate(Beer, Rating) -> Beer#beer{rating=Rating}.

to_list(#beer{name=Name, rating=Rating}) ->
    [{name, Name}, {rating, Rating}].

In the shell:

1> B = beer:new("Smuttynose Old Brown Dog").
{beer,"Smuttynose Old Brown Dog",undefined}.
2> B2 = beer:rate(B, "pretty damn good").
{beer,"Smuttynose Old Brown Dog","pretty damn good"}.
3> beer:to_list(B2).
[{name,"Smuttynose Old Brown Dog"},
 {rating,"pretty damn good"}]

Presenter Notes

Summary: Our Monitor

Solves a real problem
No error handling, except detecting DOWN
No tests
This will work well in any production environment!

Presenter Notes

Yeaaaah, we did it!

Presenter Notes

Things We Could Enhance

Multiple Checks
Multiple Notification Lists

Presenter Notes

Multiple Checks: Easy!

Provide a list of check URLs in config
Use a dedicated supervisor to start and monitor the checks

Presenter Notes

Erlang Supervisors

Responsible for starting, monitoring, and restarting processes
Life cycle managers for processes
We already used one with our app: the root supervisor
Unless you're an expert, run all of you processes under supervision!

Presenter Notes

`monitor_check_sup.erl`

-module(monitor_check_sup).

-behavior(e2_supervisor).

-export([start_link/0]).

start_link() ->
    e2_supervisor:start_link(?MODULE, checks()).

checks() ->
    check_specs(monitor:required_config(checks)).

check_specs(Checks) ->
    [{{monitor_check, start_link, [URL, Interval]}, [{id, Id}]}
     || {Id, URL, Interval} <- Checks].

Supervisor specializing in tasks
Reads list of checks from config
Uses list comprehension to map config to specs

Presenter Notes

Multiple Checks in Config

[{monitor,
  [{smtp_account, "ceug.monitor@gmail.com"},
   {smtp_pwd, "sesame0620"},
   {checks,
    [{foo, "http://localhost:8000/foo", 10},
     {bar, "http://localhost:8000/bar", 20}]},
   {notify_email, ["ceug.monitor@gmail.com"]}]}].

Presenter Notes

Next: Cleanup email notification

Presenter Notes

Option 1: In-Process

Presenter Notes

Option 2: Shared Service

Presenter Notes

When to Use a Service

Serialize access to shared resource (e.g. to memory/state, disks, database connections, etc.)
Maintain state across function calls

Presenter Notes

When to Not Use a Service

Whenever possible
Performance critical code
Need to run operations in parallel

Presenter Notes

Service Anatomy

Presenter Notes

Notify Service Stub

-module(monitor_notify).

-behavior(e2_service).

-export([start_link/0, send_msg/3]).
-export([handle_msg/3]).

%%%------------- Public API (client context) ------------

start_link() ->
    e2_service:start_link(?MODULE, [], [registered]).

send_msg(To, Subject, Body) ->
    e2_service:cast(?MODULE, {send, To, Subject, Body}).

%%%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%%%~~~~~~~~~~~~~~~~~~~~ Ether Barrier ~~~~~~~~~~~~~~~~~~~~
%%%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

%%%---------- Message Handlers (server context) ----------

handle_msg({send, To, Subject, Body}, _From, State) ->
    e2_log:info({todo_send_email, To, Subject, Body, State}),
    {noreply, State}.

Presenter Notes

Service API

Client Context

start_link starts the service
call sends a message and waits for a reply
cast sends a message and doesn't wait

Service Context

init used to optionally initialize state in service context
handle_msg handles messages sent by the client
terminate used to optionally handle process termination

Presenter Notes

Implement monitor_notify

Presenter Notes

Our Monitor!

Simple but effective
Somewhat flexible
Easy to enhance (i.e. do the next obvious thing)
Will keep checking until the Sun runs out of fuel [1]

[1] Or you turn your computer off

Presenter Notes

Wrapping Up

Presenter Notes

Erlang

Let's you build complex systems using simple pieces, less code
Encourages rigorous problem solving
Surprisingly productive
Works!

Presenter Notes

Crux of Erlang Design

Process isolation provides bulwarks throughout your system
"Let it crash" eliminates error handling code
Process supervision and restart provides fault tolerance without error handling

Presenter Notes

Going Forward

Identify a problem and just dive in!
Buy and read every Erlang book written
Ask questions on Erlang mailing lists (erlang-questions, ceug, etc.)
http://e2project.org/erlang.html has some useful links

Presenter Notes

Stickers! [1]

[1] I don't actually have stickers :(

Table of Contents	t
Exposé	ESC
Full screen slides	e
Presenter View	p
Source Files	s
Slide Numbers	n
Toggle screen blanking	b
Show/hide slide context	c
Notes	2
Help	h