Lookery Dev Blog

Building Voldemort read-only stores with Hadoop

Thu, 18 Jun 2009 10:27:54 -0400

Building Voldemort read-only stores with Hadoop:

At Lookery, we have been working very hard to transform most of our data processing tasks into batch-oriented workflows in order to deal with growth. For example, we were already using Hadoop to compute our index and data files for our largest database, but the process of serving that information took place over too many network hops (load balancers, reverse proxies and Amazon S3). Therefore, as soon as I learned that Project Voldemort supported offline building of distributed stores, I decided to try it and we’re now running it in production. Just visit the guest blog post over that Project Voldemort blog for full details.

Thoughts on "Efficient"

Tue, 17 Feb 2009 13:29:06 -0500

Interesting conversation at lunch yesterday. Elias and I had a different take on what it means for a dev team to be “efficient” (I’ll have to make a special post when the two of us find ourselves in total agreement on something ;-). I’m going to caricature what we were saying a bit, because I think it captures two things developers strive for:

Efficiency Definition 1: developers spend as little time as possible not coding (meaning, as few meetings as possible, as little interaction with ticket systems as possible, etc).

Efficiency Definition 2: developers spend as little time as possible coding features which no one needs

Those two aren’t exactly in direct conflict, but they do imply different tradeoffs. If you go for 2, you’re willing to hit a few meetings, jump through some ticket system hoops, because you live in abject fear of writing the wrong code. That’s where I find myself, at this point in my career. That’s part of why I don’t trust dev teams that romanticize overwork — it’s almost impossible to avoid writing the wrong code when you’re pulling 60 hour weeks, and working in constant low-grade fear.

-Dan M

Erlang/Eunit Test Structuring Idea

Thu, 29 Jan 2009 09:44:00 -0500

Nifty post from Adam Lindberg on the Erlang-Questions mailing list, about how he structures his eunit tests:

In myapp/src/foo.erl I put a small header file inclusion:

-ifdef(TEST).
-include("foo_test.hrl").
-endif.

And myapp/test/foo_tests.hrl looks like this:

-include_lib("eunit/include/eunit.hrl").

foo_test_() ->
   [?_assertEqual(ok, public_function())].

bar_test_() ->
   [?_assertEqual(ok, private_function())].

With the right compile options, you get tests only built into the code during dev, and you get to keep your tests in a separate file (but still have full access to the internal functions, which I find absolutely necessary).

-Dan M

My Own Private OTP

Wed, 28 Jan 2009 16:53:22 -0500

Elias and I figured out how to write a shell script which starts up a (locally) distributed erlang node, then makes a series of rpc:call()’s to get code:purge() and code:load_file() called in another, existing erlang node.

Why is this cool? Well, now we can specify a list of modules on the command line, and get those hot upgraded in a running server, with no downtime.

But, wait, there’s more: today, I figured out how to get fabric to sync the .beam files to our set of servers, and then run the module upgrade remotely on each of them.

Is it possible I’ve taken the first foolish steps towards reimplementing the whole OTP release process? Sure! But, for now, this is a pretty good simple/useful sweet spot:

fab prod upgrade:rotator_web,rotator_utils

-Dan M

One of Those Days...

Wed, 28 Jan 2009 15:34:28 -0500

… where everything you do seems to throw up some mysterious configuration error, and no sooner do you fix one, than you get caught up in another, and you find yourself saying things like “Holy f#%ing god why does rsync have so many goddamn options”, and you realize that people in the next cube (or, if, like me, you work from home, your blameless wife, sitting in the kitchen), may be disturbed by the loud obscenities.

Here’s what I’ve learned from today’s version of that show:

- Erlang cares very much who you start it as, if you’re starting it in distributed mode

Because it needs to read/write a cookie in $HOME. Otherwise it crashes, cryptically.

This interacts… interestingly with monit, which wants to start things as root, and then change the uid to something else. Finally got that working, but my god, it was an ugly morning.

-Dan M

OTP Release Mgmt -- Worth the Complexity?

Fri, 16 Jan 2009 10:54:11 -0500

Sort of an open question — anyone out there use the full, somewhat forbiddingly complex system for Erlang OTP releases? I finally think I at least understand it (thanks in no small part to Mitchell Hashimoto), but now that I do, I’m really not sure it’s worth it.

It seems like you put in place a bunch of nigglingly specific little config files, and then write a bunch of detailed specs for how code changes from one version to another. And that lets you do some nice upgrade/downgrade stuff (via further nigglingly specific “appup” files).

However, for our case, I think it maybe boils down to, either:

The code change is very simple, and I can make it happen via the `code:load_file()` and `code:purge()` functions without restarting the server.
The code change is complex, and I’ll just restart the server outright.

I mean, sure no 9 9’s uptime then, but I’m just not sure we need that (and if we do, we might be able to get it via some haproxy games instead).

The OTP system seems to me to take over a bit of what the OS does in Unix-land. In some ways, that’s good (the os-process-like Erlang processes are much better than shared-memory threads), but in other ways, now that I’m trying to manage a deployed server, it’s just, some things are a bit odd.

Anyone have any experience to share?

-Dan M

Very Nice OTP Examples/Tutorial

Wed, 14 Jan 2009 14:53:04 -0500

Meant to add this to the last post. One Mitchell Hashimoto has written up a very nice series of tutorials on how to use the various bits of OTP (gen_server, gen_fsm, etc), which is a very, very nice addition to the dry, example-free OTP design principles.

His site is:

http://spawnlink.com/

And the series of tutorials starts with this one:

http://spawnlink.com/articles/introduction-to-the-open-telecom-platform/

Very much worth checking out…

-Dan M

Back in Erlang Today...

Wed, 14 Jan 2009 14:47:50 -0500

…and, I have to say, for writing async/parallel code, it’s just awesome. I sometimes think I’m drinking the shiny new technology-flavored kool-aid when I talk about Erlang (which kool-aid I desperately try to avoid, personally), but then someone gives me a task like:

Write a chunk of code to load balance HTTP requests between two identical cache servers, failing over to one if the other goes down.

And as I write it in Erlang, it’s just such a perfect fit.

Oh, and for any regular readers of this blog, I actually caught myself today saying “Hey, records are working really well right here.” That’s right, records. I know. After all the abuse I’ve heaped on them. Built-in dicts are still very much missing from the langauge, don’t get me wrong, but records work pretty well for holding state inside gen_server loops.

-Dan M

The Absolute Most Useful Thing I've Done At My Job So Far

Fri, 09 Jan 2009 16:20:32 -0500

I think the most satsifying thing about being a programmer is when you manage to take something which is difficult, or boring, or stressful, and you get the computer to do it for you. I don’t know about you, but for me, well, pretty much nothing makes me happier. (side note: I think this is why I adore test-driven development — it lets me be lazy, without having that fear that someone later on will say “Um, Dan, why does your program make horribly obvious and embarassing error X?” I don’t like having to think about that stuff all the time. With tests, I don’t have to.)

Which brings me to today’s post. I spend a good bit of time hovering over a bunch of Hadoop Map/Reduce jobs. They form a directed, acyclic graph (natch), and we run them continually, on various repeating intervals (evey hour, every four hours, etc).

Elias did the original set up, and when I came on, I spent a while working on the code which launches each job, checking to see if its antecedents are ready, getting the output into the proper place in the HDFS tree, etc.

My single best day was when I realized that, with just a couple of easy python functions, I could get that code to (via the magic of graphviz), create an image of the dependency graph. And then, by checking to see how old the data in HDFS was, I could find late jobs, and color them yellow/red appropriately. Here’s an example from earlier today:

I can’t tell you how happy this makes me. One of our jobs (“audience_metrics”) had, in fact, died because of some odd strptime formatting issue (found that in the log, knew exactly where to look). Without the graph, I’m honestly not sure when I would have noticed this. Days at least. Likely it would have happened when one of our publishers asked us “Hey, how come my visitor counts haven’t updated in forever?”. Which is the kind of thing I hate.

Anyways, thank you graphviz. And yay for monitoring graphs.

-Dan M

Great Error Message

Fri, 09 Jan 2009 15:05:18 -0500

Tailing some log files, forgot to enter the filename, just entered ‘tail -f’, got this response:

tail: warning: following standard input indefinitely is ineffective

Love it. Dunno why. Just love it.

-Dan M

Graphviz Considered the Absolute Frickin' Bomb

Wed, 24 Dec 2008 10:06:33 -0500

Okay, so maybe all of you are on this tip already, but if not, I just have to briefly rave about graphviz, a project I adore. What is this graphviz, I hear you say? Well, it’s a library for creating graph images (graph visualizations == graphviz). Why does Dan love it so? Well, beyond the fact that its core functionality is way, way useful (pictures of graph data are something humans are pretty much built to parse), I love, love, love the interface.

What is this interface? Some elegant object-oriented system with bindings in every modern language, surely? Oh, no, it’s oh-so-much-lovelier than that. Graphviz defines a declarative mini-language for creating graphs, reads that from stdin, and writes the resulting images to stdout.

What? Like pic, tbl, eqn and all those ancient Unix tools? Exactly. And it works incredibly well. Easy to generate (if, say, you can create strings in your language of choice), easy to debug (write those strings to a temp file, read it with your text editor, muck with it until the resulting images look the way you want), easy to string together in pipelines, etc.

One of the central discoveries of my programming career has turned out to be: declarative mini-languages win (and when I say, discovery, I mean, not my discovery, but more my rediscovering what lots of people already knew). Kernighan and Pike are a gerat source for this, in the marvelous Practice of Programming.

-Dan M

Setuptools Considered Harmful

Wed, 17 Dec 2008 11:34:23 -0500

Setuptools Considered Harmful:

You know what I love? When you have a vague feeling of loathing for an application/library/framework, but don’t fully understand the problem it solves well enough to really justify your hatred, but then you read someone’s excellent description of why you were right to hate said application/library/framework.

I hate setuptools. There, I’ve said it. Now James Bennet backs me up.

-Dan M

Hey, how's dumbo running a M/R combine? Uh, well...

Tue, 16 Dec 2008 15:36:01 -0500

So, debugging sluggish Hadoop M/R jobs is like a whole new world of things I’m trying to learn. Which, y’know, good and all, but, whoa, there’s some challenges.

Anyways, here are interesting things I learned today:

For procps-based top, the %st CPU state means “stolen from the virtual machine”

Which is to say, the xen hypervisor is using some of your cycles (either because you’re idle, or because you’re only getting a slice of the machine — I think which it is depends on how accurate your copy of top is)

You know what else is fun: the man page for top didn’t admit there was such a thing as an %st state. Good times.

dumbo, the python lib for writing Hadoop streaming jobs, is both very nice and horribly under-documented

And when I say under-documented, I mean source code with not only absolutely no comments, but also a… distinctive python style which is not altogether read-friendly.

And thus: when I was first learning how dumbo works, I was somewhat surprised to see that it offered the ability to write combine steps in python. Because all the Hadoop streaming docs said you couldn’t use combine with streaming.

How does dumbo do this? Well.. by holding the entire output of the map stage in memory, sorting it itself, and only then passing lines back to java.

Which, if, say, your data nodes are memory bound, has the potential to bring your entire system to its swapping knees. Which is what happened to me this afternoon.

So, um, beware of combine steps in dumbo, I guess.

-Dan M

Definitely Rec "Release It!"

Wed, 10 Dec 2008 11:36:03 -0500

Most of the way through the Michael Nygaard book I ref’d in the previous post, and I would say, totally worth picking up. Beyond the just fantastic stories of software disasters, (which, apparently, I can’t get enough of), the main thing I think it has to offer is insight into the vast array of things which can go wrong. He does sketch in some useful, design-pattern-ish things you can put in place to help save yourself, but, mostly he’s trying to communicate the idea that you have to program with assumptions of failure. And the vivid stories of all the different kinds of failure are, I suspect, a really good way to etch that in your mind.

There’s no silver bullet for this kind of thing, but I will say, Erlang is a fantastic fit for the kind of approach he recommends. If you use Erlang, will your software magically work in the face of component failure? Good god, no. But, will there be a much clearer path to using timeouts everywhere, monitoring and supervising bits of the application, etc? Yes.

-Dan M

Two Books Which Seem To Have Been Written Exactly For My Job

Fri, 05 Dec 2008 10:57:29 -0500

Followed a rec from Steve Vinoski (whose blog is truly excellent), and picked up “Release It!”, by Michael Nygaard. Just started it, but, man, it looks like he wrote it for someone in exactly the situation we’re in. He’s got over a decade’s worth of experience in releasing/monitoring/maintaining/extending big, critically-up systems, and has stories to tell, recommendations to make, etc. Very excited to burn through that and see what he has to say.

(side note: I have a career-long distrust of the title “architect” in the software biz, since it all too often seems to mean someone who believes what vendors say about their byzantinely complex systems, and doesn’t hesitate to put together one of those unspeakably terrible “best-practices” amalgamations of XML files, Java get/set methods, and hundreds of hours of junior programmer busywork. That said, I get the sense that Michael Nygaard may be the other kind of software architect, one who really deserves the title. Which would be great).

Also got O’Reilly’s Programming Collective Intelligence, which has lots of math and ideas for finding information in huge sets of data. Which, y’know, we’ve so totally got. A bit slow to get going, because, though I love me some math, I also happen to intensely dislike wildly overstatated claims of the utility of collaborative filtering and/or machine learning. Both of which have value, but which are often conflated, by an innocuous-seeming rhetorical trick, with the kinds of intelligence humans evidence when given a big mass of unstructured data. Which is so absurd and irritating I can’t write about it right now (it sums up everything I find disappointing about AI, which, instead of seeking to fully understand the astonishing depth and richness of actual intelligence, seems to be satisfied in making systems which appear to be intelligent—but those appearances are hugely deceptive, because humans are overdetermined to see things as intelligent. In other words, AI is all hugely cheaty, and they don’t admit that. See Douglas Hofstadter’s writing on this in, e.g. Fluid Concepts and Creative Analogies, or Pinker’s discussions of common sense in How the Mind Works. Okay, maybe I did have time to write about it.)

Anyways, I think there will be valuable stuff within there, and I’ll just have to hear little diatribes in my head while I read it.

Will report back once I’ve made more progress.

-Dan M

Time to Sleep the Sleep of the Just

Thu, 04 Dec 2008 00:51:01 -0500

You know that feeling you have when you’ve taken a sprawling, complex chunk of code, so complex you can barely understand it, but somehow, after a binge of work, you’ve rewritten it so that it makes sense, and necessary new features can now be added?

That’s how I’m feeling right now.

Also, tired.

Night all…

-Dan M

Lookery on average is 65% faster than Google

Mon, 01 Dec 2008 20:57:00 -0500

Lookery supports millions of daily users worldwide across thousands of publishers every day. We know our publishers care a lot about how quickly their pages load — we certainly do.

To minimize latency and eliminate page load delays, we take great care that any code that Lookery asks publishers and partners to put on their sites, blogs, etc is very lightweight and speedy.

Part of our process is to actively and very closely monitor and benchmark ourselves (and others) using third party performance monitoring services. For monitoring our response times we’ve been using Pingdom for the past few months. Pingdom monitors our servers and infrastructure 24/7, all year long.

You can view Pingdom’s public report card for Lookery here, and Google Analytics here.

Yes, according to Pingdom, Lookery on average is 65% faster than Google Analytics!

This makes all of us at Lookery very happy for all sorts of reasons — most of all because our hard work is paying off and resulting in us providing world class performance to all our publishers 24/7/365.

- Jay M.

Erlang Thoughts For Now

Mon, 17 Nov 2008 17:52:35 -0500

Well, we’re close to launching the thing I’ve been writing in Erlang, and I’m thus moving on to doing some work with Hadoop (my job rocks, is what I’m saying here ;-).

Few wrap-up thoughts as I go:

Overall, very happy with the experience. If you’ve got interesting concurrency, distributed programming and/or high availability problems, I think Erlang is very, very much worth your time. It makes things which are profoundly difficult in other languages surprisingly clean and easy.

Wondered if I would find the no-mutable-state thing an uncomfortable straitjacket, but I pretty much didn’t. (I’m probably somewhat atypical, to be fair, since Lisp was a formative early language for me, and functional thinking has always felt pretty natural — but I do use mutable state in python all the time, quite happily). If you need mutable state, you can get it by writing a little separate gen_server module, which is very easy to do (and then, by magic, you can start to take advantage of multiple cores more easily). Or you can use ets.

In this (and many other, important ways ;-), Erlang is nothing like Haskell, where you seem to need some kind of dispensation from the church to work with mutable state (if you want to hear Dan rant about Haskell, just say the word).

Overall, I think Erlang has two of the most important virtues a programming language can have: it feels simple, and it’s been used heavily in the real world.

Few specific hits and misses:

Hit: eunit. I can’t say enough good stuff about eunit. My learning arc was much like my learning arc with python — everything was easy to get started, and then I gradually realized there was tremendous power and flexibility under the hood. Go ProcessOne guys, go. And it’s also an excellent example of how you can write DSL-style code. Just, really great stuff.

Miss: strings. Arg. Lots of bits of my code have to obsessively check if they’ve been passed a binary or a list (= string), and if I forget to do that, it throws up all over the place. This did not get less frustrating with time.

Hit: mochiweb. Although there is, like, no documentation, it’s a very nice set of modules for building your own custom web server applications. Which is what we were doing.

Minor Miss: the strange mix of line termination/separator syntax. Thanks to emacs, this wasn’t too much of a pain, but it wasn’t great. But pretty minor.

Slightly More Major Miss: the fact that you can’t call methods on objects. As in, to look up a value in a proplist, you’ve got to call proplists:get_value(Key, Proplist). Instead of, say, Proplist:get(Key). It ends up being a lot of extra typing, you have to remember the argument order, and it makes it very hard to just swap in a new data structure.

Even More Major Miss: records. Man how they suck. If you could add a built-in functional mapping/dict type, with a literal syntax, it would be such a huge win.

Hit: OTP. Both the fairly comprehensive library (see real-world use above), and also the system for putting together applications.

Miss: lack of documentation on OTP in Joe Armstrong’s otherwise very solid book. He walks you through setting up a basic OTP app in late chapters, but doesn’t touch on a lot of key facets of the whole powerful but somewhat forbiddingly complex system. I’ve downloaded the OTP Design Principles docs, but it’s not always easy to figure out how people actually use those design principles. E.g. do you really need separate boot files? When? What’s involved in rolling out a release? How do I best manage / supervise apps on multiple nodes? Etc. Maybe I’ll learn more about all this as we deploy the system I’ve been writing, but I would sure be happier if Joe Armstrong had been able to write 2-4 chapters more on how to use OTP in the real world.

Hit: the Erlang Questions mailing list. Lots of responsiveness from impressive Erlang guys, both within Ericsson and outside of it. Feels like an early, small community, if that makes sense. Love that feel.

Anyways, I’ll leave it there for now. Had a great time so far, look forward to doing more. If you have any specific questions, feel free to leave ‘em in the comments…

-Dan M

Hey, Where's That Function Call?

Wed, 05 Nov 2008 16:59:46 -0500

I had a funny moment as I was just looking at an Erlang stacktrace, saying to myself “But, why isn’t it showing the function that called foldl?”, and then realizing that, of course it can’t show that function, Erlang is, y’know, tail recursive, so the function which ended with the call to foldl, was popped right off the stack.

I mean, duh. But somehow, this was a bit of a “we’re not in Kansas anymore” moment, even though every looping fun I’ve written is tail recursive, so it’s not like I shouldn’t have known this would happen.

I guess what I’m saying is “In Erlang, your stack frame can just go away! It’s crazy, I tell you! Crazy!”.

-Dan M

Testing / Mochiweb / Stacktraces

Wed, 05 Nov 2008 11:17:37 -0500

So, I’m somewhat test-obsessed (because I accept my own prediliction for making dumb mistakes over and over, and I like having the computer keep an eye out for me. Also, it makes programming a lot more fun).

With a mochiweb app, for certain kinds of testing, I want to launch an instance of my mw server, hit it with an http request, and check the responses. Fortunately, Erlang’s lightweight processes make this very easy. Unfortunately, when I kept things simple, I wasn’t seeing any stack traces when the mw server failed—the outer process, which had launched the mw server under test, would just get a crypic connection closed message, and that was it. And, ‘cause I do like to test as I go, I see failures all the time.

I asked on the mw list about this, and someone suggested starting erlang with -boot sasl, which gives it the OTP infrastructure, which does capture stack traces. But it also produces an incredibly verbose output, making the stack trace very hard to find. And, I couldn’t figure out how to get Emacs, which has a fantastic Erlang compile/shell feature, to launch erlang with -boot sasl.

I was beginning to think of really complex solutions involving somehow attaching the shell to the server process, when I suddenly realized there was a simple solution I was missing: I could just wrap my server handler in a catch, use eunit’s debugFmt to print out the stack trace, and I’d be all set. And it worked beautifully. Here’s the code. In my test function, I took:

    RotatorHandler = 
        fun (Req) ->
                handle_rotator(Req, {RuleUrlTmpl, ProfileUrlTmpl})
        end,

And changed it to:

    RotatorHandler = 
        fun (Req) ->
                try 
                    handle_rotator(Req, {RuleUrlTmpl, ProfileUrlTmpl})
                catch Class:Exception -> 
                        ?debugFmt("~p", [[{Class,Exception} |erlang:get_stacktrace()]]),
                        throw({Class,Exception})
                end
        end,

And now it produces a simple stack trace on failure. Woo, woo!

-Dan M

Lookery Dev Blog

Building Voldemort read-only stores with Hadoop

Thoughts on "Efficient"

Erlang/Eunit Test Structuring Idea

My Own Private OTP

One of *Those* Days...

OTP Release Mgmt -- Worth the Complexity?

Very Nice OTP Examples/Tutorial

Back in Erlang Today...

The Absolute Most Useful Thing I've Done At My Job So Far

Great Error Message

Graphviz Considered the Absolute Frickin' Bomb

Setuptools Considered Harmful

Hey, how's dumbo running a M/R combine? Uh, well...

Definitely Rec "Release It!"

Two Books Which Seem To Have Been Written Exactly For My Job

Time to Sleep the Sleep of the Just

Lookery on average is 65% faster than Google

Erlang Thoughts For Now

Hey, Where's That Function Call?

Testing / Mochiweb / Stacktraces

One of Those Days...