Logging is an obvious requirement when it comes to being able to debug non-trivial systems. We’ve been thinking a lot about logging, thanks to the large-scale, distributed nature of the Zolodeck architecture. Unfortunately, when logging larger Clojure data-structures, I often find some kinds of log statements a bit hard to decipher. For instance, consider a map m that looked like this:
When you log things like m (shown here with println for simplicity), you may end up needing to understand this:
Aaugh, look at that second line! Where does the data-structure begin and end? What is nested, and what’s top-level? And this problem gets progressively worse as the size and nested-ness of such data-structures grow. I wrote this following function to help alleviate some of the pain:
Remember to include clojure.pprint. And here’s how you use it:
That’s it, really. Not a big deal, not a particularly clever function. But it’s much better to see this structured and formatted log statement when you’re poring over log files in the middle of the night.
Just note that you want to use this sparingly. I first modified things to make ALL log statements automatically wrap everything being logged with pp-str: it immediately halved the performance of everything. pp-str isn’t cheap (actually, pprint isn’t cheap). So use with caution, where you really need it!
Now go sign-up for Zolodeck!
There is a spectrum of productivity when it comes to programming languages. I don’t really care to argue how much more productive dynamic languages are… but for those who buy that premise and want to learn a hyper-productive language, Clojure is a good choice. And for someone who has a Java background, the choice Clojure becomes the best one. Here’s why:
- Knowing Java – obviously useful: class-paths, class loaders, constructors, methods, static methods, standard libraries, jar files, etc. etc.
- Understanding of the JVM – heap, garbage collection, perm-gen space, debugging, profiling, performance tuning, etc.
- The Java library ecosystem – what logging framework to use? what web-server? database drivers? And on and on….
- The Maven situation – sometimes you have to know what’s going on underneath lein
- Understanding of how to structure large code-bases – Clojure codebases also grow
- OO Analysis and Design – similar to figuring out what functions go where
I’m sure there’s a lot more here, and I’ll elaborate on a few of these in future blog posts.
I’ve not used Java itself in a fairly long time (we’re using Clojure for Zolodeck). Even so, I’m getting a bit tired of some folks looking down on Java devs, when I’ve seen so many Clojure programmers struggle from not understanding the Java landscape.
So, hey Java Devs! Given that there are so many good reasons to learn Clojure – it’s a modern LISP with a full macro system, it’s a functional programming language, it has concurrency semantics, it sits on the JVM and has access to all those libraries, it makes a lot of sense for you to look at it. And if you’re already looking at something more dynamic than Java itself (say Groovy, or JRuby, or something similar), why not just take that extra step to something truly amazing? Especially when you have such an incredible advantage (your knowledge of the Java ecosystem) on your side already?
My talk at The Strange Loop conference this year was recorded and is now available at InfoQ. I talk about why we’re using Datomic, Storm, and Clojure for our backend on the Zolodeck project.
Let me know what you think!
Here’s another useful function I keep around:
Everyone knows what map does, and what concat does. And what mapcat does.
The function definition for pmapcat above, does what mapcat does, except that by using pmap underneath, it does so in parallel. The semantics are a bit different:
First off, the first parameter is called batches (and not, say, coll, for collection). This means that instead of passing in a simple collection of items, you have to pass in a collection of collections, where each is a batch of items.
Correspondingly, the parameter f is the function that will be applied not to each item, but to each batch of items.
Usage of this might look something like this:
One thing to remember is that pmap uses the Clojure send-off pool to do it’s thing, so the usual caveats will apply wrt to how f should behave.
I kept using an extra line of code for this, so I decided to create the following function:
Another extra line of code can similarly be removed using this function:
Obviously, the raw forms (i.e. using doseq or map) can be far more powerful when used with more arguments. Still, these simple versions cover 99.9% of my use-cases.
I keep both these (and a few more) in a handy utils.clojure namespace I created for just such functions.
Alan Perlis once said: A Lisp programmer knows the value of everything, but the cost of nothing.
I re-discovered this maxim this past week.
As many of you may know, we’re using Clojure, Datomic, and Storm to build Zolodeck. (I’ve described my ideal tech stack here). I’m quite excited about the leverage these technologies can provide. And I’m a big believer in getting something to work whichever way I can, as fast as I can, and then worrying about performance and so on. I never want to fall under the evil of premature optimization and all that… In fact, on this project, I keep telling my colleague (and everyone else who listens) how awesome (and fast) Datomic is, and how its built-in cache will make us stop worrying about database calls.
A function I wrote (that does some fairly involved computation involving relationship graphs and so on) was taking 910 seconds to complete. Yes, more than 15 minutes. Of course, I immediately suspected the database calls, thinking my enthusiasm was somehow misplaced or that I didn’t really understand the costs. As it turned out, Datomic is plenty fast. And my algorithm was naive and basically sucked… I had knowingly glossed over a lot of functions that weren’t exactly performant, and when called within an intensive set of tight loops, they added up fast.
After profiling with Yourkit, I was able to bring down the time to about 900 ms. At nearly a second, this is still quite an expensive call, but certainly less so than when it was ~ 1000x slower earlier.
I relearnt that tools are great and can help in many ways, just not in making up for my stupidity
While working with the new Datomic database, we’re having to get used to using ns-qualified keywords in our maps. To make day-1 go faster, I wrote these versions of the demonic insert and load functions:
The idea was to use them as follows (for instance):
where datomic-key->regular-key is something like:
The idea behind all this was that a map like:
Could become something like:
This initially seemed to be more usable and familiar. However, we’ve since moved away from it, seeing some value in having keys that are indeed namespaced to the entity they belong to (for example user entity has keys called :user/first-name and :user/last-name).
Just thought we’d share.
I am working on a project (more details about this later) that integrates with social networks. To start with I am integrating with facebook. In my code only facebook.gateway namespace will be directly calling facebook graph API. Rest of my code will be dependent on facebook.gateway to interact with facebook API. I believe in automated unit and integration testing.Facebook allows you to create test users and make them friends. You can treat facebook test users as normal users and call all facebook API against them. I want to write facebook integration tests some what like this,
- In facebook
- jack = create-test-user “jack”
- jill = create-test-user “jill”
- mary = create-test-user “mary”
- login-as jack
- become-friend-of jill
- become-friend-of mary
- Verify whether my facebook.gateway/friends-list function returns two users
So I started a project called clj-social-lab. It is a clojure library to help you do integration tests with social networks. To start with, this library only integrates with facebook.
Currently you can ,
- Create Facebook Test Users
- Delete Facebook Test Users
- Delete All Facebook Test Users
- Make Facebook Test Users friends
This library also comes with some testing utilities that make it easy for you to write integration level tests.
For example, this is how my facebook.gateway/friend-list test looks like with clj-social-lab,
in-facebook-lab : This macros takes care of deleting all test users that are created as part of this test. So you do not end up with too many facebook test users. In fact, facebook has a limit of 500 test users per application.
This is just the start. I will be adding more functionalities when need arises in my project or when there is enough demand for a functionality.
Feedback is appreciated.
Announcing a new blog: blog.zolodeck.com. Just wrote the first post on my work with Datomic. I’ve put some of it into a project called demonic, and hopefully, you’ll find it of some use!
I’ve been writing some code to work with Datomic, and thought some of this might be useful to others. So I’ve put it into a small utility project called demonic. There are a few concepts in demonic:
The first is that of a demarcation. A demarcation is kind of a dynamic binding, within which, all datomic transactions are held until the end. When the scope exits, all the datomic transactions are committed at once. This is useful in a scenario where you’re updating multiple entities in a single request (say), and you want them all to happen or you want them all to roll-back. Here’s how you’d use it:
The good thing is that this makes it easy to write cleaner tests. There are two macros that help, namely demonictest and demonic-testing, which are respectively like deftest and testing. You could use them like this:
As you can see, there are several CRUD operations provided by demonic, and in order to get the above benefits (of demarcations and testability), you need to only go through these functions (and not directly call the datomic functions). Here are these basic functions:
- demonic/insert – accepts a map, if it doesn’t contain a :db/id key, an insertion occurs, else an update will occur
- demonic/load-entity – accepts a datomic entity id, and loads the associated entity
- demonic/delete – accepts a datomic entity id, and deletes the associated entity (and all references to it)
- demonic/run-query – accepts a datomic query, and any other data-sources, and executes it against the current snapshot of the db (within the demarcation)
By the way, there’s another helper function for when you’re building web-apps using Compojure:
- demonic/wrap-demarcation – sets up a demonic demarcation for the web-request
So these are a few things I’ve got in there right now. I’m also working on making it easy to create and maintain datomic schemas. I’ll write about that another time, once it is a bit more baked.