So you want to use OpenSocial gadgets on your site, Google FriendConnect isn’t the right choice for you since you want to leverage your own social graph, but your site isn’t written in PHP or Java, so how do you use Apache Shindig then?

For this scenario we’ve created the meta-data interface, it allows you to post a gadget URL to it, and it will return a JSON structure with all the meta data of the gadget that you can use to build your app gallery, user preferences UI and create the iframe’s in your site that point to Apache Shindig (which will render the gadget for you). The quickest way to check out how the meta data interface works is by loading up the sample-metadata.html sample, and inspecting the requests using something like firebug, here’s a live example:
http://modules.partuza.nl/gadgets/files/container/sample-metadata.html

With that you have all the required info to build your site’s UI, however you also need to create the proper security tokens, and link shindig to your data source:

Security Tokens.

The gadget get’s it owner id / viewer id / app id / container string / module id from the security token, in a typical production setup this would be an encrypted blob generated by the container (ie ‘the social website’), and put on the gadget’s iframe with a &st=<encrypted blob>. Encryption ensures that that data can’t be tampered with so that people can’t spoof their user id’s etc.

Now both java and php shindig both use the same methodigy to generate their tokens (see shindig/php/src/common/{BasicBlobCrypter,BasicSecurityToken,BasicSecurityTokenDecoder}.php for example), so you really have two options, either implement the same crypto logic / format (it’s basically a text string, with entries seperated by :’s, and aes128 encryption with a shared secret) in your language of choice, or create your own crypto and implement a PHP class that can decrypt it (and tell shindig to use that new class using the ‘security_token_signer‘ and ‘security_token‘ configuration keys, similar configuration can be found in java shindig in one of it’s .xml config files), The first option is probably the simplest solution as long as aes128 is available in your language of choice.

There’s a bit more information about how security tokens work and how to implement them in this blog post:
http://www.chabotc.com/partuza/about-partuza-and-shindig-security-tokens/

Social Data.

Now java and php differ slightly here, with php you use the same configuration as I mentioned above to tell shindig which classes to use, and with java you’d use Guice to inject the right classes, but the basics are the same: You’ll need to somehow get the social data requests that happen in shindig to your data back-end.

Now when you are running php or java already this is pretty simple since you can use that code directly, however in this situation you really have two choices, either you implement the DB logic in PHP or Java (for PHP, you can use Partuza’s (an example opensocial implementation) service implementation as example: http://code.google.com/p/partuza/source/browse/trunk/Shindig/PartuzaService.php and http://code.google.com/p/partuza/source/browse/trunk/Shindig/PartuzaDbFetcher.php), or you can implement an RPC method (using something like json-rpc) where shindig relays all social data calls to your back-end.

OpenSocial and thus Shindig has 4 data call types, People, Activities, AppData and Messages; Now the spec states that the only must-have of those is the People interface, and you could return a NOT_IMPLEMENTED error code on every other call type.. However if you want to run the typical OpenSocial app, it’s advisable to also support the Activities and AppData interfaces.. Messages is a nice to have, but in practice most social sites don’t support this so it’s not a big issue at all if you don’t.

If your using or planning to use php-shindig in production, your probably wondering what knobs can be twiddled to get the highest possible performance out of it; So here’s some tips on how to make it go just that bit faster.

The first thing to look at is the general PHP performance on your server. Out of the box PHP is very poorly equipped for high traffic websites, and chances are you already have a few of these optimizations in place, combined these tricks will more then double the performance of your shindig deployment.

To be able to give some semi-meaningful measurements I’ll be using the QPS (queries per second) of how fast php-shindig can render the OpenSocial Dev App (OSDA) on my workstation, which has a single Quad Core 2 Duo @ 3Ghz and a single SATA II disk, so a standard of the rack server should be able to do a bit better then the results I’m getting here. The base line of a plain outo-of-the-box php and shindig configuration is ~ 210 QPS.

Opcode Cache

Perhaps the most important tip for php performance is to install an opcode cache.. PHP is still an interpreted language, and that means that in it’s default configuration, on every page hit, it will read your PHP code and interpret it into opcodes. An op(eration)code is the instruction that the Zend engine actually performs, and this constantly interpreting will take most of your server’s resources! The solution is to install an opcode cache that caches these compiled instructions in memory, so that your webserver can focus on actually executing the code. My personal favorite is the Alternative PHP Cache (APC)

Using APC the QPS on my workstation goes from 210 to 385, a pretty impressive performance increase!

PHP FatMM

The next tip is to use the php-fatmm patch, the fat memory manager patch changes the way that the PHP engine allocates memory, instead of (de/)allocating memory on the fly it will pre-allocate a configured amount of memory, and only if that runs out will it allocate more; The end result is that it can save PHP having to do thousands of alloc()’s, which means it can spend more time doing useful stuff This does mean your server process will use more memory, and I wouldn’t advice to use this on a server that also serves static content or very simple php scripts, but for a server that is dedicated to php-shindig, this is just what the doctor ordered.

Re-benchmarking our OSDA rendering using both APC and php-fatmm brings us to 410 QPS, while not as earth-shattering as APC, still a very nice improvement!

Disk IO – Caching back-ends to use

Now with the basic PHP bottle-necks out of the way, lets focus on the other thing that will bring a server to it’s knees: Disk IO. Harddisks are the slowest component in your server and having a lot of reads and writes happening will cause your CPU’s just just spin idly in their sockets waiting for data to appear.

The highest impact change here is to go from a disk based cache in php-shindig (the CacheFile class which is the default configured cache back-end since it works on every platform/configuration) to a memory based caching class. Now there’s two distinct types of cached information in php-shindig, the one is the ‘feature_cache’, where it stores the processed (and if configured, minimized) feature information.. parsing hundreds of little xml and javascript files is a very expensive operation so caching this adds a lot to the efficiency of php-shindig. The other cache type is the general ‘data_cache’, which is used to cache all remote resources such as the gadget xml files, message bundles, all proxied files, etc.

The features information doesn’t take so much space to store, and as such using a close-to-home cache for the feature_cache like the CacheApc class is the best solution, The data cache on the other hand will hold  many thousands of files, so will be considerably larger to store.. as such using a shared cache between your shindig servers is much more efficient, so using CacheMemcache is the optimum solutions there. So by changing the following bits:

'data_cache' => 'CacheMemcache',
'feature_cache' => 'CacheApc',


and rerunning the benchmark brings the total QPS to 435.

Disk IO – PHP Shindig configuration

The last bit of tweaking we can do is to disable the file checking, php-shindig has been coded to fail gracefully and check everything rigorously however that does cause some extra disk IO, however as long you feel certain you won’t go and delete files from your production systems, it’s pretty safe to turn this off, this is done by setting ‘check_file_exists’ to false in the configuration. We can also save some disk IO by changing the relative-path-resolution in the configuration to absolute paths, so change any paths like “realpath(dirname(__FILE__) . ‘/..’) . ‘/’” to their actual locations (like “/var/www/html/shindig/”). And setting ‘debug’ to false removes the last few of these checks. So in the configuration we set:

'debug' => false,
'check_file_exists' => false,
'base_path' => '/var/www/html/shindig/',
'features_path' => '/var/www/html/shindig/features',
'container_path' => '/var/www/html/shindig/config',
'javascript_path' => '/var/www/html/shindig/javascript',
'private_key_file' => '/var/www/html/shindig/php/certs/private.key',
'public_key_file' => '/var/www/html/shindig/certs/public.crt',

Conclusion

With all these measures in place, the benchmark now shows us doing some 450 QPS, more then doubling the 210 QPS we started out with, quite a satisfying result if you consider this means we end up transferring more then 45 megabyte/sec over the network at that point.

Now for the real speed freaks, all these results are based on the 1.0.x release tree from the svn repo, which is what you should be using since the trunk is seeing heavy development for the upcoming 0.9 changes, however the trunk has a re-factored gadget renderer that is a ~ 20-30% faster then the current release, so more performance goodness will be coming your way once the trunk stabilizes and becomes release ready.

If your like me your probably already exploring what FriendConnect is all about and in what interesting ways you can break^H^H use it.

Now remember the OpenSocial Dev App (or OSDA as we lovingly call it)? If not you should! It’s an incredibly nifty application that allows you to type in OpenSocial javascript directly into an edit box and execute it on the fly, making it a awesome way to poke at FriendConnect.

With it you can explore how it exposes the Owner (which is the site), Owner friends (the members of the site), the Viewer (that’s you! unless your not signed in, in which case you’ll get a lovely “badRequest: Cannot ask for viewer when anonymous” error as it should do), viewer friends, etc.

Also you see what profile details are available through GFC by going to the canvas view (click on the canvas tab), and go to the “Data Viewer” tab, select one of the options like “Viewer”, and click “Show details” to see a full dump of the person object with all it’s available fields.

To add this to your FC enabled site, goto http://www.google.com/friendconnect select “Social Gadgets”, “Custom Gadget >” and enter: “http://osda.appspot.com/gadget/osda-0.8.xml” as the URL and copy and paste the generated html into your site.

In my case I’ve added the generated HTML to a wordpress page (yes that is possible), and presto we have a OSDA running on GFC on a wordpress site! Ah I love it when you can combine technologies to make something cool happen

See http://www.chabotc.com/osda/ for my OSDA gadget on this GFC enabled site.

Happy hacking!

Since a few people had problems figuring out how security tokens work in and between Shindig and Partuza, I thought it might be useful to write up a quick post that details how this works.

So to start off, the security token is generated in Partuza in Application/Views/gadget/gadget.php :

A bit of background: php-shindig expectes an encrypted and encoded security token, since this way you can use the cryptography to be sure that the owner and viewer ID's are not faked, a very important assumption if you don't want people to be able to spoof identities (and do naughty things like setting someone else's status update to 'i'm a big fan of poodles' (make up your own examples to get an idea of the true impact of this)).

However the out-of-the-box html samplecontainer (shindig/javascript/samplecontainer/*) uses plain text tokens, so there's support build in for this as well, but it should be turned off *as soon as possible* since it makes for a completely non-secure situation.

So Shindig does expect a "owner:viewer:appId:domain:url:modId" type token, but it can consume it in either plain text, or encrypted form.

What partuza does is generate an *encrypted* security token and it does so using the password phrases listed in partuza/html/config.php:

The shindig receives the secure token, and decodes it with it's password phrases (shindig/php/config/container.php) :

Do note that these have to be the same on both ends, otherwise it won't be able to decode the token into anything sensible, and you'll get an error code returned.

After you have this working, make sure to disable the plain text security tokens, since that's a huge wide open security hazard waiting to happen, so change this to false (in shindig/php/config/container.php)

Now the last thing that is interesting to know, is that all the classes in php-shindig that might be different between site specific implementations are completely configurable, and we provide basic examples (which are easily identified by their Basic*.php naming) which are production ready and actually see massive production usage too, but if you want to implement your own, so that you could for instance add custom fields to it, simply point the configuration to your own class, and off you go:

shindig/php/config/container.php:

That rounds up our security token tour for the day, if there's anything that I missed out on or something the shindig community can help you with, you can find us on the shindig-dev@incubator.apache.org mailing list.

This was to good not to share, Scroobl’s awesome take on Google Friend Connect vs walled gardens:

(click to view full size)