I think it should be possible to use the SP.UI.PopoutMenu but the MSDN documentation is more then poor on this element and can't find any tutorial how to use it.
If I just try to open it like this nothing happens:
var anchorId = 'myAnchorId', menuId = 'myMenuDivId', iconId = '', anchorOpenCss = '', textDirection = 'ltr', closeIconUrl = '', isClustered = false, x = 500, y = 500, height = 500, width = 500; SP.UI.PopoutMenu.createPopoutMenuInstanceAndLaunch(anchorId, menuId, iconId, anchorOpenCss, textDirection, closeIconUrl, isClustered, x, y, height, width);
I have never used the PopoutMenu object before, but what you are pointing too is the Context Menu or the Edit Control Block (ECB) menu. I tried to come up with a working sample for the PopoutMenu, but you are right, its difficult to use. I did find this article on using a custom ecb menu in a sharepoint gridview if this helps: http://blogs.msdn.com/b/powlo/archive/2007/02/25/displaying-custom-data-through-sharepoint-lists-using-spgridview-and-spmenufield.aspx
Well, i might be too late for the party but i got it working on SharePoint 2013 There is still no real documentation to be found on the front end popoutmenu, but messing around with the back-end version i reverse engineered everything :)
Disclaimer: I haven't tested all of the possibilities with this PopoutMenu thingy, i'm not sure if the elements below have to be in said order with said tags, i'm almost sure they don't in fact, and you can have them in any other, tags, classes. But don't take my word for it.
This configuration just makes everything look nice.