The fastest, cleanest way to get rid of the parts of WordPress you don't need.

Disable native widgets, clean head tag from RSS, RSD, WLW Manifest links, disable XML-RPC, cleanup admin panel from columns, metaboxes, menu items.