New Hotfix for Office Client applications has been released to fully integrate uploaded documents in Forms based authentication (FBA) sharepoint sites using office client applications.
Before this hotfix, we couldn’t open sharepoint office files using client office applications in FBA SharePoint sites because the client can’t authenticate the user in FBA SharePoint Sites.
Now with this hotfix, the client will be prompted with a page to enter the username and password, then the request will be sent to the sharepoint site to authenticate the user and reply with a response contains the requested document.
Other than this hotfix you have to go on the FBA SharePoint settings sites and set enable client integration check box.
Read More :
Hope this helps.