Partially I have solved the problem.

It was done by defining views in a type of document:

  <default-view>view_myview</default-view>
  <create-view>create_myview</create-view>
  <edit-view>edit_myview</edit-view>

It also needs to define navigation cases:

<navigation-case>
  <from-outcome>view_myview</from-outcome>
  <to-view-id>/view_myview.xhtml</to-view-id>
  <redirect />
</navigation-case>
<navigation-case>
  <from-outcome>create_myview</from-outcome>
  <to-view-id>/create_myview.xhtml</to-view-id>
  <redirect />
</navigation-case>
<navigation-case>
  <from-outcome>edit_myview</from-outcome>
  <to-view-id>/edit_myview.xhtml</to-view-id>
  <redirect />
</navigation-case>

and applications:

  <view id="/create_myview.xhtml">
    <theme>galaxy/popup</theme>
  </view>
  <view id="/edit_myview.xhtml">
    <theme>galaxy/popup</theme>
  </view>
  <view id="/view_myview.xhtml">
    <theme>galaxy/popup</theme>
  </view>

Then the javascript function showFancyBox() works very well for showing and editing the document.

There is also nice functionality for deleting a document by using Seam.Component.getInstance('popupHelper').deleteDocument(mydoc.id,cbfunc);

But I still don't solve how a new document should be created by using a popup window.

It needs to inject or set changeableDocument. In actions and commandlinks there is used documentActions.createDocument, but how it should be implemented in javascript ?