When you use a "normal" submit button, the form will use the "required" flag that you can assign to form fields in order to figure out if the form is ready to be submitted. When using the Doc.mailDoc function, you need to do this yourself.
One way I've done this in the past is to have a loop that goes through all form fields and looks for the "required" property (http://livedocs.adobe.com/acrobat_sdk/11/Acrobat11_HTMLHelp/JS_API_AcroJS.89.726.html). When it finds a field that is required, it then checks to see if the value is different from the default value - or different from an empty string. It's up to you to decide what constitutes a valid entry in a field.
If you find one or more required fields that have no valid data, you would then pop up an alert, and not call the mailDoc function. If everything is OK, the document would be mailed.
Karl Heinz Kremer
PDF Acrobatics Without a Net
PDF Software Development, Training and More...
http://www.khkonsulting.com