I've been trying to get the vector graphics renderer from Presenta to play nice with the F# interactive mode. I only just discovered the problem and, in fact, I think it is the same problem that wreaks havoc with all Windows Forms applications when run from the interactive mode (including the bundled F# samples).
The problem arises when you try to spawn forms and continue using the interactive mode. The UIs are then executing on the same thread as the interactive mode itself. This would be fine provided everyone played nice and yielded every now and again using Application.DoEvents() but the interactive mode blocks waiting for user input. If you're playing with one of the F# samples from the interactive mode and its GUI freezes then try hitting return in the interactive mode to make the GUI responsive again.
Anyway, I tried to fix this in various ways, spawning threads, and had no luck... until today. It turns out that the fix is really easy. The fundamental problem is that Windows Forms are not thread safe. They use a "message loop" to convey things like the need for repaint and this whole loop must execute on the UI thread and not in any other thread. You can invoke messages from other threads but, if you just want to spawn a background window (e.g. for visualisation) and don't care what happens to it, there is no need to worry about invoking messages.
So all you do to create a completely separate "application" is spawn a thread, create your form and apply Application.Run(form) to it. The call to Application.Run sets up a completely new message loop for that thread. I've been playing with it this afternoon and it seems to work perfectly. I can spawn multiple windows visualising different data from the interactive mode and they all run beatifully. Moreover, they can take advantage of my dual core because they are running in separate threads.
One final discovery: there is a thread.IsBackground boolean property that is false by default. All background threads are killed when the last foreground thread dies. For example, if you have a GUI thread and a worker thread then you can make the GUI thread the foreground thread (the default setting) and set IsBackground in the worker thread to true so that it is killed when the GUI is closed. I've been bitten by not doing this during development, when a process hangs around after I believed I had finished it and I can't then rebuild because the EXE is in use.
Anyway, I'll be putting lots of essential bits of information like this in my book "F# for Scientists"...