One way that I evaluate a new IDE is by opening a file in a fresh install and determining how much I can do with that file. Obivously features that need to know about the other files in the project won't work. I recently tried to use NetBeans and JDeveloper to generate getters and setters. Neither would allow me to do that and one of them allowed me to choose that menu item but it just did nothing at all. Maybe creating a project would have helped but that should be needless. When an IDE "gets in the way" with what I want to do like that then it feels more like a straightjacket than a tool. I know that is a simplistic evaluation but when that test fails I wonder how constraining the rest of the IDE is.
Do you think that is a fair way to evaluate?