First method
To delete a file or folder without moving it to the Recycle Bin, hold down the Shift key and use Delete command. You can use following three way for this.
•Select the file or folder, hold down the Shift key, and then press the Delete key with keyboard.
•Right-click the file or folder, hold down the Shift key, and choose the Delete item on the context menu.
•Select the file or folder in Windows Explorer window, hold down the Shift key, click the Organize button, and then click Delete.

Second method
If you don’t want to use shift key, to delete a file or folder without moving it to the Recycle Bin, you can do with change recycle bin setting.Right click recycle bin icon and then click properties key and select “Do not move files to the Recycle bin .remove files immediately when deleted” Radio button, and then apply and ok.

Now you don’t need to hold down the Shift key, use only delete command for permanently delete
In all condition when you trying to delete any file or folder .Windows vista display the Delete File dialog box. Click the Yes button to delete the file or folder.

Tags: How to, Windows Vista