This is more a guide to make it work in 3D than a fix, because there aren't any (fixable with shader modifications) issues.
1- Download dgVoodoo 2 from here (latest version at this moment is 2.53): http://dege.freeweb.hu/
2- Unzip it somewhere and copy the "DED8.dll", "D3DImm.dll" and "DDraw.dll" files from its "MS" folder to the folder where "Rogue Squadron.EXE" is ("I:\GOG\Star Wars - Rogue Squadron 3D\ROGUE" in my case).
3- Open "dgVoodooSetup.exe". In the "Config folder / Running instance" option, press "Add" and choose that "ROGUE" folder I mentioned before.
4- Configure it like this pictures. Some graphics settings can be configured to your liking. Then hit "Apply".
6- Open the game. It will be in 2D by default if you use the fix from step 5. That's because having 3D enabled before loading screens messes up some graphics. So wait until you can control your ship in a mission and then use the emitter button or ctrl+t or the hotkey you use for that.
- "q" or "right stick click / R3" in your controller: three convergence presets.
- Mouse "back" button: hold it for 0 separation.
- "Left stick click / L3": 0 separation toggle.
Edit: editing the "Rogue Squadron.EXE" file with an Hex editor, it's possible to use widescreen: https://www.gog.com/forum/star_wars_rogue_squadron_3d/rogue_squadron_at_higher_resolutions_60_fps_windowed_mode/post14
After that, select a 16:9 resolution in dgVoodoo.
- Limit fps to 60 with Rivatuner Statistics Server. It will make frame pacing perfect in 3D. Switch that to 120fps (or to your refresh rate) when you want to play it in 2D instead.
- Use a controller to play the game. The mouse input is a bit strange, making it hard to aim. The controller feels more analog.
- The crosshair is always at screen depth, and that's annoying. You can disable it ingame and use the Nvidia crosshair (ctrl+F12) instead. It will work properly in cockpit view, but not in third person mode (the ship gets in the way and affects the crosshair depth). Aiming in this game isn't hard without a crosshair anyway.