Mods to Avoid
This list is far from comprehensive. There are still tons of mods not listed here that will break your game. This is why it is very important to have some modding knowledge before adding any mods.
Always view every plugin you install with FO4Edit to check for conflicts.
Bugfix & Optimization Mods
ENBoost and ENB's ForceVideoMemorySize
Both are unnecessary and bring the possibility of more issues. ForceVideoMemorySize was made to fix an old bug with game's streaming system, which has been fixed for a long time (this is coming from Boris himself in the ENB Discord server).Claims stating that this feature is required in order to use RAM as VRAM are false. RAM (and pagefile) have been part of the virtual memory pool used as VRAM since WDDM 2.0 in Windows 10. Crashing because you're out of VRAM is not a thing unless strictly enforced by the program (not the case here).
FO4FI FPS Fix
Causes crashes in some locations.
FPSL - Lighting and Tweaks Performance
Contains many edits that are not related to the mod's goal at all (such as water output from a pump and generator cost), does not really improve performance in any relevant amount and it will make light sources look miserable.
Mods from Qrsr
Many of their mods have been proven to be released untested with tone deaf changes masquerading as fixes. Fixer, specifically, contains many invalid fixes and subjective changes.
Any configuration file for Buffout 4/X-Cell/High FPS Fix
Mods like this are advertised as some magic solution but they are not. TMR has all the changes that you need and it's especially important to set up HFPF yourself as its configuration can vary a lot based on your setup.
Visual Mods
Oversized textures
Based on the object size and how the texture is wrapped around it, you should pick textures with a reasonable resolution and avoid the very big ones also based on your display resolution. Keep in mind that the textures will not be displayed with their full resolution at all times, in fact they will be scaled down because of mipmaps based on distance and other factors, as shown with these sliders. Just for reference, a maximum of 2K at 1080p is a pretty safe bet in terms of balance.
ENB
Fallout 4 does not need ENB as much as older titles such as Skyrim (even though SSE has Community Shaders). Using ENB also brings a flat CPU performance reduction of about 25%, which is far from worth it especially if you use ENB for simple effects such as a LUT.
AI-upscaled texture packs
Mods upscaled with AI or other lazy techniques generally are not worth the higher VRAM usage.
Fallout 4 Official HD Texture Pack DLC
Almost twice the size of the base game on its own, not worth the extra VRAM usage (from around 3GB to 5.5GB on average) considered how close it looks to vanilla.
Functional Displays
Causes crashes, you can fix this by deleting everything in the meshes folder except for the Functional Displays folder.
Pip-Boy Flashlight
The shadow option from this will severely affect performance.
NAC and NAC X
Bloated, riddled with bugs and performance intensive, as shown here (scroll down).
Content & Gameplay Mods
Any mods that edit timescale
Vanilla AI packages and scripts are based on the vanilla timescale, and can break and/or run when they aren't supposed to if the timescale is changed.
Armor and Weapon Keywords Community Resource
Extremely bloated and buggy, makes tons of out-of-scope/undocumented changes.
Armorsmith Extended
Old and buggy, not to mention extraordinarily overpowered.
VIS/VIS-G
Requires patches with any mods that change/add literally any items, also makes random changes to item names to make them more "realistic" for some reason.
Scrap Everything/Settlement Cleaning Mods
Most of these mods will break precombination, which will destroy performance especially in denser areas. A notable exception to this is The Rebuild Collection - AIO.
Damage Threshold Framework
Occasionally causes crashes due to calculation errors.
The Beantown Interiors Project
Breaks precombineds, can accelerate save bloat due to the massive amounts of items in each interior.
South of the Sea
Very buggy and will cause crashes due to deleted nav-meshes. Use South of the Sea - Atoms Storm instead.
Most mods that raise settlement happiness
Most are made incorrectly, while your happiness number will go up, it doesn't actually do anything. Please refer to Tenhat's comment for a more detailed explanation. Notable exceptions to this are Radios Give Happiness - Settlement Workshop Tweak and Simple Happiness Emitters - New Buildable Object for Settlement Workshops.
Miscellaneous
Sinitar's Modlists
If you have a few hours and want more information about Sinitar's lies, scamming and lack of modding knowledge, check out this twenty-eight page essay that goes into excruciating detail about it.
HUDCaps
Breaks the saving system, a fix is available in the comment section.
The MOFAM mod list
Contains various misinformation and bad recommendations, such as running a pre-NG version of the game and leaving the user with many outdated fixes.
The BiRaitBec mod list
Heavily outdated and misinformed.