What is commonly lumped together under the general term "textures" is actually a collection of related files that serve different roles in creating the image of objects within the game. Some of these terms, such as "meshes", may have already been encountered in mod descriptions or install instructions, but once you start dealing with "texture replacement", you need to become aware of the technical differences and very specific meanings associated with each of the terms given here.
Such "replacer" mods include differing combinations of the related files. When you are looking to create a specific effect (such as adding custom hair, eyes, skin textures, or features like tattoos and scars), you need to recognize which specific types of files will achieve that result. Combining files from different mods may be required, and if you do not understand what each file type contributes to the whole, you will have an unhappy result that could have been avoided. Hence, at least a cursory look over the Glossary and other terms on this page is an essential first step. Refer back often as needed.
Bear in mind what sort of platform the game was written for originally. For instance, Morrowind and Oblivion assumed hardware with a single processor. As such, they do very little with multiple cores; CPU speed (the faster the better, starting around 2.8 GHz) is more important than the number of cores for including mods. The amount of system RAM and video RAM (VRAM) are also crucial to keep in mind for graphic improvements. As a "rule of thumb", double the game's minimum requirement for VRAM before attempting to use more detailed textures. Minimum requirements are not usually sufficient for modded games. Exceeding the "recommended" requirements is necessary the more you want to add, but never expect to be able to include EVERYTHING. See the Nexus wiki article 2-4GB game memory limits and solutions for details on how to maximize the use of what system RAM you have.
There are many articles on the Nexus Wiki about the various tools and techniques for and about modding textures. Those interested should take a look at the site's Orphaned Pages link, in addition to their specific game's forum. (There is nothing wrong with these articles. They are 'standalone': not referenced or linked by another page, which is why they are considered 'orphaned'.)
In the Options menu, you'll see a choice marked Textures, with Small, Medium, and Large. The Oblivion Tweak Guide goes into more detail, but here's an excerpt (note that this was written several years ago; some of the information here might not apply):
Textures are the 2D images placed around all 3D objects in the game. A boulder for example is a wireframe skeleton in 3D, with a 2-dimensional boulder surface texture covering it (to see this, use the twf Console Command). When set to Large, the resolution of the 2D texture images used is highest, meaning the textures appear more crisp and real, especially when examined close up. At Medium and Low respectively these textures progressively drop, losing a lot of detail.
The texture setting can have a significant impact on performance and stuttering, because these textures load into your graphics card's Video RAM (VRAM), and the larger the texture resolution, the bigger the size of the texture files and hence the more swapping your graphics card has to do to constantly load new texture data from the hard drive into VRAM to decompress and display them. Whenever you enter new areas or see new objects/creatures/buildings, the higher this setting the more loading pauses/stutter you will get, and the lower your FPS will drop, especially if you have less VRAM, system RAM and/or a slower hard drive. For those with less than 256MB of VRAM, I recommend Medium as a good balance of visual quality and performance. For those with slow hard drives and/or less than 1GB of RAM, Low is recommended if you constantly find the game stuttering or slowing down. See the settings below, the Troubleshooting Tips and the Advanced Tweaking sections for a range of other tweaks and settings which in combination with each other can significantly reduce stutter. There is no single magic fix to this issue.
People seem to make the assumption that larger is always better in texture sizes for increased detail, without giving any consideration to either what is involved or the practical aspects of rendering that image on their hardware. Thanks to Pherim on the Bethsoft forums for the following explanation in answer to a question about the availability of 8k textures.
Why would you want to have 8k textures? You would see downsampled versions of them 99% of the time, if not all the time, anyway, as the screen resolution even on FullHD is only capable of displaying about 1/32 of the entire texture in its original size, and I can't think of any situation where that would even be the case. And while you would effectively see the 2k mipmap of the texture at most (and usually not even that), the whole 8k texture would have to be loaded into memory, anyway, and I hope I don't have to tell you that a single 8k texture takes up 16 times the memory of a 2k texture - every single 8k texture [compressed] would take up more than [40 Mb of memory, and almost 90 MB with alpha channel (transparency) data included, so] you'll reach the RAM and VRAM limits quite fast that way I would imagine. In Skyrim it may make sense to have certain textures in 4k resolution, even in some cases in Oblivion, but that is only the case if only parts of a texture are used to cover a large area. I am sure in almost all cases you would not be able to notice the difference between 2k and 4k, let alone between 4k and 8k. As I said, most of the time you would see them rendered in 2k or lower, anyway. In fact, they would look a lot worse if the full-size texture was rendered at all time and downsampled in real-time, instead of using pre-downscaled mipmaps. Just try using a texture that doesn't have them (e.g. in .bmp format instead of .dds), and you will see what I mean. If you have no idea what I am talking about: http://en.wikipedia.org/wiki/Mipmap
As you can see here, a texture with mipmaps contains several smaller LOD versions of itself, with every mipmap level having half the width and height of the next larger one, down to 1x1 pixel:
So a 1k texture, for example, consists of the full resolution image with 1024x1024 pixels, but also of a 512x512 version, a 256x256 version, etc. If a .dds texture with mipmaps is opened in GIMP (you'll need a plugin for that), the mipmaps are represented as separate image layers. Now when the texture is rendered ingame and there are no mipmaps, the engine only has the full resolution image to work with, but has to downscale it to match the current view. This leads to texture pixelation and flickering, like in this picture, but it looks a lot worse in motion:
With mipmaps, the various smaller mipmap versions the texture includes already have pre-calculated anti-aliasing as they have been downsized in advance, and the engine automatically chooses the mipmap which matches the actual size the texture is displayed in the game most closely. So for example if you look at a wall that is currently 1000 pixels wide on your screen, a 1k texture will be displayed in full resolution, but if the texture is bigger, you actually see the 1K mipmap. The smaller the texture is displayed in the game (the actual pixels it takes up on your screen at any given moment), the smaller the mipmap that will be displayed. So even on FullHD, a 2K texture is only shown in full size if it takes up the whole screen width, and then you can only see half of it. In case of an 8K texture, there is barely any chance that you will ever see it in full size ingame. If a texture is displayed in several sizes at the same time, for example on the ground, where it becomes smaller the further away it is from you, the engine will use several mipmap levels at the same time and stitch them together. Then, anisotropic filtering can be used to make it appear sharp and detailed and without visible mipmap transitions.
What many people maybe forget to take into account when creating high-res textures is how big the texture will actually be seen ingame. For a wall that covers big parts of the screen if you stand close to it, it makes sense to have a texture size that exceeds the screen resolution, but for small items which are never seen up close and only take up a few hundred pixels on the screen, most of the time, you probably won't need anything bigger than 1K.
If you're planning to use any kind of texture mod, there two types of memory you need to consider: system RAM and video RAM (VRAM). VRAM is dedicated to the video processor and is integrated into the video card component. VRAM of 512MB is a recommended minimum; 1GB+ of VRAM is preferred. Use of one of the Texture Optimizers (under Utilities on the navigation bar) to reduce the size of the textures may provide a satisfactory compromise for VRAM limited systems.
As with VRAM, more RAM is better. Unfortunately, most Windows games (especially the TES/Fallout series) don't use more than 2GB of system RAM by default. There is, however, a simple program that can "tweak" the executable file (the one that launches the game) to make it use up to 4GB. You can read more about it here.
ENBoost is a memory management feature of the ENBSeries to address some of the limitations of 32-bit games. In the context of that very general statement it is similar to the LAA flag fix that enables 32-bit games to use more than 2GB of RAM. However, it works differently by providing dynamic caching of video object geometry in system memory when the video adapter runs out of room, instead of CTD. This most commonly occurs when higher resolution texture mods are used. (For a more detailed explanation, see the S.T.E.P. Project Guide:ENB article.)
ENBoost is a stand-alone component called "enbhost.exe". It is not involved with nor require the post-processing of the remainder of the ENBSeries, so it does not impose the FPS loss that ENB is known for. Therefore, it is recommended for all users. See the S.T.E.P. Project ENBoost Quickstart Instructions on how to install and configure for only this feature.