Introduction To Overgrowth Modding
Here is the current modding situation, based on this post by Endoperez.
Here's a short and dirty list of what's possible at the moment.
Scripting - Possible, limits are still quite vague.
Custom GUI - Possible, not documented. Done through CSS.
Terrains and levels - there's not much documentation for how it works and how to do it, but with some trial and error it should be possible to convert a grayscale heightmap into a custom Overgrowth level.
Mesh Objects - houses, walls, towers, trees, fruit, any static object is possible. The normal maps should preferably be in Object space format, but Tangent space works too, it just needs a different shader. The files should be exported in OBJ format, check scale by opening one of the OBJ files that come with Overgrowth. Custom weapons work as well as the official weapons, (pretty poorly at the moment, but they're easy to update as more weapon features appear). If you are new to modeling, check Making New Models & Textures.
Special effects, animated objects - there's a shader for plants moving in the wind. There's no shader for water of any kind. Creating custom particle textures is easy, but since particle movement and spawning has to be done through code anything more complex than a dust cloud created on point of impact takes some experimentation. Shaders themselves come with the game in GLSL text format, so someone familiar with writing shaders shouldn't have problems coming up with new shaders.
Animated characters, custom animations for existing characters - If you use Blender 2.56, the animation exporting scripts should work. You get the scripts and the .blend files with the alphas. Theoretically, it should be possible to create and animate a custom character, but the scripts are written to only use the default animations at the default paths I think, so it wouldn't really be worth the effort ATM.