RuneScapes old rendering engine was an in-house software based rasterizer(?) written to their specifications. I don't know what gives you the impression that RuneScape was 2.5d unless you were referring to RuneScape classic.
Putting a character in a scene with some static geometry isn't hard, you might want to take a look at Ogre3D or some of it's alternatives (Irrlicht (I hate it), OpenSceneGraph, etc). There are plenty of resources out there which can help you.
Although if you are really set on building your own complete game engine suite from the ground up then you're going to want to research OpenGL/Direct3D and have a good read up on any 3d graphics articles you can find. I wouldn't recommend this though, it's the equivalent of a painter making up his own paint from raw materials just to paint a wall; you're better off with a ready made solution.
OGRE – Open Source 3D Graphics Engine
Irrlicht Engine - A free open source 3D engine
osg
Note: these aren't fully fledged game engines and are focused more on graphics. If you want to use some sort of WYSIWYG tool then look at Unity3D, CryEngine or UDK.
edit: there a lot more things to consider when creating a game engine than just graphics. You need sound, input, UI, speed, physics and that's just the beginning.