I wouldn't go with the sandpaper idea, keep it smooth. I got some questions though.
1.) What kind of heater core did you buy? if it's one of those with the plastic ends then one of them may have cracked
2.) what kind of clamps are you using? If they aren't screw-tighten then take the old spring style off and trhow them away and get the screw-tighten style. If they're the screw-tight kind then tighten them to the point where they are flush with the hose material, providing that the hoses are new. This leads me to the next question.
3.) are your hoses new or the originals. Good idea to replace the hoses when you do this, even if the old ones are ok. The hoses may have conformed to the shape of the old hearter core ends.
4.) where are the clamps located? Make sure that the clamps are past the bump on the ends of the heater core.
5.) how did you get the hoses onto the ends of the heater core? If you pried them on then the hoses may be damaged. Use a little coolent on the inside of the hoses to lube then and help them slip on.
I know that most of this might be a bit insulting, but it's worth checking over. If you've done all of these things then the new heater core may be bad.