As John mentioned, security is not really an issue. The area of Philly it's in is rather affluent.... million $ mansions... etc... down near Lincoln Drive, where you're closer to Germantown, it's a little worse... but has improved quite a bit over the past few years... seems the yuppies have "discovered" G-town and started buying some of the nicer properties there... on the Roxboro side - solid middle-class...
It's still the city... so you have to keep your guard up to some extent, but especially during the daylight there's little to worry about... Lately, I've been parking at the movie theater off Main St at the southern end of Manyunk and starting my ride from there... easier parking and saves a bit on driving as well (closer to the hwys).
In the Spring it can be VERY crowded... we usually ride elsewhere until around July when things seem to settle down a bit... or ride early in the morning... Definitely don't ride there planning to hold a race-pace or anything... as you're liable to find a hiker or horse around the next corner...
First day of fishing season - stay away. Far away... mainly because parking is just insane...