The fast idle cam can only fall as far as the choke will allow. As the choke opens, the fast idle cam can drop further. However, the fast idle cam can only drop if the throttle is opened. This is because the fast idle speed screw (which is connected to the throttle linkage) pins the fast idle cam in place. So if you don't touch the throttle, the choke can open, but the fast idle cam would stay put. The fast idle cam would "catch up" and drop once the throttle opens. This is all in line with your description.
If you imagine the throttle is open such that the fast idle cam can move freely as the choke opens, the fast idle cam is "in the way" of the fast idle speed screw (preventing the throttle plates from closing further, thereby increasing idle speed) with the choke completely closed. Likewise, the cam is pointed away from the fast idle speed screw if the choke is completely open. Somewhere in between, the fast idle cam makes this transition. Ideally the fast idle cam "travel" is adjusted such that the fast idle cam is "in the way" of the fast idle speed screw for most of the time the choke is opening. In other words, you don't want the fast idle to drop until the choke is most of the way open (which indicates the engine is warm and can idle at normal speed). In practice, however, the fast idle linkage is usually adjusted such that the fast idle drops out "earlier" than that. Refer to figure 5 on page 3 of the rebuild sheet below for proper adjustment if necessary.