For any non-zero value of m the acceleration is independent of the mass. But with m=0 the force is zero. When did Newton say that you need to smooth non-continuous functions by taking the limit? Did Newton even consider light as a particle?

To answer the last question first: Yes, Newton believed that light consisted of "corpuscles". His reasoning was faulty, and so light was then regarded as waves until Einstein proved the existence of photons by the photoelectric effect. Newton had no reason to think that anything could exist with zero mass, so I don't expect the need to take a limit occurred to him in this case.

Back to the previous question: Yes, Newton devised differential calculus in parallel with Leibniz, though Newton called his version "fluxions". A central process in differential calculus is taking limits, particularly the limit of a function as a variable tends to zero. In this case the function (acceleration in a given gravitational field) is a constant, so the limit is trivial.

Newton/Leibniz calculus does not deal with non-continuous functions, but I don't see why you think a non-continuous function is required here. If the undefined nature of the acceleration due to gravity at m=0 bothers you, simply define it as being equal to the limit.

For an example of a function rigorously defined in this way, see ...

http://mathworld.wolfram.com/SincFunction.html