HLSL frac, fmod, and modf functions

[Clint Eastwood]

Hello,

Does anyone have experience using the HLSL frac, fmod, and modf
intrinsic functions?

Regardless of whether I use frac(x), fmod(x,1.0), or modf(x,i) to
obtain the fractional part of a function, the fractional part always
appears to be clamped and re-scaled such that if I were to graph the
fractional part of the line y = x, I would get this:

____ ____
/ | / |
/ | / |
/ |/ |

instead of this:


/| /|
/ | / |
/ | / |
/ | / |
/ |/ |

It definately appears to be a clamping problem, because if I scale the
result (e.g., frac(x) * 0.5), the clamped effect goes away and my end
result was greater than 0.5 (about 0.6), as if the frac() function was
returning values larger than 1.0 and then clamping them.

Does this sound familiar to anyone?


I'm using DirectX 9.0c (don't remember if it's summer or october SDK
update). I've tried this on both an nVidia GeForce 6800 and an ATI
x800 Pro and get the same result.

Thanks,
Matt

[Clint Eastwood]

I would like to retract this question. It turns out that there is
nothing wrong with the HLSL functions (of course). I was getting odd
results because I was doing multipass rendering with fixed function
alpha blending.

--Matt

[Clint Eastwood]

I tried to post a reply to this yesterday, but for some reason it
didn't go through.

Anyway, it turns out the frac/fmod/modf functions are fine, and the
problems I was seeing was a result of improper use of alpha blending in
the fixed function pipeline (I was doing multipass rendering).

--Matt

Post by Clint Eastwood
Hello,
Does anyone have experience using the HLSL frac, fmod, and modf
intrinsic functions?
Regardless of whether I use frac(x), fmod(x,1.0), or modf(x,i) to
obtain the fractional part of a function, the fractional part always
appears to be clamped and re-scaled such that if I were to graph the
____ ____
/ | / |
/ | / |
/ |/ |
/| /|
/ | / |
/ | / |
/ | / |
/ |/ |
It definately appears to be a clamping problem, because if I scale the
result (e.g., frac(x) * 0.5), the clamped effect goes away and my end
result was greater than 0.5 (about 0.6), as if the frac() function was
returning values larger than 1.0 and then clamping them.
Does this sound familiar to anyone?
I'm using DirectX 9.0c (don't remember if it's summer or october SDK
update). I've tried this on both an nVidia GeForce 6800 and an ATI
x800 Pro and get the same result.
Thanks,
Matt