It's not an assumption. We know for near certainty that we are the product of genes and environment. Even if I grant you the slim possibility that there is some other factor, such as the existence of a soul. We still wouldn't be able to claim responsibility for the soul that we have; therefore, the argument is still valid. There is no possible factor that could contribute to the choices we make that is within our control. The fact that you cannot even conceive of such a factor proves the point. No one in history has been able to so much as imagine a possible other factor that would be compatible with free will, no matter how crazy or unlikely.
There is no evidence in the sense that you are using the word. There is only deductive reasoning, which is as powerful an epistemological tool as you could hope to utilise.
The fact I can't conceive a confounding factor proves one doesn't exist??? That's not a great argument.
As I say, though, you keep making that assumption. Essentially, someone could come up with a free decision they made, one which could have been different had they chosen for it to be so, and your response will simply be "nah, there's been environment and genes in play there, even though I can't truly show just how that happened".
It's like arguing with a religious nut about the existence of God.
Let's go wild as you suggest - if there's the possibility of an uncaused event, like the very first one as we logically believe there must have been, then why not the possibility that somewhere in our brain exists the ability to override a chain of events in there, or start a new chain of cause and events?
That's pretty unlikely, but not complete fantasy if we believe there was an uncaused cause somewhere in history.
Also, from an evolutionary standpoint - why this desire to believe we're in control? We know when we're not in a real situation, and become very uncomfortable with it (pretty much the premise behind Inception). We can tell the difference between a dream and reality. We have this absolute need to feel in control of what we do. If we never have been, why has that behaviour been one which has developed through natural selection, and why would it be seen as a helpful one to delude ourselves like that? I know there's flaws in just about every species in some way, but that one is pretty massive.
As I've said earlier, I actually reckon you might be right here, but it's purely speculation all round. You can't dismiss other arguments for flawed logic when the logic here isn't as solid as it would seem.