En passant is only if you COULD have captured if white had done the normal move, A2-A3. It doesn't matter what move that white A pawn made, your black A pawn can't possibly capture it.
If en passant allowed the kind of move you're suggesting, blocked pawns would be a thing of the past!