This paper focuses on the randomized Milstein scheme for approximating solutions to stochastic Volterra integral equations with weakly singular kernels, whose drift coefficients are non-differentiable. An essential component of the error analysis involves the utilization of randomized quadrature rule for stochastic integral to avoid the Taylor expansion in the drift coefficient. Finally, we implement the simulation of multiple singular stochastic integral in the numerical experiment by applying the Riemann-Stieltjes integral.