The protesters said if government fails to cancel an invitation to Modi, they will demonstrate again by donning a shroud