Can I go with this new company without telling the agency
If you change employers, you need to let your current employer know. If a new contracting firm is taking over, I'm sure they expect many employees to leave and go work for the new company.
However, I'd also encourage you to explore the other opportunities within your existing company before you switch. Showing them loyalty may get you a better position.
If you mean to apply for a permanent position at the new company then you do not need to tell anybody.