Huge difference, tbh. Simply said, working in-house basically looks like you're working for 'one' client, in agency you are supplying multiple clients.

An in-house designer is employed specifically to work on their employers project(s) and product(s). The company you work 'in-house' for could be anything from a startup to a fortune 500 company. Comparatively, an agency designer could work on/for dozens of different clients and projects, in many different industries.

Working in-house gives you greater access to and understanding of the business, its needs, audience, data, and stakeholders. A project doesn't end at launch — you learn, iterate and scale. You are likely more invested in the project, as you are tied to its success, so to speak. The work you do is likely problem solving focused, over creating award-winning work, and you're fine with that. The pay/salary is probably much higher (including stock/shares too) in-house, with less churn, and the hours are probably much kinder.

An agency designer can ship a project and never see or hear of it again. Unless you have a close bond with the client, you likely won't see any data, learn what worked and what didn't, and may not have a chance to iterate on work that shipped. Most likely everything is done 'on the clock', billed hourly/daily and tight deadlines are more common, which can be reflected in the hours you work. The work you do is likely more short-term focussed, with more emphasis on how it looks (think: portfolio and awards) than long-term strategy, MVPs, product rollouts etc.

No doubt both have their own pros and cons, culturally, politically and the type of work you do. They are very different, neither are for everyone.

P.S. Note my repetitive use of 'likely', 'probably' and 'may' in the descriptions of both :)