Be sure to contact your new host to get the new DNS information to give to the old host. This change usually takes 48 hours (two days) to take effect, at which point the site will load from the new web host. Once you've requested to move your DNS record, it can take a few hours to a full day for the change to take effect. If you don't feel that level of comfort or you're worried about the negative reports you've seen on your web host, it might be time to make a change.
The change in the hosting environment can cause this to happen, and it's not unlikely that you'll see some 404 errors (not found). In general, no: changing the web server won't affect the SEO of your website, assuming you keep the structure and content of your website the same. Fortunately, this isn't the end of the world; people change their web servers all the time, for several reasons.