11
38 Comments

How do you setup your (web) Saas backend architecture ?

Hello all
bit technical question , i wonder how do you setup your backend Saas architecture i mean
Fail over , DB , single point of failure , monitoring ,security .

on September 15, 2019
  1. 11

    Hi, the best piece of advice: You don't need to auto-scale, you don't need a failover plan and it's totally fine to have several single points of failure.

    I run a business with 5000 customers, that it's doubling its size in users every month, receiving more than 150k-200k requests a day with really expensive write operations (dozens of tables, it's hard to cache anything), and I'm starting to worry about infrastructure right now, after 1 year of operations.

    There is a big probability that a $20 DigitalOcean droplet is totally enough for your first year of operations (with the web app, database, redis, nginx, task queues, everything there, in a $20 server), just make sure to have backups, and if something is wrong and you have to take 1 day to restore the backup, it's totally fine.

    Focus on growing your business, forgot about servers the first year, there is a 99% chance that you will fail and you never will have any infrastructure issue.

    1. 1

      how do you do database backup ?

      1. 2

        Hi, in DigitalOcean you can do weekly backups of the whole instance, just enabling backups. You can also do manual snapshots of instances and/or volumes (I don't recommend using DO volumes for the DB as they have low IOPS compared to the native instance storage).

        I've used this guide to do daily backups as I'm using Postgres:
        https://coderbook.com/@marcus/how-to-automatically-backup-postgres-to-s3-and-cloud/

        But there are plenty of guides out there depending on the database you are using and where do you want to store the backups.

    2. 1

      The 99% may just as well be related to not thinking deeply about all kinds of future issues in the business ;)

      I agree, a droplet will be fine for most services in the beginning. But the choice is still important.

      In my startup, I initially went for a small managed docker hosting company, because it was easy at the time, and I thought nothing of it.

      Then they ran into some glitches, but I could handle it. Then some downtime, an hour or two, once every week for no clear reason. Still stuck with them, hoping they'd figure it out.

      And then they had multiple 24h+ downtimes in one week. I had to race to another provider, set up a production system with hundreds of paying customers, rightfully outraged.

      I still get panic flashbacks to that day.

      Make sure your infrastructure plan is solid for a while. It's important to have backups and a quick evacuation strategy.

      And yes, no great infrastructure plan makes any sense with a product used by customers that warrants that infrastructure. But as bootstrapped founders, we wear all those hats at the same time. And in each field, we can run into trouble. At least we have some control over the infrastructure part.

      1. 2

        Hi Arvid, yes we had some issue like that in the past, a third-party provider for one important "micro" service that was having downtime decreasing the quality of our service, and I know that feeling.

        What I want to highlight is, you don't really need autoscaling and high availability when you are just starting out, I've committed the mistake of creating the perfect state-of-the-art high available and scalable infrastructure for some projects that never worked and it was just a waste of time.

        1. 1

          If only we knew which ones would work out and which ones would not :D

          Thanks for sharing. Do you think this is being solved automatically by systems like Kubernetes? Did you get to re-use any part of your system in later projects?

          1. 4

            I've learnt a lot, that was the positive part, about how to handle 100 terabytes of videos and processing thumbnails and traffic in a $500/month budget.

            I think using Kubernetes (the right way, with rolling updates, continues deployment, zero downtime in the app server, DB replicas, etc) is an overkill for a project that is just starting out, I would use Docker and docker-compose, or just having a quick deploy script is totally fine.

            I have an abandoned side-project that used to have 480.000 monthly visitors in 2016:
            https://i.paste.pics/341f78af7f56d5e110d3b8a316df6c37.png

            and all the time it was on a $20 DO Droplet without any issue.

            As I've said, it sounds counter-intuitive but I think you should NOT worry about infrastructure when you are starting, obviously, there are exceptions if infrastructure is part of your core business.

            1. 3

              I agree, I have a side project which has seen significant traffic on a $10 DO instance (excluding separate db), and for my day job deal with significant traffic (millions of visitors a year) on a slightly more complex setup (but not much more, still no kubernetes), and architecture concerns are just not important for startups till you start to succeed, and if you try to guess them in advance by copying much larger companies you'll get it wrong IMO.

              Better to start small and do things that don't scale at first, till you find out if you actually need scale at all.

    3. -2

      This comment was deleted 4 years ago

  2. 3

    For my enterprise clients(I do technical consultation for various companies) I prefer to use Microsoft Azure, AWS (now-a-days mostly prefer Azure over AWS).

    But for quick and easy prototype and develop an app I use Google Firebase. Firebase is very quick to develop, no-headaces with hosting(ssl etc), functions works great, not much of setup. For an example I completely developed https://dearele.com over a weekend on firebase (its down now because of Chinese product spammers). I hardly spent any time on setting up anything other than writing code.

    1. 1

      I second this. I just created a prototype app and put it on Firebase(my first encounter with it). In about 2 hours I had, hosting, domain, database, authentication, authorization, and cloud functions that will scale. And I haven't spent a penny yet.

      Now I can focus on iterating my app, not infrastructure.

      1. 1

        I forget about authentication. Authentication with firebase is the easiest I have seen. I think email and social media authentication for my project https://watermark.ink is completed in couple of hours.

  3. 2

    It really depends on the level of technical expertise you have.

    I set our system up as a Docker-based deployment that could be run either on Kubernetes or in self-hosted Docker setups, with a Database-as-a-Service solution.

    I highly recommend using a Cloud Provider like Google Cloud, using their Kubernetes offering called Google Kubernetes Engine with a cluster spanning a few availability zones (for failover and high availability). Host your database on the same Cloud Provider, and use their hosted database offering. That way, all hardware parts of your system are maintained by and colocated at the same company. That also allows you to use enterprise-grade security products, firewalls, intrusion detection systems. Auto-scaling is also an option in all those services.

    You can do all of this on the Google Cloud, AWS or Azure. It will require some engineering. If you don't want that, you can go for more expensive but less complicated solutions like Heroku, DigitalOcean or UpCloud.

    Using containers for your backend will make it very movable, should you need or want to migrate the system at any point. This might not sound important, but imagine expanding or selling the company, where your requirements change.

    For monitoring, think about uptime monitoring like Monitis or Pingdom as well as error/performance monitoring like New Relic, Sentry or AppSignal. Pick services that work well for your specific circumstances.

    1. 4

      but imagine expanding or selling the company,

      I wonder why should we even consider selling the company while designing our system architecture?

      1. 2

        It sounds kind of like optimizing for divorce considerations when planning a first date.

        1. 2

          OMG, that's a perfect analogy.. lol

      2. 1

        Let's say you've build a system that runs on dedicated Hetzner servers and requires special knowledge to operate. There is a number of hacky shell scripts for deployments, code pushes trigger certain scripts, and cron jobs handle overnight backups.

        Compare this to a containerized system that is being built on a CI system with config files that are fully documented. The deployment process is being done through version tags, and it triggers a rolling update.

        When your business is being acquired, there is a thorough due diligence phase. In that phase, the purchasing party will inspect all of your tech. The hosting/deployment mechanisms will be of particular interest, as this determines if their existing resources can handle this in the future, if they will need to hire experts in those technologies, or if they might even have to hire you to do this job. This will impact the value of your business significantly.

        You can also think of it another way: you will eventually need to hire someone to either take over the dev/admin job or at least help out. The second scenario will mean less training, a bigger pool of candidates and thus likely lower expenses for you.

        I argue that building for the exit will result in lower long-term expenses (while it might lead to higher short-term expenses). I have been burned badly by "ah, this will be fine for now"-choices before, and this position is a result of that.

        1. 2

          Got it. Your points are good, companies do consider how well the new(acquired system) integrates with their current existing system.

          I wish I can plan things that far :)

          1. 2

            There is always hope! Hope-driven-development :D

            But seriously, It's a good practice anyway. Any big company will want to have a reliable, efficient and cheap system. All of those things are also important for bootstrapped companies.

    2. 1

      How did you arrive at the conclusion that Digital Ocean is more expensive than AWS? In my experience, every size of EC2 instance is at a higher (or rarely at an equivalent) price for the DO droplet with the same listed stats, and every time I've perf-tested two setups with equivalent listed stats, the DO server is always much more performant.

      If you're using S3/spaces and RDS/hosted DBs, the gap is even bigger.

      1. 1

        Like in the other comment, I was not aware of the DO of today, I still had their offerings from a few years back in mind when I wrote this.

        I would like to add that some companies that are known to purchase software companies have many, most or all of their portfolio companies on the same host, which seems to be rooted in having negotiated non-standard contracts with those hosting companies. That, for any business that is looking at exiting this way, results in needing to be at least conscious of maybe having to move the software from one cloud provider to another.

        Thank you for your performance insights. I will take a closer look at DO for future endeavors.

        I do recall reading about someone having a lot of trouble with DO shutting down their business (due to same false positive in their abuse alarm system) a while back. Have there been more of these kinds of problems?

        1. 2

          I do vaguely remember something like that and I agree with you about the importance of being ready to move in a heartbeat.

          In my case, I have db backups, VM images, Ansible scripts for my OS config and a Gitlab CI/CD workflow for my app, so there are multiple recovery methods. A few months ago, it was just git, db dumps and a bunch of bash scripts, but I could have weathered a host issue.

          What's really important, IMO, is not relying on proprietary features from AWS (or GCP or Azure) specifically. Ditto for Github. Once you do that, you're one bad customer service experience from being disabled without recourse.

          1. 1

            Good thing git is distributed :)

            1. 1

              It is! I think that dev had Github itself tied into his CI/CD as opposed to just using it to host code.

              1. 1

                This is one of those situations where it's so convenient to put all your eggs into one basket :) In a prior job, we locked ourselves into Gitlab the same way, but we had no problems. Just comes with the complexity of things I assume.

                1. 1

                  Unlike Github, you can self-host Gitlab. It's open source!

    3. 1

      Thanks!
      DigitalOcean or UpCloud is cheaper then AWS ... or am i wrong ?

      1. 2

        Digital Ocean is perfectly fine for most sites, you don't need multi-zone for example, the vast majority of businesses never reach that scale (including some which make a lot of money).

        It is cheaper yes.

      2. 0

        I would say it's not, and here is the reason:

        Most cloud provider offerings, particularly with their orchestrated solutions like GKE and Google Cloud SQL have features like automatic backups, multi-zonal deployments, maintenance window control priced in.

        I can not speak for UpLoad, but on DigitalOcean, getting these additional features set up will end up costing you more than setting stuff up yourself on the big cloud hosters.

        Services like DO have started offering hosted solutions with additional features too.

        So if you go by price, then they might be cheaper for the same amount of features you need now. You might be fine with that for a while, maybe even forever.

        Whatever you do, make sure you use a Service that gives you all the tools you think you'll need a year from now. Moving a production system from one host to the other is quite complicated once you have customers.

        1. 2

          DO has automated backups and its hosted DBs are less expensive than AWS or GCP's. Unless you're doing something really enterprisey or expect to scale faster than Uber did (without needing AWS, btw), then there's no rational reason to believe DO couldn't serve your needs.

          Some of their customers serve billions of events daily.

          1. 2

            Yes, it was a mistake placing DO with Heroku and the like. I admit my ignorance to their most recent offering, which now stands corrected. Thank you!

  4. 1

    There is great HN post that going on strong which talking exactly about this:
    https://news.ycombinator.com/item?id=20985875

  5. 1

    I'm kind of passionate about this; I wrote a whole article about it.
    Tech Decisions and Developer Guilt

    Hosting and tech

    Don't over complicate things. A few things to look at to get running quickly:

    Use their serverless functions as a service (FaaS) to get off the ground quickly.

    Also, look into (slightly more complex)

    You don't really need to worry about fail-overs if you do this correctly. (until you are at a massive scale) Throwing static assets on a CDN is enough to CYA.

    Databases as a service
    Don't run your database yourself.

    Other options
    All my advice biases towards full-stack JavaScript development. To counter that, let's explore some no-code resources.

    Overall, do the simple thing. Don't spend too much time on tech.

  6. 1

    There was an excellent Twitter thread from an experienced AWS guy on all infra:
    https://twitter.com/dvassallo/status/1154516910265884672?s=19

    1. 1

      wow this is great tweet!

  7. 1

    I rent a VM at UpCloud. It runs CoreOS, and the app consists of several Docker containers (Traefik as a reverse proxy and https endpoint (via Let's Encrypt), the rest are Python images). When the server restarts everything is up and running within a few minutes. Most importantly, it allows me to test the exact environment that gets deployed, and rolling deployment is near instantly.

    I've used UpCloud's console to setup an automatic full-disk backup every week. One container does automatic backups of the databases every day, using a scheme that leaves snapshots for a week ago, a month ago, etc.

    I use (one of the many) website monitoring services to get an email when my server is out. When it's out longer than 7 min I get an SMS.

    1. 1

      how do you do failover ? scale ?

      1. 2

        Failover is handled by UpCloud. They can move to VM to different physical machines when needed, and have a 100% SLA.

        As for scaling, I've done some experiments and calculations, and expect to be able to scale to 100k -1M or so users by only scaling the "hardware" (amount of RAM en number of CPU's). After that, I'll need to change the architecture. Worrying about that makes little sense now though :)

      2. 1

        Failover is important for your data only, the other instances can just be replaced by another instance in a few minutes (you can just redeploy in a few minutes hopefully, if not, try to fix that).

        For the data if you make sure you have great data backups and can failover the db (you can set up streaming replication for example), you really don't need to worry much about failover of app instances - put a couple behind a load balancer if you're worried about it (Digital Ocean offers this for example). Assets you may have to do something similar or use cloud hosting. Try to keep anything unique off your app instances so that they can be thrown away.

        Scale is not really important till you have customers and even then you can't really plan well for it, just accept that you'd have to adapt a little at each new level of scale.

  8. 4

    This comment was deleted 4 years ago