How to Start?
I'm struggling to think of a satisfying way to begin this post. This will be largely me rambling, but it's also about how I process game and idea development.
Ever since my introduction to D&D 3.5 I've always wanted to create my very own game system. I've been big on game development for literally my whole life, but I struggled to really think of a good way to tackle it until recently. Countless times in the past have I started and immediately stopped trying to develop my game creation skills. It wasn't until I discovered homebrewing for D&D 5e that I finally found my method of game development that's still going on about 5-6 years later. And now I want to achieve that goal, to create something people love to play.
What do I want to achieve?
This is a question I've been slowly answering over the past few months. As a gamer you know what you enjoy, but when placed in the position to create it you have literally no idea where to begin. This has spiraled into the search for what I want from a game and how to achieve it. I've been doing 200 word games to help hash out smaller ideas, rather than aiming to make an actual game. It forces you to think of the raw essence of your goal and distill it into the purest form. It's my philosophy that if you can make a fun game in 200 words or less, then it's better than a large game with no flavor.
This is going to be a really long journey. It took me 4 years of homebrewing for D&D until I was finally getting a grasp on development, balance, and flavor. Developing my own system is going to take just as long. Probably longer. But honestly that idea doesn't scare me. In fact I'm excited just typing this. Though this excitement wanes and waxes as it pleases. I'm not always 100%, and you won't either!
Learning to Love Challenge, and other skills
This is what I consider a very important aspect if you want to seriously game develop. Learning to love a challenge will drive you harder than forcing yourself to develop something to meet a quota. It also allows you to gain motivation through more work. Being excited for the unknown is an extremely powerful tool that keeps you interested. Honestly I didn't have this at first - I had literally over 16 years of ideas trapped in my head, which allowed me to create an absolutely inhuman amount of content when I first started development (As stated within my first blog).
But once you hit a wall where you need to develop new skills is the right time to learn to love the challenge. Get excited for not knowing how to tackle an idea, because you don't know how awesome the end results will be. That's what truly drives me. Unfortunately not all ideas are winners, and it's best to accept they're okay, or bad, then move onto a new project. The idea is to keep the ball rolling, then tackle a bad idea again later. Be sure you save all your work, you'll need it later down the line, even if it's a really bad idea. It's my personal philosophy that no idea is inherently bad; it's the execution. Take the Diablo franchise as an example. Imagine if all it had was fighting monsters in a new environment every few hours. Not interesting or fun. But when you add skills, random potential, powerful and interesting loot, and a plot line, the idea becomes more grand. By exploring separate ideas, they can become a conglomerate and create a greater whole.
Collateral Inspiration
Another skill to learn is to work on ideas adjacent to the first one when you're stuck and you will often solve two problems at once. I call this collateral inspiration. This is such a powerful tool and I didn't realize I had it until later. Once I realized its existence, I put it to practical use and suddenly jumped forward in progress. Sometimes you can create exploding collateral inspiration, where solving two problems you were stuck on creates new ideas.
An example of collateral inspiration is you want to work on an ice themed warrior. You might be stuck with some mechanics or imagery behind it. Then work on something else ice themed, perhaps a monster, or create an ice priest. Another amazing idea is to invent a new character within a world you're familiar with and imagine how they interact with it. Creating a character more often than not creates explosive collateral inspiration.
Make something that excites you
I can't emphasize this enough. If you aren't excited by what you're looking at then you should work on the idea more. An interesting idea doesn't mean it's fun, either. There's plenty of paved paths that are really fun despite never deviating from the normal, as in they're fun not just because it's interesting, but because it's fun and interesting. This is by far the hardest challenge you'll face. It's one I'm dealing with now. How do you create a meaningful mechanic fun and flavorful without making it too obtuse or have it weighed down by its own idea? The idea of a blood mage is really fun, but the issue is the mechanics are weighing itself down; it's defeating itself before it can even get out of the gate. Many blood mages I see are too complicated for their own good.
Make something for you, not them
Don't make a game that'll interest hardcore players. Don't make something around a theme just because it sounds interesting. Make it for you, and if you find it fun and exciting, then other people will follow. You can't make something that pleases everyone. You are the first person that you should please. When looking for feedback, ask it only from people you trust and love. People are cruel and uncaring about how or why you arrived at a conclusion; if you start revolving your game around that, you are no longer creating something for yourself, you are making it for them.
Once you have created something you love, and you have tested this idea to be as balanced as it can be, then you release it into the wild. The wild is full of animals that will rip and shred your material, but it's also full of civilized people. When reading criticism from an animal, treat it like a dog who has torn up its new toy immediately. Disconnect yourself from your work when looking at these comments, or ignore them entirely. Look for people who are interested in aiding you. Look for people who give you a how and why ideas work and don't work. You can usually tell when their words come from a place of love and or respect.
There are innumerous people that disrespect D&D 5e, but there are still thousands of players that love it.
Follow the Fun
The last musing I'll post. Follow the fun. You find something that isn't a core part of your project, but find it very fun. Incorporate it, or rework your system around the idea and you'll arrive at a better place. There are countless ideas that changed drastically from the original idea because they found a fun idea that worked better than the one they were working on. I've done this several times for my game project despite having a dire attachment to my old ideas that no longer fit the project. You'll hate it at first, then you'll love it later.