This is probably one of the battles of the heart for many writers. Mostly, because when we write, many ideas come to mind, and many of these cross the limits of the actual work in progress. My belief is that I do not want to chain myself to writing in a specific genre, but I recognize the need to focus on the one that speaks to me the most, at least in the beginning of my writing career. Doing this has its benefits. It will help you concentrate, grow, and polish your skills, as well as develop a name brand.
There is always one genre that attracts our interest in the early days of writing. Later on, we want to spread our wings and soar to other lands. If you are publishing in the traditional way or have a contract, this will present a challenge for obvious reasons. If you have built your brand long enough that a particular genre speaks of it, it also presents a challenge, as readers will tend to associate your brand with the genre, and this is the reason many authors use pen names when writing in other genres. For the independent author, the lines are less defined, as the pen is free to write without the need to follow protocol, except the one owed to readers. If readers expectations demand from you the work that they so much have loved, then a brand has been established, but it does not mean that an author will abandon the desire to explore other genres, and create, possibly at a different pace, works that will please other readers, and maybe, win the interest of faithful readers.
In the long run, write what you love, do it from the heart, and share it in a way that it is presented to readers in the best light, and worthy of their time and respect.