Lets start with some basics and, before I continue, this is a criteria based answer.
Short answer: 30. (OK that sounds nuts, but hear me out). That's about the upper, upper, gonzo, bananas limit for planet definition and long-term stable orbits. I'm tempted to say 25 as an upper limit just because 30 seems too improbable.
The gist of the problem, is that a star and protoplanetary disk is unlikely to form the maximum possible number of planets. Gravity tends to clump around the larger objects. Planetary perturbations and migration make the maximum possible stable number unlikely to be reached, but with luck of a "just right" formation and some planet capture, I reached a ballpark estimate of about 30.
Long answer: let's assume we're talking about only stable planetary orbits by the definition of having cleared out their orbital path and don't cross each other's orbits. This eliminates any Trojan planets and doesn't eliminate, but makes highly elliptical orbits problematic because they span a greater orbital range.
And lets dismiss any large planetesimals that might be planet sized and any planet sized dwarf planets that cross other planet's orbits. We're only counting orbit dominating planet definition planets.
Lets also eliminate any binary or trinary systems, and only use single star systems, but the star could have some very massive planets that are borderline brown dwarf stars if you like.
Using our solar-system as a guideline and quoting from the planetesimals article above:
It is generally thought that about 3.8 billion years ago, after a
period known as the Late Heavy Bombardment, most of the planetesimals
within the Solar System had either been ejected from the Solar System
entirely, into distant eccentric orbits such as the Oort cloud, or had
collided with larger objects due to the regular gravitational nudges
from the giant planets
I'd also like to set some kind of time-limit because young solar-systems can have hundreds of large planitesimals. By about 700 million years of age, our solar-system had, for the most part, settled down into the 8, maybe soon to be 9, planets that are currently known.
A larger star probably has the potential for a good deal more than 9. But if it takes 700 million years (give or take) for a protoplanetary disk to work itself out into planets with stable, semi-permanent orbits, that puts a limit on the size of the star.
A 40 solar-mass star has a lifespan of only a million years or so before it goes Supernova. That's far too short a lifespan for planetary systems to form. Even a 10 solar mass star lasts just 30 million years or so. Again, too short.
A 4 solar mass star has a lifespan some 30 times shorter than our sun (using the 2.5 power rule, which I've also seen as a 3 power rule, but all this is pretty ballpark. Point is, a star with 4 solar masses has less than 400 million years for it's planetary system. 5 solar masses, as little as 200 million years. That's pretty close to what I'd call the minimum amount of time for a planetary system to have relevance, so I'm going to go with a 4 solar mass upper limit. The romantic notion of a star 20 times the mass of our sun, with 100 planets might make good science fiction, but it's not realistic.
A 2nd factor to consider is the mass and size of the planetary debris field. Our sun is about 99.8% of the mass of the solar-system, leaving 0.2% of the solar-system's mass to form all the planets and other stuff. There was probably more mass in the debris field originally, some of which was lost as rogue planets, rogue comets and asteroids, so the original planetary debris field might have been higher, but not all that much higher. Larger objects can cast out smaller ones. The ratio of lost debris to remaining debris shouldn't be all that high. (if anyone knows, feel free to post a comment).
The highest percentage of mass in a forming solar-system is difficult to calculate and it depends on the total angular momentum of the debris field that collapses into the spiraling disk of matter, but it's improbable that the % of mass gets too high. 1%-3% might be on the upper limit. If we go with 3% of the mass of a 4 solar mass star in the planetary disk that's about 40,000 Earth masses or about 125 Jupiter masses. That's obviously ballpark, perhaps too ballpark, but it helps to have a sense of how much stuff we have to work with.
The size of a debris field is important too. By this article, the largest debris field ever observed is about 1,000 AU in diameter (500 AU in radius) with a debris field mass of about 3.1 += .6 Jupiter masses and a central star perhaps less massive than our sun. Whether such a system could form planets as far out as 500 AU is hard to say, but I'm inclined to think that the outermost planet would form comfortably inside that debris field, not at the observed edge.
It's worth pointing out that planetary formation is a chaotic mess. A young protoplanetary disk, especially one with some 125 jupiter masses worth of material could easily form over 100 planet sized objects early in formation, but it wouldn't retain that many.
Planets perturb each other's orbits and they need space. You'd get collisions like the collection that formed our Moon and larger planets can send smaller planets any which way. No system could keep 100 planets. It's too many and would be much too unstable. There would be far fewer when a mostly stable formation is reached.
Jupiter, for example, is believed to have migrated towards the sun when our solar system was young, them migrated back outwards, called type II migration. Migrating Jupiters are both good and bad if you want a lot of planets. Jupiter's migration is believed to be the reason why there's no planets and so much empty space between Mars and Jupiter and why Mars is so small. Jupiter's migration may have also sent Uranus, Neptune out to their current distant orbits, so gas giant migration can move planets around, but it can also cast them completely out of a solar-system. The larger the gas giant, the greater a kick it can give to smaller planets.
Very massive planets are bad if you want the highest number of planets because they cause greater perturbations and demand the greatest space around them. With a lot of debris in a planetary disk, very large planets are likely to form so more debris isn't always better. What you probably want is a larger, more spread out disk, where you don't get any super massive planets, but some massive enough to push some young forming planets outwards to create more planets at greater distances. Planets are unlikely to form at very great distances, but they can be tossed out there by larger planets to very distant orbits. By tossing a number of fledgling planets outwards early in formation, the total number of planets in a solar-system could increase.
How close can the planets be to each other?
Planets don't like to be too close to each other. While we can't see small planets very well, Kepler observations seem to confirm this that very close planets are rare. When they're too close, there's orbital instability. Earth and Venus are the closest planets by multiple, where Earth is 1.38 times the distance from the sun as Venus. By this short article, a multiple of 1.4 to 1.8 times the distance between planets is suggested. Observations of exo-solar-systems find very few planets closer than 1.4 times their nearest observed neighbor, so for an entire system, a 1.4 to 1.8 multiple seems about right on average.
Planets around small stars, like Trappist 1 can get very close to each other, close enough that they can appear about moon sized from their closest neighbors, but those systems are almost entirely around small red-dwarf stars with very tight orbits, often with orbital resonance and even with very close orbiting planets, they still average out to about the 1.4 multiple or greater. Planets in a 3/2 orbital resonance that corresponds to a 1.31 distance multiple, and such resonances depend on the interactive tidal force that are only possible at close distances around smaller stars.
Kepler 36 is an oddball with two very close planets with a 7:6 orbital resonance, but building an entire solar system from planets that close seems enormously improbable. So, a key criteria to my estimate is the 1.4 distance multiple, and that's probably conservative over an entire system.
How close can the closest planets be to the star?
The heat of a 4 solar-mass star is a problem for very close planets. A 4 solar mass star, (while the luminosity changes over it's lifetime), is over 100 times more luminous than our sun, so the innermost rocky planet should probably begin at roughly about 10 times the distance Mercury is from our sun. Much closer than that and the planet would be in danger of being vaporized. So for a 4 solar mass star, 3 AU might be a good starting point. Applying the 1.4 multiple to a 3 AU starting point. A hot Jupiter might survive closer than that, but a hot Jupiter couldn't form that close, so that would probably require too much migration for our goal of highest number of planets.
so, if we start at 3 AU, and we do a 1.4 distance multiple, then our 4 solar mass star can have up to 30 planets within an orbits less than one light year, and just 32 within 2 light years, so you don't add much by doubling the distance, at least, using the 1.4 multiple.
An obvious question that follows might be, well, maybe the 1.4 multiple no longer applies at larger distances, but planets would need to grow fairly large to effectively clear out their orbit and have an effect on near-by asteroids and comets, like Neptune does and Planet 9 is believed to, so as the distance grows, you can't have mercury sized planets and define them as planets, and as the distance grows, the planets gravitational effect on each other remains consistent, so the 1.4 multiple rule should still apply even at very distant orbits.
Mercury for example, is massive enough to be a planet where it is, but if it was out past Neptune, it would be perhaps too small to clear out it's orbit. Here's a question that discusses this in more detail and it raises the problem that if Pluto was some 15-20 times more massive, the minimum mass it would need and assuming it wasn't crossing Neptune's orbit, That theoretical object would still need a billion years to clear it's orbit and that's over twice our star's lifetime and the necessary minimum size grows a larger at greater distances.
So, if we go with our one-light year proposal, an object orbiting around a 4 solar mass star at 1 light year distance has an orbital period of about 8 million years and an orbital velocity of about .23 km/s and it would have a required minimum mass to clear out it's orbit of at least several Earths. Planet 9, for comparison, is thought to have an orbital period between 10,000 and 20,000 years and an orbital velocity in the .5-.7 km/s range and a semi-major axis of about 600-800 AU or about 1/90th of a lightyear. Those numbers are all ballpark and just posted for comparison. But it points out the difficulty in recognizing a planet in a very distant orbit.
And for a planet to get that distant, it would need to be thrown out there by a larger planet, presumably undergoing type II migration or, perhaps captured from a passing star. I think you'd probably want some of both to maximize the number of planets. A star with a very large very distant planet could be effective in helping capture planets and/or debris from nearby stars that pass too close.
In both cases, the planet cast out very far or captured planets would initially have a very eccentric orbit and it would take some time for any such planets to circularize and you'd need the orbits to circularize, because a handful of eccentric orbits don't meet the planet criteria if they cross other planets.
Again, using our solar-system as a model, the outer planets, Uranus, Neptune and Planet 9 (if it exists) are all thought to have formed quite a bit closer to the sun than where they are now and migrated outwards, presumably by Jupiter.
A large star could have upwards of 100 Mercury or maybe even Earth sized objects in it's orbit, but no where close to that many that would meet the planet criteria. 30 is pushing it.
A large star capturing planets whether rogue, or capturing planets off a smaller star is certainly possible. 3 body dynamics does make planet capture possible, but there's still the problem of eccentricity and orbits crossing other orbits not meeting the criteria of a planet. If you dismiss that standard orbital criteria or a planet, then the number goes up.
So, using the criteria for a large star (4 solar masses) an innermost planet (3 AU) an outermost (1 light year - a bit of a stretch), and distance multiple (1.4 - also probably on the low side), a 4 solar mass star could have a maximum of 30 planets. If you run different criteria, you get different numbers, but I think that's a pretty good upper benchmark, perhaps on the generous side. Such a system could have a lot more objects that meet the dwarf-planet criteria, some of them even what we think of as planet sized, but meeting the complete planet criteria, 30 seems a pretty good gonzo upper limit.
Something interesting happens if you make the star smaller. If we make the star 2 solar masses instead of 4 and put the outermost planet at the inverse square law or .707 light years, not 1 light year. A 2 solar mass planet is about 12-16 times as luminous as our sun and 12-16 times less luminous than a 4 solar mass star, so the outermost planet that wouldn't get vaporized is now about 1 AU, not 3 AU. So the inner part of the planet region is 3 times closer and just 1.4 times close on the outside, so curiously a 2 solar mass star could perhaps maybe hold more planets than the 4 solar mass star. It wouldn't capture as many, on average, but the upper limit still goes up, using the same criteria to 32 or 33 for a 2 solar mass star and continues to grow as the star gets smaller.
At the same time, as stars get smaller, the upper end mass of the planetary debris field grows smaller too and the ability to capture planets drops, so I don't small stars are good candidates for the most planets, but interestingly, smaller stars with smaller protoplanetary disks could still, on average have as many planets as their larger neighbors. When James Webb starts to take a look, maybe we'll get an answer on this.
Obviously if you had all no criteria, and a star a few million light years from the nearest galaxy or massive object, you could design something with many more planets, but I'm thinking formation within a galaxy and I'm thinking that both planet capture and the right set of circumstances during formation would both play a role in maximizing the number of planets. A star that far from other stars would be unlikely to capture any planets.
Hope that's not too world-building an an answer or too long. I'll try to check it for typos tomorrow. (kinda late now).