You can develop Swift without it. I assume you mean SwiftUI / apple's sdk, which is what most people assume (a note, I'd say, to advocates of Swift who wish not to address SwiftUI: it's the biggest target, and its poor performance on it should speak volumes about the language).

You can also sometimes avoid xcode, see https://github.com/xtool-org/xtool

although, at this point, just use expo!