Direct Stream makes some streams glitchy

Hello, I haven't had any luck googling my query for some time now so here's to hoping you guys can help.
I have a pretty beefy computer hosting my server. On this computer I have a bluray drive which I use to rip blurays to put on the server. Most of the time I can put the complete uncompressed bluray rip in my plex folder and it would direct stream perfectly fine. Except for some titles. An example is the last episode of Fringe (S5E13 - An Enemy of Fate) I have the uncompressed bluray rip and when direct stream is used on the web player, it cuts out a few seconds in and stops playing and I am presented with a play button and the Season's background. However, when disabling direct stream, it performs fine at original quality with some stuttering.
I can't find any distinct reason why this is happening and would like to use direct stream for all my titles being they are all mkvs and I don't want to convert them to mp4s because I want the optional subtitles.
All help is appreciated, thanks for reading this all!