ireactions wrote:

I think this means I should redo the Pilot at 24fps.

Yeah if you did IVTC to 30fps the file will be 30p @ 1:1:1:2 cadence, in other words every 5th frame will be a repeat.  It looks a bit janky imo and I don't like it when streaming services do it that way, but it's easy to fix with TDecimate to remove 1 in 5 duplicates.   
In MPC-HC you can set a hotkey to framestep so you can confirm exactly what you've got.

ireactions wrote:

unless pneumatic could offer a new version that keeps all the improvements (sharpening, smoothing out the interlacing issues) but has a more pneumatic-approved way of limiting the framerate to 30fps?

I shouldn't have done it in 60fps in the first place - that was just to preserve the rare occasional 60fps elements as I'm a stickler for that. 

After you reported the issue with 60fps not playing on your device, I rendered another one at 24fps here (script included): https://sliders.tv/bboard/viewtopic.php … 463#p14463

The "# 24fps IVTC" comment is the relevant bit.  The only difference with the 60fps version was to do a DoubleWeave() before the TFM(), everything else was the same.

RussianCabbie_Lotteryfan wrote:

margins are slim and if they paid too much on the licensing fee in this one and overestimated demand, it would have probably put them in financial loss territory.

Yeah of course, that's why it would have to be the sale price that increases by $5-$10 to pay for the extra discs (assuming my estimate of disc costs was correct... for all I know it may be way off).

RussianCabbie_Lotteryfan wrote:

As far as bitrate -- could that just be a product of them trying to fit more episodes per disc?   Because even an added disc cuts into Mill Creek's slim profit margin.

I guess it depends how much it costs for them to manufacture an extra disc.  Suppose we wanted double bit rate, say 7mbps instead of 3.5, then a 20 disc series is going to become 40 discs.   If we allow +$5 for the total cost of the product, that would allow for 25c per disc.   I'm not sure how much a DVD disc costs to manufacture but I'd imagine most people would be willing to pay an extra $5 for better picture quality.  Maybe +$6.60 @ 33c/disc or +$8 @ 40c/disc?

RussianCabbie_Lotteryfan wrote:

The thing about Mill Creek was their business model is offering very casual fan of a series, essentially 3rd, 4th run, of a disc set (that they didn't buy at 1st, 2nd run) the series at a super affordable price, and usually in "bargain bin/shelf" type placements in brick & mortar stores.   So they are trying to keep their costs on the production side as low as possible (as few discs at possible).   They may have lazily encoded the stuff, you'd know better than I, but they really are not meant to be a premium distributor, and therefore are less equipped to "getting it right" in the look of the content, and won't spend much time fiddling before moving onto the next title.

As far as I can tell Mill Creek isn't in the business of colour grading. If I had to bet money on it I'd say they are probably just doing the disc authoring & printing/pressing.  I could be wrong.

I wouldn't even mind the lower 4.5mbps bit rate of the Mill Creek release if they had at least used the same masters as the Universal version.  As an example I just bought this knowing full well that one of the reviewers has mentioned the reduced bit rate due to the lower disc count compared to the original 2008 CBS release.  Although if it has serious mastering errors it's going back on ebay.

You'd think over time the image quality would get better, but with DVD they are actively making it worse.  Which really hurts since DVD is already suffering at 720x480 standard definition, and there is often no better option due to the streaming services being even worse (except of course for shows that were remastered in HD at great expense to the studio).

Given the surprising popularity of DVD some 28 years after its conception, maybe they should consider coming out with a "DVD+" spec for better quality video while retaining backwards compatibility and cheaper production costs compared to bluray.   For example the DVD+ spec could specify a minimum bitrate and have predefined encoder profiles agreed upon by industry experts.  They could retire the interlaced format and require all video be stored on disc as progressive, and if generated from an interlaced master it would need to comply with a certain standard of deinterlacing & ivtc defined by industry experts.   The raster size should also be expanded to 852x480 for NTSC and 1024x576 for PAL to retain 4:3 pixel density at 16:9.

I feel like I got extremely lucky with an ebay seller in the US who was willing to do international shipping on their Universal NTSC discs.  Most sellers wouldn't bother shipping internationally for such a low dollar value item, so I am very grateful to that seller.   It took about a month to arrive.   I've currently lent it to a family member and told them to be extremely careful with the discs as they are irreplaceable.

There is an encode of the Universal discs on channel BT.  The quality is ok, but I still don't like being forced to watch someone else's encode for something so beloved.   It's like it has their fingerprints on it or something.  Although that one isn't too bad compared to some others - still highly watchable - and much MUCH better than the load of crap shoveled out by Mill Creek - they're the real bad guys here, am I right people?

ireactions wrote:

In other news: my Android TV cannot play pneumatic's 60fps, 8mbps file properly.

Those ones I did at 24fps with pretty standard x264 settings (-tune film -pix_fmt yuv420p -b:v 8000000).  Not sure why your box can't play it... out of curiosity could you try this 8mbps x264 that someone else did so I can learn hopefully about what makes files more compatible with devices - https://drive.google.com/file/d/1Q2o0Cz … sp=sharing  (353MB)

RussianCabbie_Lotteryfan wrote:

it's likely because they likely have global settings for all content they transcode, if they do any processing.  kind of a general laziness and also a practicalness to it.

Sounds about right.   This is actually the problem I've been trying to solve the past few months with an Avisynth script which I'm calling "adaptive IVTC" (90% complete).   Basically you can throw any content at it and it will give you the best conversion from interlace to progressive without having to configure anything.   Optionally you can set it to a profile to tell it more info about how much of the content is film-based but you don't have to, it will still work at the default setting with high accuracy, and for the scenes gets it wrong it errs on the side of deinterlacing with QTGMC antialiasing which still looks pretty decent and in a double blind test can be hard to tell apart from IVTC.

RussianCabbie_Lotteryfan wrote:

someone tell me how the streaming quality of this is acceptable?

this is what NBCU is distributing:

https://therokuchannel.roku.com/watch/e … 586f4c077f

I can't view it due to geoblock, but in my experience the technicians who prepare the videos for streaming services don't seem to understand how older video formats work, and the DVD is almost always better quality. 

Recently bought Alfred Hitchcock Presents on DVD because Peacock's streaming version is downres'd to 240p due to the field discarding deinterlacer they used, and has a baked in frame stutter (1:1:1:2 cadence) as a result of that process.

ireactions wrote:

Hmm, yes. At 4mbps vs 10mpbs side by side, I definitely see that the 4mbps version has lost some of the fine texture that the sharpening brought in, but it still looks better than the DVD.

But it's a transcode of a transcode of the DVD, so it will look considerably worse!

This is why I'm such a stickler about using Avisynth in realtime to watch episodes, as it's the only way to do video processing without any generation loss.   It also saves storage space - 5mbps for the DVD vs 10mbps for the x264.  And no need to burn your CPU for days to render the files.

The only issue with realtime playback is you need enough CPU power if you want to do a lot of advanced AI upscaling in realtime.  However Avisynth+ supports multithreading & if I enable it by putting a Prefetch(8,6) at the end of the script I posted above, it manages to play in realtime at around 90% CPU load on an i5-4570 (4 cores 4 threads @ 3.2Ghz) which is not a very powerful machine at all.   btw Prefetch(8,6) means use 8 threads and prebuffer 6 frames in advance.  I found that even on my 4 core machine, 8 threads still loads up the CPU better.

ireactions wrote:

I'm going to try re-encoding at a 4,000 kb bitrate and see if the file is playable.

Please don't!  It's already bad enough going analogue -> DVD MPEG2 compression -> X264 compression. Another round of X264 compression on top of that will surely damage the picture quality.  I can see slight visible differences between the MPEG2 & X264 even at 10mbps.   I'll rerender the clip at 8mbps, hopefully it should be compatible.

edit, link: https://drive.google.com/file/d/1np2d_d … sp=sharing

Full script used to make the above clip:

file = "S01E10 Luck of the Draw.mkv"

# Create source clip
video = LWLibavVideoSource(file, stream_index=-1, repeat=true, cache=true)
audio = LWLibavAudioSource(file, stream_index=1, av_sync=true, cache=true)
AudioDub(video, audio)

# Cut requested scenes
  Trim(0, 7200)      ++ BlankClip(last, 30) ++
\ Trim(10196, 18249) ++ BlankClip(last, 30) ++
\ Trim(61702, 65834)

# 24fps IVTC
TFM(mode=0, slow=2, PP=3, clip2=bwdif(field=-1, thr=2, edeint=nnedi3(field=-1)))
TDecimate(mode=1, cycle=5, cycleR=1, hybrid=1, viddetect=2, vidthresh=4.5,
\         denoise=true, chroma=false, hint=false, display=false) 

# QTGMC antialiasing - not used for this episode to preserve sharpness
# QTGMC(InputType=1, TR2=1, preset="slow", EdiThreads=2, Sharpness=1.0, Rep0=13)

# Convert to HD colour
z_ConvertFormat(                        
\ colorspace_op="601:601:170m:full=>709:709:709:full",
\ resample_filter="spline36",
\ dither_type="ordered",
\ interlaced=true )

# Thin edges
aWarpSharp(10.0, blurlevel=1) 

# Sharpen
CAS(sharpness=0.85, y=3, u=2, v=2, opt=-1)

# AI upscale to 2880x1920
nnedi3_rpow2(rfactor=4, nns=1, nsize=0, cshift="Spline36Resize")

# Downscale to 1440x1080
Spline36Resize(1440, 1080)
ireactions wrote:

When you say you "prefer to let as much resolution through", is that via the magic of QTGMC? Or do you mean using QTGMC minimally and only for combed frames?

QTGMC softens the image slightly...

http://avisynth.nl/index.php/QTGMC#Sharpness wrote:

The core of the algorithm involves a binomial smooth to remove shimmer. So the result needs to be resharpened to counteract this blur. The main setting Sharpness defaults to 1.0, which is a level designed to retain the sharpness of stable areas. However, this level can cause moving areas to be oversharpened so you may wish to reduce the value depending on source.

We can control the amount of smoothing/softening by adjusting the TR0, TR1 & TR2 parameters, but it still won't retain as much texture detail as not using QTGMC at all.

Did you mean the PAL version?   Because I didn't do anything to it.

Stunt actor lol

https://i1.lensdump.com/i/kCmiak.png

ireactions wrote:

I would say that the scenes that stand out to me as blurriest in "Luck of the Draw" are the wideshot during Wade's voiceover in the teaser, the horseback riding shots, the fishing sequence, Wade and the Professor arguing about walking Henry the Dog

Here's a clip of those scenes:  https://drive.google.com/file/d/1WcV86w … share_link

I think that's about as sharp as I can manage in Avisynth without going overboard with the sharpening.

The relevant filters were:

# thin edges
aWarpSharp(10.0, blurlevel=1) 

# sharpen
CAS(sharpness=0.85, y=3, u=2, v=2, opt=-1)  

# AI upscale 
nnedi3_rpow2(rfactor=4, nns=1, nsize=0, cshift="Spline36Resize")

# 1080p
Spline36Resize(1440, 1080)

I don't know anything about cameras but it seems like on some of the shots the depth of field is set to focus not on the actor's faces but on other objects slightly in the foreground or background?  Or something like that?

Here's a clip of Luck of the Draw PAL DVD (remuxed, no transcoding, no processing at all)

https://drive.google.com/file/d/1xsgovx … share_link

You might notice their voices are chipmunked due to the 24->25fps speedup - this can be reversed with AssumeFPS(24000, 1001, true) to slow the video and audio back down to 23.976fps same as the NTSC version.

ireactions wrote:

I have to ask: at this point, would it be better to just stick to TFM and TDecimate on all the episodes and just omit QTGMC due to how it smooths out an already blurry video image? Would it be better to accept a litte aliasing and moire here and there? In my case, I prefer noise to smoothness.

Well it's entirely up to you and how you want it to look.   Personally I prefer to let as much resolution through if the flaws aren't visible at normal viewing distances on a TV.   Usually this means the image looks bad up close on a PC monitor like those Eggheads screenshots I posted.

There's an Avisynth filter called aWarpSharp which can do edge thinning so I'm going to have a play with that in combination with CAS and nnedi3 to see if it can do anything special with these softer looking episodes.

If we were pros doing a restoration project we'd probably do per-episode and per-scene processing...

clip = "C:\Video\Sliders\Season 1\S01E10 Luck of the Draw.mkv"

# apply this processing to all episodes in season 1
if (FindStr(clip, "\Season 1\") != 0){            
    SubTitle("Now playing: Season 1")
}

# apply this processing only to Summer of Love & Luck of the Draw    
if  (FindStr(clip, "S01E03") != 0
\ || FindStr(clip, "S01E10") != 0){    
    SubTitle("Now playing: Summer of Love or Luck of the Draw")
}

# apply QTGMC processing only to opening street scenes of Last Days
if  (FindStr(clip, "S01E06") != 0){

    scene1 = Trim(0, 193)       # frames 0 to 193 
    scene1 = scene1.QTGMC(InputType=1, preset="slow", EdiThreads=2, Rep0=13)
    scene1 = scene1.SubTitle("Now playing: opening street scene with QTGMC")    
    
    remainder = Trim(194, 0)    # frames 194 till end
    scene1 ++ remainder         # join scene 1 and remainder 
}

ShowFrameNumber()   # overlay the current frame number
                    # can set a hotkey in MPC eg. ctrl+left/right to framestep

Also there seems to be more film jitter in Luck of the Draw.  That thing where the film frames are jittering slightly in the spool causing the whole image to move around slightly by a few pixels.  QTGMC repair was automatically correcting that with its stabilisation so you might notice it in that clip since I disabled QTGMC.  If  that bothers you it's possible to correct it with something like

# Image stabilisation - http://avisynth.nl/index.php/DePan
mdata = DePanEstimate(last)
DePanStabilize(last, data=mdata, initzoom=1.025) 

I think initzoom=1.025 means crop 2.5% around image edges as buffer zone for image stabilisation.

ireactions wrote:

Interesting. Is the broken TFM v1.0.27 producing 240p on all the episodes or just "Luck of the Draw"?

All episodes.   I think 1.0.27 isn't detecting the alternating field order produced by DoubleWeave() - I've reported it to the author.   I just happened to update to 1.0.27 today so I mistakenly thought it was just this episode.

Anyway I was looking at Luck of the Draw some more and I'm starting to see what you mean - it does look quite soft and blurry at times.  Some shots look ok though, others look like they've been through some kind of degradation of sharpness

Unsharpened , CAS sharpening , MadVR sharpening

CAS seems to bring out the contrast in the ground's brick texture more, but has fatter edges (MadVR sharpening includes edge thinning).  I think CAS + thin edges might look nice.

To counteract the softness I tried skipping the QTGMC repair to try and let more detail through at the expense of more noise and aliasing, and then readjusting the CAS sharpening a bit lower to compensate (80% instead of 90%).  Clip: https://drive.google.com/file/d/1JiHrjX … share_link

A better strategy might be to thin edges before NNEDI3 AI upscaling.  That way NNEDI3 has thin smooth edges to work with and can make better guesses at what the higher resolution edge would look like.   Whereas if you give it a soft fat edge it doesn't really see that as a high contrast edge and won't do much with it.

The latest version of TFM (v1.0.27) appears to be broken when feeding it 60fps via DoubleWeave(), which is what the script I posted here uses.   

For some reason v1.0.27 fails to fieldmatch a 60fps doubleweaved input, whereas the previous version (v1.0.26) succeeds.  This is the version I've been using since last year.     

I tried using all the fieldmatching modes but they all produce the wrong result - either failing to fieldmatch or producing the wrong output cadence (2:2:2:4 instead of 3:2).

Failure to fieldmatch means combed frames which triggers deinterlacing, so if you were using v1.0.27 you'd regularly be dropping to 240p on moving objects.

Think I've identified the issue - setting TFM(display=true) shows the 60fps TFM is returning combed frames with this episode for some reason, so it's actually deinterlacing most frames, resulting in 240p on motion.    The standard TFM.TDecimate to 24p doesn't have the issue.   Might be something to do with the field order, I'll get back to you...

ireactions wrote:

I am wondering if pneumatic might turn his video powers to "Luck of the Draw" and consider that episode separately from 1.02 - 1.08.

1.02 - 1.08 look pretty good to me after pneumatic's scripts with TFM and TDecimate and QTGMC and the contrast adaptive sharpener and the nnedi3 upscale and the spline36 downscale and pneumatic's tireless spit and polish. But "Luck of the Draw" is a unique case. It is uniquely bad.

Had a look just now with just TFM.TDecimate to see the raw frames, and I can't really see anything exceptional apart from the 1:1 sections at around 11:00 - those will look terrible with stutter and combing as TFM wasn't designed for deinterlacing 1:1 sections (still working on my adaptive deinterlacing script to solve this... around 1500 lines so far).

The field alignment seems no worse than something like Eggheads, and there's definitely 480p to recover from the source after IVTCing with TFM.TDecimate, judging by the smoothness of diagonal or curved lines.

I am sharpening the image with MadVR though - subjectively I find standard definition unwatchable at 0 sharpness.  For maximum effect sharpening should be applied before upscaling otherwise the sharpener is looking at soft upscaled pixels which don't sharpen as well.

Is the issue that the image doesn't look sharp enough?   Or too noisey/grainy?

Also it's 60fps - more frames to process.

ireactions wrote:

Interestingly, the sharpener is so processor-demanding that I can't play the file by running the AVS script in Media Player Classic.

Hmm that's weird, CAS is almost free on my desktop system (i5-4570 4 cores @ 3.4ghz).  AMD claims it's "highly optimized".

QTGMC and nnedi3_rpow2 are very CPU intensive... try commenting everything out except CAS, toggle CAS on/off and compare CPU before/after (give it about 10 seconds of playback for buffers to fill... after that my CPU usage settles around 7-10%). 

If you're still getting high CPU try playing with the opt parameter (opt=0/1/2/3) in case the default setting (-1) isn't correctly autodetecting your CPU capabilities.  If it's a mobile CPU that might have something to do with it.  The slowest should be opt=0, fastest should be 1 or 2.

Avisynth can also suffer from bottlenecks depending on what order you call filters in, eg. if there is a slow filter higher up the script, that can grind everything under it to a halt, but I don't think that's an issue here.

edit: also QTGMC's stronger smoothing (TR2=3) is very expensive on my system - I'm seeing a 30% increase vs default (TR2=1).

RussianCabbie_Lotteryfan wrote:

I might tone down the level of sharpness to 70-80 percent to better support closer distance viewing, but this looks like another great tool.

Well, I set the sharpness for that one on my PC monitor and it looked "just right" there, but on the lounge TV it looks overly sharp.   

RussianCabbie_Lotteryfan wrote:

I'm curious about whether Pneumatic notices the dreary color tone to the content.  And if there are algorithms that adjust for that.

That particular episode looks ok to me.   If you want to add some saturation, see ireaction's script for how to do it with the Tweak() function.

RussianCabbie_Lotteryfan wrote:

Is it also possible that maybe we are better off not converting to HD color?

The colour conversion doesn't actually change the colours that you end up seeing, it just ensures you get the same original colour when the media player decodes it with the HD colour matrix.   Otherwise you would be getting SD colours decoded with HD colour matrix and the colours would be wrong.

pneumatic wrote:

Avisynth's built in Sharpen() is poor imo, but there are other third party sharpeners here.  Although many are 10 years old and don't support the latest 64-bit version of Avisynth.

Contrast Adaptive Sharpening seems to be decent - ported from AMD FidelityFX.

Preview: https://drive.google.com/file/d/10HEOoD … share_link

Would recommend viewing at normal viewing distances.

Script:

# convert to HD colour
z_ConvertFormat(                        
\ colorspace_op="601:601:170m:full=>709:709:709:full",
\ resample_filter="spline36",
\ dither_type="ordered",
\ interlaced=true )

# 60fps IVTC
DoubleWeave().TFM(mode=0, scthresh=100, micmatching=1, ubsco=true, mmsco=true,
\   display=false, slow=2, PP=3, metric=0, cthresh=9, MI=80, hint=false,
\   clip2=propdelete("_FieldBased").bwdif(field=-2, thr=2, edeint=nnedi3(field=-2)))

# QTGMC repair - high strength, no sharpening
QTGMC(InputType=1, TR2=3, preset="slow", EdiThreads=2, Sharpness=0.0, Rep0=13)

# Contrast adaptive sharpening - 90% strength
CAS(sharpness=0.90, y=3, u=2, v=2, opt=-1)

# 4x AI upscale to 1920p
nnedi3_rpow2(rfactor=4, nns=1, nsize=0, cshift="Spline36Resize")

# Downscale to 1080p
Spline36Resize(1440, 1080)
RussianCabbie_Lotteryfan wrote:

Looking at the screenshots, I think you once called it the "teeth chattering effect"?

Combing, mice teeth, interlacing ... these words are all referring to the same phenomenon where field 1 and field 2 have 2 different images in them (as they are supposed to have by design).   This is definitely not the case in any of the screenshots, but there is field alignment issue and that will be made more visible by sharpening.

How I like the the image to look is not necessarily what anyone else will like.  Some release groups add their own sharpening to their videos and imo that is a bad decision - better to let the user decide how much sharpness they want (unless it's artistic intent of the director/producer of the content).


ireactions wrote:

The problem scenes...
"Prince of Wails" with the humvee

The PAL version is softer and doesn't have those high contrast 1px pattern on the Jeep grille to begin with - they've been smudged out by whatever equipment Universal used to create the PAL version.  If the PAL version was sharper and resolved all the resolution in the grille, you would get the same issue with TFM detecting the grille as combing and switching to bwdif for those frames which causes the issue with moving parts of the image dropping to interpolated 240p with aliasing / twitching on moving pixels. 

Perhaps you would prefer some stronger QTGMC repair smoothing:

# default 
QTGMC(InputType=1, preset="slow", Sharpness=0.0, FPSDivisor=1, EdiThreads=CPUcores/2)

# medium smoothing
QTGMC(InputType=1, TR2=2, preset="slow", Sharpness=0.0, FPSDivisor=1, EdiThreads=CPUcores/2)

# high smoothing
QTGMC(InputType=1, TR2=3, preset="slow", Sharpness=0.0, FPSDivisor=1, EdiThreads=CPUcores/2)

# max smoothing
QTGMC(InputType=2, TR2=3, preset="slow", Sharpness=0.0, FPSDivisor=1, EdiThreads=CPUcores/2)

Although the image will look pretty soft at max smoothing so you might need to set Sharpness to a nonzero value.



ireactions wrote:

"Fever" in the pharmacy

If it's this scene then it's the same issue as the Jeep grille
https://i2.lensdump.com/i/EMasNm.png


ireactions wrote:

"Eggheads" with the houses

Again same issue as the Jeep grille - TFM sees legitimate image data as combing. 



ireactions wrote:

"Last Days" streets

That section has a completely different issue - the studio deinterlaced that section to 30p using their own bob-weave filter similar to bwdif in Avisynth.  Because the camera is wobbling ever so slightly it keeps creating small amounts of motion that keeps tripping their bwdif into bob (interpolation) mode for moving parts of the images - all the twitching / flickering lines are pixels dropping to 240p for a split second.  QTGMC repair mode works wonders on that section - video here




ireactions wrote:

And my Universal question for pneumatic was really: the videotape files are fuzzy and blurry. You've been able to clear up the distortions significantly and add a bit more detail. Do you think that the files can become sharper via your scripts or other third party sharpeners? Or are we pretty much at the limit of how much these files can be improved?

It's hard to say as it depends on the subjective quality of what you prefer in the image.  If the PAL/Turbine version looks better to you then you'd want to investigate smoothing the image more with those QTGMC settings, and then if the image looks too soft, adding some sharpening.

Another option is to just QTGMC deinterlace instead of TFM+TDecimate but you'll get moire on high contrast 1px patterns.

ireactions wrote:

Why don't the Turbine files have the aliasing? The jagged edges? The comb lines?

I'm not really seeing that in the screenshots I recently posted - is there a particular episode which is problematic?

Combing should not be an issue if you have TFM's post processing enabled and an MI value of around 80 to 100.  Unless it's 1:1 cadence stuff like the Mindgame sequences in Eggheads - that's what I'm currently working on to solve using a latching system where once 2 or 3 combed frames start coming out of TFM, the rest of the scene gets deinterlaced regardless.  And then after n latched scenes, declare the content type as video and start every scene deint latched until the latch is broken the other way, and vice versa for film based scenes.

ireactions wrote:

do you think your future scripts can make the video sharper and clearer?

Avisynth's built in Sharpen() is poor imo, but there are other third party sharpeners here.  Although many are 10 years old and don't support the latest 64-bit version of Avisynth.

I haven't played with any of them as I do all my sharpening in the video renderer (MadVR) which runs much faster on the GPU and has good customisation like "thin edges" and "enhance detail" etc. which I personally like.  Here are some comparisons...

1. Original , sharpened
2. Original , sharpened
3. Original , sharpened
4. Original , sharpened

edit: the above are all from Eggheads but none seem to be visibly affected by the field alignment issue.  Here is a shot from the same episode where I think the field misalignment just happens to be occurring where his eyes are positioned: https://lensdump.com/i/ES6w8K

ireactions wrote:

I think the unwillingness to redo the effects shots is because the STAR TREK: THE NEXT GENERATION HD blu-ray was a sales failure. It was a high budget remastering with rebuilt effects that came out just in time for streaming to wipe out the home video market. Paramount and CBS lost a ton of money.

I do like the BD of Next Gen, but I'm definitely keeping the DVD version as a reference of how the show originally looked - soft focus, dreamlike quality, yellowish skin tones, lo-fi audio etc.   It's a different experience.  The BD is of course objectively superior in all possible ways, but it's still a mod.  A great mod that I would enjoy on my second playthrough.

ireactions wrote:

I thought the pneumatic Avisynth+ script had dealt with the jagged edges in this Pilot shot when reviewing a 540p version, but looking at it in 1080p output, the jagged edges are still there:
How would pneumatic address it?

Just checked that scene on my end and it seems the culprit is QTGMC's sharpness=1.0, as setting it to 0.0 or commenting out QTGMC gets rid of the aliasing on the radiator grille.   

Usually radiator grilles cause aliasing due to the grille looking like combing artefacts to TFM, triggering deinterlacing and its associated aliasing on moving objects, but even at MI=40 it's still not getting detected as combed so it's not that.

Looks like QTGMC sharpening should be avoided then.  I do all my sharpening in MadVR so that's probably why I didn't notice it - sorry.

ireactions wrote:

There's certainly a cooler, bluer look to the LOIS AND CLARK HD rescan with a higher brightness level. And all the effects are just the videotape sequences that have been stretched, so any time there's any kind of video effect, it goes from HD to terrible looking SD. I only saw the first season, but it looks like they didn't even try to upscale the SD footage.

I know EXACTLY what you're talking about.  Funny thing, the special effects actually look BETTER on the DVD.   They are animated at a higher frame rate (30/60fps on many effects which gets bodged down to a stuttery 24fps for the HD version) and the colour of the effects is better matched to the scene they are superimposed onto.

Very interesting results with Topaz, thanks for sharing that. 

What it does to facial texture and grain is quite remarkable.  The Artemis one in particular looks the most impressive to me in terms of trying to reconstruct a HD image.

Subjectively I'm not sure if I would use it on a first playthrough.  Probably on my second playthrough I'd use it.  Kind of like completing a game before trying out mods.

It's weird because sometimes I prefer HD remasters, but not all the time.  I've got the Lois and Clark HD remaster and I don't like it.  The new tonemapping & colours just doesn't feel right to me.   So I bought the DVDs which are low res, have analogue artefacts like dot crawl, but I seem to prefer the colours and lighting.   The HD remaster feels a bit sterile and cold.  The night scenes have a lot more shadow detail which takes away the mystery a bit.   On the other hand I seem to like the Macgyver HD remaster.   Maybe cause it's shot more outdoors?   Something about the indoor shots feel more like they're on a set.

ireactions wrote:

Is it possible that in terms of blu-ray player playback, the Turbine files are better suited to progressive scan and upconverting than the Universal files?


Yes!   My Sony DVD player for instance, when playing a PAL 2:2 cadence DVD (pretty much all PAL DVDs) it inverse telecines (weaves) to 576p.   But with NTSC 3:2 cadence DVDs like Sliders it just deinterlaces using BWDIF style deinterlacing (weaving only static elements, but anything that moves drops to 240p).  So I think that could explain the difference you were seeing.

Also for NTSC DVD's there are ones that are authored as "soft telecine" where the video is stored as 24p on the disc and there are "repeat field" flags in the video stream telling the DVD player when to add fields to make a 3:2 pattern for 480i output.   On these discs my Sony DVD player appears to ignore the repeat flags and just outputs 24p.  But Sliders DVD uses "hard telecine" where it's stored as 30i on the disc with the 3:2 pattern baked in, and the only way to recover 24p is to use inverse telecine (eg. TFM) which my Sony DVD player doesn't appear to support, so it just deinterlaces every frame BWDIF style.

I'm having the Lensdump issue as well - seems to be a problem with their site.  Disabling my adblocker seemed to fix it but now it's playing up again.  The reason I use Lensdump is cause they don't compress the images.   I'd imagine they are aware of the issue and will fix it soon.

Thanks to a forum member for lending me their copy of Turbine BD, here are some comparisons of NTSC DVD vs PAL DVD vs Turbine BD.

All are remuxes from the original discs, all were inverse telecined with TFM to 576p then Bicubic upscaled to 1080p in Photoshop (otherwise the screenshots are tiny).

101 [NTSC DVD] [PAL DVD] [Turbine BD]
104 [NTSC DVD] [PAL DVD] [Turbine BD]
206 [NTSC DVD] [PAL DVD] [Turbine BD]
305 [NTSC DVD] [PAL DVD] [Turbine BD]
405 [NTSC DVD] [PAL DVD] [Turbine BD]

According to MediaInfo the Turbine BD is MPEG2 576i25 @ ~5mpbs, the same ancient technology as the PAL DVD (MPEG2 released in 1996).

imo the NTSC DVD is still the best and the PAL DVD is perhaps slightly better than the Turbine BD.

ireactions wrote:

Tweak(sat=1.45) #Sorry, pneumatic, I know you don't approve of this.

lol   

Honestly I just think it's great that we have these kinds of tweaking capabilities - I'm constantly editing my .avs files and tweaking my MadVR tags to get things looking "better" for certain seasons of shows and it's great.

ireactions wrote:

If the field misalignment on 1.02 - 1.09 is not too severe, I shudder to think what is severe...

Mill Creek.

ireactions wrote:

I ran pneumatic's script on the Pilot (Universal) along with pneumatic's nnedi3 settings to output it to 1080p.

Which script was it - this one?  Because that one doesn't repair the field alignment - only QTGMC in deinterlacing mode does that (InputType=1 is repair mode, InputType=0 is deinterlacing mode).

Maybe the improvement you're seeing is the antialiasing provided by QTGMC repair mode?  It does work wonders at times.

ireactions wrote:

What exactly is the difference between deinterlacing interpolation and deinterlacing weaving?
You've specified that QTGMC doesn't weave but instead interpolates.

Interpolation is really just a synonym for upscaling - taking some samples like pixels and making a guess at what the pixels in between them might be based on the surrounding pixels.

The simplest form of deinterlacing is Bob deinterlacing which just takes each 720x240 field (60 of these per second in a 30fps interlaced file) and upscales them to 720x480, typically using Bicubic or Bilinear resizing in the vertical direction.  If you want to see what it looks like type Bob() in Avisynth.  The result is soft and flickery, but frame rate is perfect and combing artefacts are impossible.

QTGMC deinterlacing is like Bob deinterlacing but NNEDI3 upscaling each field from 720x240 to 720x480, followed by some advanced techniques to smooth and antialias the image.  I believe it looks at the previous/next field as part of this process.  But it never weaves fields together - a test pattern of alternating 1px white/black horizontal lines always becomes solid white/black in each output frame, just like with Bob.  I would characterise QTGMC deinterlacing as the best possible version of Bob deinterlacing.  It tremendously repairs the field misalignment issue in the Mill Creek release.

QTGMC "repair" mode doesn't deinterlace at all, it just looks at progressive frames and applies its antialising magic to it.  Film grain is slightly suppressed but the 1px black/white lines are fully resolved.  Field misalignment issue is not repaired. 

ireactions wrote:

My (shaky) understanding is that weaving is when the deinterlacer combines the even and odd fields and then doubles that frame in order to meet the video's existing framerate (or the framerate would otherwise be cut in half).

It depends what the cadence is.  1:1 cadence is where every field is a temporally unique image captured from a new moment in time, like the Mindgame scenes in Eggheads.  Sports, news, talk shows and soap operas tend to use it.  If we weave every pair of 2 fields together on a 1:1 sequence the resulting image is combed:

top field        A-C-E-G-I-K-M-O-Q-S
bottom field     B-D-F-H-J-L-N-P-R-T
combed?          Y Y Y Y Y Y Y Y Y Y

(letters indicate temporally unique image)

To avoid combing on 1:1 we can use Bob, but a better solution is BWDIF (bob-weave deinterlace filter) which uses Bob deinterlacing for moving parts of the image and weaves any static parts.  Result is 480p60 same as Bob.  From memory this method of deinterlacing became mainstream around the late 2000's and was touted as "per pixel deinterlacing".

If the cadence is 2:2 (every 2 fields = a temporally unique image) like the spinning earths in the intro, weaving them results in 480p30:

cadence          2 2 2 2 2 2 2 2 2 2
top field        A-B-C-D-E-F-G-H-I-J
bottom field     A-B-C-D-E-F-G-H-I-J
combed?          N N N N N N N N N N

If the cadence is 3:2 (99% of the show) we get:

cadence            3 2 3 2   3 2 3 2
top field        A-A-B-C-D-E-E-F-G-H
bottom field     A-B-C-C-D-E-F-G-G-H
combed?          N Y Y N N N Y Y N N

TFM field matching applied to the above results in:

cadence            4 2 2 2   4 2 2 2
top field        A-A-B-C-D-E-E-F-G-H
bottom field     A-A-B-C-D-E-E-F-G-H
combed?          N N N N N N N N N N
duplicate?         Y         Y 

TDecimate can then remove the 1 in 5 duplicates to produce 480p24.

So we have a mix of 480p24, 480p30 and 480p60 coming out of the original 480i30 container.  If your goal is to preserve these original rates, they would fit best inside a 480p60 container, which is what I'm currently working on in Avisynth.  99% of the show is 3:2 though, so most people would probably be fine with putting it all inside a 480p24 container and tolerating the odd bit of stutter/combing on 2:2 and 1:1 sequences.


ireactions wrote:

My (shaky) understanding is that when dealing with video like SLIDERS 1.02 - 1.09 where the fields were scaled separately and misaligned, progressive scan produces all ugly visual anomalies on the Universal discs (as does weaving).

Yes, although I think the field misalignment issue is not too severe on the Universal NTSC discs.

ireactions wrote:

My (very shaky) understanding from pneumatic's script and posts: it's TIVTC/TFM for detelecining, and then QTGMC is deinterlacing and interpolating any combed frames that remain (and there seem to be a lot in 1.02 - 1.09).

Yep that's right, except it's using BWDIF for deinterlacing and then applying QTGMC "repair mode" to all frames. 

If we use QTGMC for deinterlacing it looks bad cause QTGMC deint frames are so visually different in terms of the shapes of object outlines that it's noticeable when a QTGMC deint frame is occasionally substituted in.  Kind of looks like object outlines are popping slightly larger.     

I've been busy working on a script with presets to choose things like whether you want to output 24p/60p, whether to apply QTGMC repairing to deint frames or all frames (in case you don't want it suppressing film grain all the time), specifying how much of the source contains video camera content like the Eggheads episode so it uses lower combing detection thresholds for those scenes, and some other things.

ireactions wrote:

Requests for pneumatic:
I'm wondering if pneumatic would recommend any further adjustments to the Avisynth+ script for nnedi3 upscaling. I've currently got:

nnedi3_rpow2(rfactor=4,cshift="GaussResize",fwidth=1440,fheight=1080)

Seems to be fine, although I'm not familiar with GaussResize.  The wiki says it has an adjustable sharpness parameter p so if you wanted to control final sharpness you could do eg.

nnedi3_rpow2(rfactor=4, cshift="GaussResize")
GaussResize(1440, 1080, p=30.0)    #1.0=blurry, 100.0=very sahrp
ireactions wrote:

Also, what Avisynth+ command might you recommend for Lanczos? I could do two sets of encodes for comparison. I wonder if Lanczos, even if it isn't as sharp, might be more consistent between closeups, mediums and wide shots.

I'm just using the standard LanczosResize() with default settings which I believe is 3 taps. 

I don't normally do the upscale in Avisynth as it uses CPU so I prefer to let the media player do it on the GPU.  There's nothing particularly special about Lanczos, I just like it cause it's inexpensive and doesn't look objectionable to me.  Some people complain it's too ringy (overshoot artefacts) but I don't seem to notice it.

ireactions wrote:

Also also: my TV is a 4K display. Is it possible that a 1080p video being scaled to 4K is causing video quality loss? I wonder if scaling these episodes to 4K would be worthwhile or if we've hit a point of diminishing or flat-out non-existent returns.

Depends on how good your TV is at upscaling 1080p to 4k - I'd expect it should be perfectly fine & I'd be surprised if it was bad.   Although current model LG OLEDs have surprisingly low sharpness at max sharpness setting.

ireactions wrote:

I'm also experimenting with Lanczos4Resize or nnedi3 to bring it to a 1080p container. Which one would pneumatic use?

I'm using Lanczos or Spline36 only because my HTPC isn't powerful enough to do TIVTC + QTGMC repair + nnedi3 upscaling in realtime.    If you have the power to spare you might as well use nnedi3 as its very good at antialiasing high contrast edges - https://imgsli.com/MTU5MTAz

For future reference all TIVTC and QTGMC dependencies are in a zip file here & I've updated all previous references in this thread to point to it as well.

ireactions wrote:

Gives me this error:
https://i.ibb.co/5FBTjbr/error-message.jpg

Ah yes, it's another dependency I forgot about - sorry.

The dll is here and needs to go in C:\Windows\System32 , as per instructions under "runtime dependencies" here.   It's not actually an Avisynth dll, it's a third party dll that does fast fourier transforms used by QTGMC.   DLL's are basically precompiled C code that any application can use by calling the precompiled functions inside them.   

ireactions wrote:

Also, after the script is running, how do you save the rendered video as an MP4 file?

I'm using ffmpeg at the windows command prompt, eg. paste the following into command prompt:

"c:\program files\ffmpeg\bin\ffmpeg.exe" -color_range 1 -colorspace 1 -color_primaries 1 -i "C:\YourAvisynthScript.avs" -c:v libx264 -tune film -b:v 8000000 -pix_fmt yuv420p -color_range 1 -colorspace 1 -color_primaries 1 "C:\Detelecined.mkv"

'-colorspace 1' and '-color_primaries 1' mean HD colourimetry - use this if you upscaled to HD in your avisynth script.   SD colourimetry is 6 for both.   Note there are two instances to set in the above line - the first tells ffmpeg what the colourimetry of the input file is (i.e what the Avisynth script is outputting) and the second is what colourimetry ffmpeg should use for the output mp4 file.

'8000000' is the bit rate in bits per second (8mbps)


ireactions wrote:

I'm also wondering if pneumatic might advise me on h.264 settings:
https://i.ibb.co/jWwH6VF/h264.jpg

It's a bit hard to say as Hybrid seems to be using x264.exe which is a different package to ffmpeg.exe with different naming conventions.   I can see the tune is set to grain - I'd set that to film (unless you want it to preserve more grain?  I remember trying the grain preset on an episode of Oz some time ago and didn't like the way it made grain look more chunky, but maybe it was because I didn't set the bitrate high enough - 4mbps).   Underneath I can see it's got '--colormatrix bt470bg' which means SD PAL colourimetry which would produce wrong colours if the output file is HD.  NTSC colours are 170M, HD is 709.


ireactions wrote:

And I'm curious if the default NNEDI3 settings should be adjusted:
https://i.ibb.co/mB6sN3M/nnedi3.jpg

It seems ok.

ireactions wrote:

Do Rep0=13 and EZKeepGrain require additional DLLs?

I don't believe so, for example QTGMC(preset="slow", Rep0=13, EZKeepGrain=1.0) is working on my system with just those dll's I put on gdrive.  I double checked by removing all other dll's from my C:\Program Files (x86)\AviSynth+\plugins64 folder and putting only those ones on gdrive in there. 

What error message do you get when doing eg. QTGMC(preset="slow", Rep0=13, EZKeepGrain=1.0)?

I do believe QTGMC may have some extra dependencies when using "very slow" and "placebo" presets though - those should be available in "optional plugins" here: http://avisynth.nl/index.php/QTGMC#Requirements


ireactions wrote:

I put your DLL pack into the Avisynth folder, but it still didn't run until I added more DLLs (avsresize, BWDIF, LSMASHSource, nnedi3, TDeint, TIVTC).

Yes, so you'll need:

1. TIVTC dll's: https://drive.google.com/file/d/1IgIze3 … FOW5F/view
2. QTGMC dll's: https://drive.google.com/file/d/1c1Duyt … YnXZx/view

But I forgot about avsresize.dll - that one is needed to do the colourimetry conversion from SD colours to HD colours if you are upscaling in Avisynth.

ireactions wrote:

I'm hoping to spend some of the Easter holiday reading up on Avisynth+ and how to use it so that I'm not dependent on the only person in this thread who knows what he's doing to type out the code.

The first thing I'd do is confirm you've got Avisynth up and running by making that hello world Version() clip.

Once you've got that running in MPC-HC, the next thing I'd recommend is set MPC-HC's audio render to "MPC Audio Renderer" as that seems to be required for compatibility with Avisynth audio (in MPC-HC options -> playback -> output -> audio renderer -> MPC Audio Renderer).

Also in MPC-HC options -> internal filters -> source filters -> tick Avisynth.  This will make MPC-HC use its own internal LAV filter to open .avs files, which in my experience is better than the default Microsoft one you will probably get if it's unticked.  If you have any troubles, right click the video image while an .avs file is playing -> Filters -> copy filters to clipboard, and paste the result here.

If you can get to this stage then you should be good to go with writing Avisynth scripts and previewing them in realtime without having to spend hours transcoding to mp4.   QTGMC's "medium" or "fast" presets may be useful too.  Don't forget to enable multithreading with Prefetch(cores) at the end of the script, as QTGMC needs multithreading to be fast enough for realtime use.

I'm currently working on a script that will default to outputting 60fps deinterlaced frames thus preserving frame pacing.  Then if a deinterlaced frame is similar to the TIVTC'd frame within a threshold, swap it for the TIVTC'd frame.  This should give the best of both worlds - TIVTC 480p resolution on film sections (most of the show) and autoswitching to 60fps deinterlaced goodness on video sequences (like Eggheads mindgame, intro sequence, etc.).

edit: here's a preview.  Still a work in progress, trying to adjust the thresholds to handle the switching behaviour on low motion

Yep, and the good thing is it doesn't smudge out resolution.   It does suppress film grain and video noise to a slight degree though, but 1px thick lines remain fully resolved, unlike QTGMC's deinterlacing mode.  Film grain can be added back in with "EZKeepGrain=n"  (n: 0.0 = off, 1.0 = a good starting point). 

Also when using QTGMC in any mode I would recommend adding the parameter "Rep0=13" which repairs motion vector errors for a very minimal CPU expense.  It seems to only be an issue when QTGMC is processing duplicate frames where the motion vectors seem to "bleed through" into the next duplicate frame when there is motion:  default , Rep0=13  .

pneumatic wrote:

Well, I'm officially stumped.   I'm looking at the opening scene of S01E06 Last Days from the Universal NTSC DVD (disc ISO, not someone else's transcode) and I'm seeing deinterlacing artefacts BAKED INTO the source.  These artefacts are not present in another copy of this episode which was transcoded by someone else....
I'm not sure what's going on, either there are multiple Universal NTSCs with different image quality, or whoever did this particular transcode of it is wielding some voodoo magic video processing that I cannot get my head around.

For completeness here's a short clip of that scene with vs without QTGMC repair - the difference is quite astonishing: https://drive.google.com/file/d/1D4PEbu … share_link

ireactions wrote:

I wanted to try the QTGMC repair processing, but... I can't seem to get it together.

I installed AviSynth+, dropped in the DLL files as advised, and then ran pneumatic's script in AviSynth, but I got a message saying: "Script error: There is no function named 'QTGMC'." I'm finding the instructions on the Wiki to be just... baffling. http://avisynth.nl/index.php/QTGMC

Yeah sorry about that - the dlls I linked to included everything except QTGMC.   

QTGMC has a ton of dependencies, it needs about a dozen other dlls which can be found under the "requirements" section of the above link.   For convenience I've put everything in a zip file here.   

The one that goes in C:\windows\system32 is from this plugin used by QTGMC (so FYI so that you know it's legitimate).

The rest can all be dropped in eg. C:\Program Files (x86)\AviSynth+\plugins64 and then it should work.

If you have a media player that supports opening avisynth scripts (eg. MPC-HC)  then you can just open your avisynth script directly in the media player and begin previewing them straight away.  This is how I'm using it and saves a ton of time, would highly recommend.

For example make a little test avisynth script, right click on the desktop and "create new avisynth script", open it in a text editor and type this in it:

Version()

Then double click the avisynth script and it should open a video in the media player like this:

https://i.lensdump.com/i/Tf0CtD.png

Basically "Version()" created a 10 second video clip containing that text overlay.

If you don't have MPC-HC I'd recommend this pack which contains it along with numerous codecs and the acclaimed MadVR video renderer: https://codecguide.com/download_k-lite_ … k_full.htm

ireactions wrote:

Considering the original state of those Season 1 files (blurriness, comb lines, jagged edges)

Just to clarify regarding comb lines or "combing", those aren't defects - interlaced video is supposed to have combed frames.  It doesn't necessarily have to contain combed frames, but it's expected and normal if it does have combed frames.

The way I'd summarise it is like this: 480i30 is the same as 480p30 in that it still has 30 frames per second of 720x480, except with 480i30 each frame may or may not contain two different images inside it - one in the even lines and the other in the odd lines (top field and bottom field respectively).  If such frames are displayed "as is" then it looks combed as we're seeing two different images from two different points in time weaved together. CRT's would display each field seperately one after the other so the resulting image wouldn't look combed.

Not all frames in the source are necessarily combed though, eg. the spinning earths intro animation is effectively 480p30 - both even and odd lines are from the same image, so it doesn't even need any processing at all and will look identical to 480p30.   The majority of the show uses a pattern of 3 progressive frames followed by 2 combed frames - a lossless method of storing 24p inside a 30i container. 

In an ideal world the entire show would just follow this predictable 3-2 pattern (cadence), but in practice the source may contain a hodge podge of different cadences and/or random cadence breaks. eg. some sequences where the camera is panning with the 3-2 pattern, and then they'll overlay credits half way through a frame, momentarily breaking the 3-2 pattern. Some frames either side of a scene change may have unexpected combed frames due to splicing half way through a frame.  In Eggheads when they're playing Mindgame, all frames are combed. 

Dealing with all of these cadence changes presents a massive headache if your goal is to figure out which fields can be weaved with others to extract as many progressive frames as possible.  TIVTC does this quite well, but inevitably ends up with leftover combed frames which contain fields that couldn't be weaved with any others due to the aforementioned idosyncrasies.  This is where TIVTC's "post processing" comes in - it detects and deinterlaces any such combed frames leftover after the field matching process.  This relies on combed frame detection which is prone to false positives or false negatives.   But when a combed frame is detected, it uses motion compensated deinterlacing which maximises resolution by weaving parts of the image which are static, and upscaling ("interpolating") parts which are moving.  So anything that moves basically drops to 240p, but anything static has full 480p resolution.  The drop to 240p on motion causes aliasing, shimmer and moire, but QTGMC's "repair progressive" mode (InputType=1) does a remarkable job at cleaning this up and making it look like 480p (except for moire patterns). 

Separate to this is QTGMC's deinterlacing mode (InputType=0) which is intended to be used on its own and not in conjunction with TIVTC. In this mode, moire is present even on static elements, because QTGMC deinterlacing never weaves fields to progressive frames - it is a completely novel solution which upscales every field from 240p to 480p, and then does some very clever blending with neighbouring fields, using motion vectors and neural network antialiasing on edges to smooth everything out.  The actual amount of resolved resolution on static elements is somewhere around 360p according to a test pattern.  On moving objects it visually appears similar to 360p as well, although I haven't measured it.  This is great for low quality sources which never contained 480p resolution to begin with (especially the Mill Creek DVDs as it blends and smooths the field alignment issue) but for higher quality sources I prefer its "repair progressive" mode (InputType=1) applied to the motion compensated deinterlacing.

For Season 1 I will probably use TIVTC + post processing motion compensated deinterlacing + QTGMC "repair progressive" mode.  The repair progressive does slightly suppress film grain though - on higher quality sources it does tend to look a little "processed".  But it does get rid of aliasing and shimmer on those occasional deinterlaced frames to the point where I have a hard time telling whether I'm looking at TIVTC'd frames or deinterlaced frames, except for high frequency patterns which still have noticeable moire.

ireactions wrote:

On a more serious note, do you have any recommendations/suggested filters/advised settings for adding film grain? A little grain texture would be nice to offset the soft-focus look.

Funnily enough QTGMC has some dedicated settings for "restoring noise": http://avisynth.nl/index.php/QTGMC#Nois … _Denoising

There's also some "AddGrain" filters here: http://avisynth.nl/index.php/External_filters#Effects

I haven't played with any of them yet - currently spending the day chasing data corruption on one of my drives containing all my remuxed DVDs, including Sliders.

Sorry, looks like I had a brain fart and misread what you wrote roll

ireactions wrote:

I'm looking at the Hybrid + Avisynth preview and at MI:200 with post processing re-enabled, I'm still seeing a few combed frames in the opening titles, but definitely not as many as without post-processing.

Turning off PP turns off combing detection, so you should be getting more combed frames not less.

Increasing MI means there has to be more combed pixels detected within a frame before it declares the frame as combed, so expect to see more combed frames when raising it.  The default is 80 which produces a lot of false positives (like the wall texture it thinks is combing) so by default the error is biased in favour of deinterlacing artefacts (eg. moire) instead of combing artefacts.   In my experiments the highest I could go without too many false negatives was 128, which is half the pixels in a 16x16 block.

Hard for me to say what's going on...maybe Hybrid has implemented it different or wrongly?  The combing threshold option in Hybrid appears is mislabeled as "chroma threshold", maybe there are other errors in the implementation.

That's true, but then you will have some combed frames on certain sequences like during the intro, or certain scene changes that weren't spliced correctly, or when they're playing Mindgame in Eggheads.   

If occasional combing doesn't bother you then you should use it as it retains full 480p resolution on 24fps and 30fps sequences which make up 98% of the show. 

If TIVTC had more reliable combing detection this wouldn't be an issue.  We can probably improve it for this episode by playing with the combing detection thresholds (cthresh and MI parameters) but in my experience this only tends to provide a choice between false positives or false negatives.

But it's still an improvement from no post processing at all, so try setting MI=200 which eliminates false positives from that street scene (no moire).  The highest number of combed pixels for that scene was 195 inside the square by the window, so setting MI to 196 or above would avoid moire on that scene:

https://i1.lensdump.com/i/TNY1XM.png

Just be aware you may get other scenes that could be combed and have less than 200 combed pixels and those would go undetected (false negative) and remain visibly combed.

To get that debug overlay in Avisynth: TFM(PP=1, display=true)

ireactions wrote:

Would pneumatic be able to share the script for TIVTC + QGTMC repair? I can't seem to get QGTMC working on Hybrid at all.

clip = "C:\S01E08 Eggheads.mkv"   # produced from the disc using MakeMKV (lossless remux)
CPUcores = 4  

video = LWLibavVideoSource(clip, stream_index=-1, repeat=true, cache=true)
audio = LWLibavAudioSource(clip, stream_index=1, cache=true)
AudioDub(video, audio)

TFM(mode=0, slow=2, scthresh=100, PP=3, metric=1, cthresh=9, MI=128,
\    clip2=PropDelete("_FieldBased").bwdif(field=-1, thr=5, edeint=nnedi3(field=-1)))

TDecimate(mode=0, cycleR=1, cycle=5, hybrid=1, viddetect=2, vidthresh=4.5,
\    denoise=true, chroma=false, hint=false)

QTGMC(InputType=1, preset="slow", Sharpness=0.0, FPSDivisor=1, EdiThreads=CPUcores/2) 

z_ConvertFormat(                      
\ colorspace_op="601:601:170m:full=>709:709:709:full",
\ resample_filter="spline36",
\ interlaced=true,
\ dither_type="ordered")

LanczosResize(1440, 1080)

# Optional: the following Trim command will output only a short snippet, useful for quick previewing:
# Trim(0, 240) # outputs the first 240 frames only (10 seconds @ 24fps)
# Trim(240, 480) # outputs frames 240-480 
# Trim(480, 0) # outputs frames 480 til end
# ShowFrameNumber() # overlay frame number onto video

Prefetch(CPUcores)

edit: alternatively, if you want the same moire reduction as above, but with original 60fps frame pacing (smooth on spinning earths, Mindgame sections etc.) just delete both the TFM and TDecimate lines and put instead:

propDelete("_FieldBased").bwdif(field=-2, thr=5, edeint=nnedi3(field=-2))

The downside is that all frames will be deinterlaced, so you'll only get true 480p resolution on static elements but not on moving elements.


ireactions wrote:

Are you running the MKV through TIVTC and then encoding the resulting MP4 file again with QGTMC? Or is it all one job with the MKV to MP4?

All in the one job, otherwise image quality won't be quite as good due to multiple rounds of compression.  I'm just opening the windows command prompt, changing to the ffmpeg folder with "cd c:\program files\ffmpeg\bin", then pasting this and hitting enter.  ffmpeg takes the Avisynth script as the input and produces the mkv file as the output.   

ffmpeg -color_range 1 -colorspace 1 -color_primaries 1 -i "C:\TheAboveScript.avs" -c:v libx264 -tune film -b:v 8000000 -pix_fmt yuv420p -color_range 1 -colorspace 1 -color_primaries 1 "TIVTC + QGTMC repair.mkv"


ireactions wrote:

Thank you again for sharing all your expertise and ongoing learnings with us.


Happy to help, and thanks for everyone's help as well.



RussianCabbie_Lotteryfan wrote:

was the non hd color version more how it played on the old tube tvs?

Well the show would have been produced with rec.601/smpte-170m colourimetry, so you'd want VLC to be interpreting it as such, and that seems to be the case with the tagged mkv.  All other mkv's I uploaded to gdrive after that were correctly tagged as well. 

I'm not familiar with VLC but there should be some extra status info or debug screen to get VLC to display details about the colourimetry it's using.  Or you could take a full screenshot on the scene with the front view of the red car and compare it to my screenshot which is using correct colourimetry as indicated by the overlay on my screenshot.

Here's that first qtgmc preview again, but this time tagged with SD colours - specifically "-colorspace 6 -color_primaries 6" in ffmpeg (6 = rec.601/smpte170m)

qtgmc 720x540p60 Rec.601 tagged.mkv

I'd expect VLC should read the tags and do the conversion properly.  Without the tags VLC probably just assumed nothing and didn't do any conversion, which is fair enough.

RussianCabbie_Lotteryfan wrote:

In the old SD one, are you suggesting that it isn't doing shades of colors well, so it sort of averages out a given tone to the nearest SD color (which may be more pronounced)?

There are 2 issues:

1. SD and HD use different formulas to convert between RGB and YCbCr.  Video is encoded in YCbCr and the media player will convert it to RGB internally so it has to use the right formula.

2. HD and SD use different gamuts (range of colour) which basically defines what wavelengths the RGB values correspond to. 

On top of this there is also some discrepancy in how different media players and renderers implement the conversions, and what assumptions they make about the monitor's properties in the process.

RussianCabbie_Lotteryfan wrote:

One question, when you compare the reds between the two videos, are they the same for you?

edit: here's a before/after:  https://imgsli.com/MTY1NDY1

For  reference, look at Arturo's skin pigment and red coat.

Yep, it's way off on the old SD one, and I'm pretty sure it's cause your media player isn't converting SD colours to HD colours to suit your display mode which uses HD colours.

But that's my fault cause I assumed that media players would see the file's resolution is SD and convert them to HD colours, as that's how mine operates - https://imgsli.com/MTY1NDcz

I'll run off another copy with the mkv file properly tagged for SD colours and then it should look fine on your end and all media players.  Sorry about that - good catch.

edit: btw, the colour is still slightly different on mine too, and I think it's cause my media player uses slightly different formulas to do the conversion compared to the Avisynth filter I used.

May have found a good balance with an alternative QTGMC solution.  It's like the original QTGMC solution but resolves a bit more resolution when camera is still.

Pros:
* full 480p resolution on static elements (credits text, moments when the camera is perfectly still etc.)
* QTGMC smoothing and antialasing on moving parts of the image (also reduces moire more than QTGMC alone)
* perfect frame pacing (spinning earths and Mindgame sections are 30fps & 60fps)

Cons:
* no smoothing of the field alignment issue on static elements (since those are just weaved to 480p)

Preview: bwdif deinterlace + qtgmc repair.mkv

New discovery...QTGMC has a separate "repair progressive" mode which can provide a significant improvement when applied to all frames post-TIVTC.  This mode doesn't deinterlace at all, it just "repairs" progressive frames in a way that reduces deinterlacing artefacts, aliasing, moire, shimmer etc. without smudging out any resolution in the source. I ran it through a resolution test pattern and 1px alternating white and black lines retain full contrast.

The downside is that it doesn't provide any benefit to video that was already deinterlaced by QTGMC, but it greatly reduces post-TIVTC deinterlacing artefacts such as:

1. "Eggheads" the moire on the wall shingles are suppressed - clip
2. "Prince of Wails" the Jeep radiator grille is now perfectly rendered
3. "Last Days" opening street scene with the tree branches is perfectly rendered

Because it only works post-TIVTC, you have to forfeit the perfect 60fps frame pacing that you would get with QTGMC deinterlacing only, such as the scenes in "Eggheads" where they're playing Mindgame (comparison: QTGMC 1080p60 , TIVTC + QTGMC repair 1080p24)

Anyway, I'm super stoked with this discovery as it means I don't have to worry so much about false positives with TIVTC's combing detection, as those frames will come out looking nice after being repaired by QTGMC.   I'm also using it on Rugrats (streaming service version) and it beautifully smooths out all the deinterlacing artefacts baked into it.

ireactions wrote:

Thanks. I'll stick with the moire patterns. It's fine.

For that particular episode QTGMC would be better imo as it preserves all the 60i sections where they're playing Mindgame - those should look buttery smooth 60fps.

QTGMC has a ton of spatial smoothing settings, I'll try having a play around with them to reduce moire, but I've got a feeling it would come at the expense of sharpness.   

ireactions wrote:

I don't know why, but I don't like the results with the Nvidia encoder. GPU acceleration is fast (7 - 10 minutes per episode), but image just looks dull and fuzzy to me, even though when I take screenshots, I can't see any difference. I'm sticking to CPU encodes (an hour per episode) for this.

Is the screenshot on a static scene or motion?

ireactions wrote:

I'm getting some moire patterns where there were previously jagged edges. This screenshot from "Eggheads":

https://i.ibb.co/Kq01GDp/Sliders-107-Eggheads-mp4-snapshot-26-09-757.png

Any recommendations from pneumatic for what addresses this? Or is it best left alone?

It's a side effect of field interpolation, which is what deinterlacing does, and QTGMC is a deinterlacer that suffers from this as well.   I had mentioned it here (check out the sleeve on her shirt).

pneumatic wrote:

By the way QTGMC is effective at suppressing the field alignment issue too, and looks sharper [1][2][3] but introduces moire artefacts on stippled patterns [4].

The only way I could suppress it is to blur/blend pixels with the one above/below (which is the left side of that 4th image comparison) but then the image becomes quite a lot softer to the point that TIVTC alone would be the clear winner imo.

edit: just checked that scene with TIVTC and it's no better, cause the wall texture has alternating 1px patterns which gets falsely detected as combing, triggering TIVTC to switch to deinterlacing mode for those frames, resulting in moire.    Setting the combing detection threshold higher isn't an option as I've already set it so high that there are some false negatives creeping in.   The way this would be handled by a restoration studio would be to flag that scene as progressive to force deinterlacing to not kick in just for that scene.   A better solution would be to have better more reliable combing detection in the first place.

RussianCabbie_Lotteryfan wrote:

It did upscale to look pretty good.  BUT, ireactions later used topaz to upscale the documentary for us with Topaz (artemis dehalo algorithm) and it was substantially better.     And things looked so gorgeous, he got a license to topaz and did it in the gaia algorithm, which he showed the end result to a tv producer, who could not even tell the footage was shot in 2003 and resuscitated from old dvd files!    So since you are really into this stuff, I would just suggest you look more at Topaz as well.   I could be wrong but I am not sure avi synth is as powerful of a tool for certain use cases.

I'm sure Topaz can do a better job than nnedi3, I just don't have $300 USD to spare right now.  Although I see they have a free trial version so I'll check it out for sure.   

edit: unfortunately I cannot use it as it's Windows 10+ only.

ireactions wrote:

it just seems better to have the file brought to 1080p with a pixelation-reducing rescaler (Lanczos) rather than my TV doing a linear stretch.

Yeah, it's probably better for us to produce 1080p files as it takes out of the equation the unknown upscaling quality of the end user's media player or TV, which could potentially be poor.   

These are the scalers included with Avisynth: http://avisynth.nl/index.php/Resize

The most popular ones seem to be BicubicResize, Spline36Resize and LanczosResize.  They can be tuned for sharpness:

BicubicResize(1440, 1080, 0.0, 0.75) means upscale to 1440x1080 with blur=0.0, sharpen=0.75

LanczosResize(1440, 1080, taps=4) is sharper than LanczosResize(1440, 1080, taps=3), the latter of which appears very close to Spline36Resize.

There is also this Sharpen() filter included with Avisynth: http://avisynth.nl/index.php/Sharpen

There are a bunch of third party sharpeners here http://avisynth.nl/index.php/External_f … Sharpeners which I'm sure can do a much better job than Sharpen(), but I haven't delved into them as I do my sharpening through the media player (MadVR).

If we upscale to HD we must remember to convert the colours to HD otherwise they will be quite off:
https://i.lensdump.com/i/TolEnr.png
(animated .png)

In Avisynth...

# convert colourimetry from NTSC to HD 
z_ConvertFormat(                        
\ colorspace_op="601:601:170m:full=>709:709:709:full",
\ resample_filter="spline36",
\ dither_type="ordered",
\ interlaced=true )

AI upscaling:

nnedi3_rpow2(rfactor=4, nns=1, nsize=0, cshift="Spline36Resize") # AI upscale 4x (720x480p -> 2880x1920p)
Spline36Resize(1440, 1080)                                       # downscale to 1080p

Preview:

QTGMC + AI upscaling (1080p60).mkv
TIVTC + AI upscaling (1080p24).mkv

To save time you can make yourself a short clip to experiment on:

ffmpeg -ss 00:00:00 -to 00:00:30 -i "C:\S01E04.mkv" -c:v copy -c:a copy "C:\ShortClip.mkv"

This will output the first 30 seconds of S01E04.mkv to ShortClip.mkv without any transcoding (MediaInfo should report the video track is still MPEG2 720x480 29.97fps interlaced - exactly what was on the disc).

Then if you install Avisynth+ 64-bit you can perform detelecine on ShortClip.mkv like so:

ffmpeg -color_range 1 -colorspace 6 -color_primaries 6 -i "C:\MyAvisynthScript.avs" -c:v libx264 -tune film -b:v 8000000 -pix_fmt yuv420p -color_range 1 -colorspace 6 -color_primaries 6 "C:\Detelecined.mkv"

This will output Detelecined.mkv at x264 720x540 23.976fps progressive 8mbps (confirm with MediaInfo).

Inside MyAvisynthScript.avs put:

clip = "C:\ShortClip.mkv"
video = LWLibavVideoSource(clip, stream_index=-1, repeat=true, cache=false)
audio = LWLibavAudioSource(clip, stream_index=1, cache=false)
AudioDub(video, audio)

TFM(mode=0, slow=2, scthresh=100, PP=3, cthresh=9, MI=128,
\    clip2=PropDelete("_FieldBased").bwdif(field=-1, thr=2, edeint=nnedi3(field=-1)))

TDecimate(mode=0, cycleR=1, cycle=5, hybrid=1, viddetect=2, vidthresh=4.5, 
\    denoise=true, chroma=false, hint=false)

LanczosResize(720,540)  

The above Avisynth script uses filters that don't come with Avisynth+ so you'll have to download them and place their dll files in typically C:\Program Files (x86)\AviSynth+\plugins64.  I've put them all in a zip file here (edit: this doesn't include the HD colourimetry conversion and QTGMC dlls - if you want those too I've put them all here). 

Alternatively here are the author's download pages: [1][2][3][4][5][6] (be sure to get the 64-bit versions).

ffmpeg 64bit can be downloaded here.  Just unzip the files to anywhere, open a windows command prompt, navigate to the \bin folder where ffmpeg.exe is and you're ready to start typing ffmpeg commands in command prompt.