Force RGB mode in Mac OS X to fix the picture quality of an external monitor

I recently bought a MacBook Pro (with ‘Retina’ screen), but when I hooked it up to my Dell U2410 monitor via HDMI cable I was shocked by the poor picture quality. The contrast was all wrong and text was misshapen. No amount of calibration in the monitor or software would fix it.

Short answer: OS X thinks my monitor is a TV, and is using the YCbCr colour space rather than RGB. I had to override an EDID setting to force the RGB colour space, and it is now working correctly.

Long answer: I haven’t owned a Mac for a while and had forgotten how difficult much of the “Apple community” can be when it comes to anything that can’t be adjusted in System Preferences. Googling for problems with external monitors on MacBooks found dozens of threads on official and unofficial Apple forums, all full of people with the same problem. The most common response was to blame the monitor, despite assurances from the stricken users that the monitor worked beautifully in Linux and Windows, even on the same machine under Boot Camp.

“You just haven’t calibrated it!”, “You are just too used to Retina now!”, “You just need to buy a Thunderbolt display!” Apple people also like to solve problems by throwing more money at it. (I realise that owning a Mac makes me an Apple person, too. Hypocritical self-loather?)

My lucky break was reading that the current colour space was “YCbCr” when I was browsing the monitor’s settings menu. I was sure that it was using RGB when hooked up to my PC, so I started searching instead for forcing RGB mode in OS X. It didn’t appear to be available out-of-the-box, but I have had some experience in overriding EDID settings for similar purposes so I searched instead for that.

I found this thread on the EmbDev.net forums. Mr Schwarz, thanks very much. Your thread and script was incredibly helpful and informative. It was written to fix problems connecting an external monitor via DisplayPort, but it fixed my HDMI issue just the same. I’ve summarised the required steps below.

My last word is to wonder what Apple is playing at. It seems that this problem has been reported by a lot of people for a long time, and I expect it would require a fairly simple software update. Do they just not care about those using third-party components, or are they actively attempting to force people on to Thunderbolt displays?

How to force RGB in Mac OS X

These steps have been updated for Mac OS version 10.11, “El Capitan”. See below for differences for previous versions of the system.

  1. Download the patch-edid.rb script from the forums thread above, or download Andrew Daugherity’s improved patch-edid.rb script from his github page. Put the script in your home directory.
  2. Disable “rootless” mode, you can follow these instructions: How to modify System Integrity Protection in El Capitan.
  3. Reboot.
  4. Connect only the external monitor(s) in question, if you can (I closed my MacBook lid, for example). The script will make override files for any connected monitor.
  5. Type “ruby patch-edid.rb” in Terminal.
  6. A new folder will be created in your home directory. Move it into the “/System/Library/Displays/Contents/Resources/Overrides” folder. You may have to create the Resources and Overrides folders. If Finder tells you that you are overwriting an existing folder, consider backing it up first.
  7. Restart your computer. The picture quality should be fixed from this point.
  8. Re-enable “rootless” mode, the instructions are available on the same guide: How to modify System Integrity Protection in El Capitan.
  9. Reboot. Enjoy your monitor.

To undo the changes, either delete the folder you had copied to the Overrides folder (if it didn’t already exist) or replace it with the folder you had backed up. You will need to re-enable rootless mode to do this.

Earlier versions of Mac OS X

The process is a little more straightforward. There are two differences to the steps above:

  1. You do not need to disable/re-enable rootless mode and perform the subsequent reboots.
  2. The overrides folder location is “/System/Library/Displays/Overrides”.

Updates

I no longer own a Macbook Pro, but if you’re having trouble with any of these steps, please have a look through the comments below (and note that there are multiple pages). Many questions have been answered with helpful tips from others.

Update, 20 Nov 2016: In the comments Marcus has proposed a faster method that doesn’t require SIP to be disabled. Others have had success with it so give it a go if you’re uncomfortable with disabling SIP.

Update, 8 Feb 2016: A comment from nos1609 below, warns about a bootloop that can occur when running other patches (like the pixel-clock patch) simultaneously, and how to get around it.

Update, 23 Nov 2015: According to Peter’s post, you don’t need to disable SIP if you use recovery mode. If others have similar success with this method I’ll update the process.

Update, 3 Oct 2015: I have amended this post to target El Capitan. I have taken the steps from bigmcguire’s process, posted in the comments. Although some are still having issues, it appears to be working for people. Thanks!

Update, 29 May 2015: Mac OS 10.11 El Capitan does things a little differently. You must first disable the new ‘rootless’ mode and then use a different overrides folder: /System/Library/Displays/Contents/Resources/Overrides. Rootless can then be re-enabled if desired, as confirmed by nos1609 in the comments below. El Capitan is still in beta, I’ll update this post if the issue is still apparent afterwards.

Update, 26 May 2014: If you have had trouble with limited resolutions being available after the fix, check out Ibrahim’s comments here.

Update, 28 Nov 2013: If the process appears to work but doesn’t seem to make a difference, consider Tom’s comments below. Depending on your monitor an extra tweak may be required.

Update, 13 Nov 2013: Andrew comments below that he has modified the script to add some useful new features, and provides a link to his GitHub for those wishing to use it instead.

Update, 27 Oct 2013: If you’ve applied this fix before, the OS X Mavericks update will overwrite it. I’ve successfully re-applied the fix by following exactly the same steps, and other commenters below have done so, too.

Replace caps lock with something more useful

I rarely use the easy-to-reach caps lock key, but I often stretch my little finger to get to the control key. And I never use the scroll lock key. So, in a fit of ergomania I made some changes to the Windows registry.

  • scroll lock becomes caps lock
  • caps lock becomes control
  • control is… still control

I have successfully done this in Windows XP, Vista, and 7. There are numerous key-mapping programs around that do the same thing but since this is the only change I wanted to make, it was easier just to set it and forget it.

Here’s how to start using your caps lock key more often. Or less often, if you are a shouty, chain-email-forwarding great uncle.

Control

To set caps lock to act as control, and set scroll lock to act as caps lock:

  1. Create a file with a .reg extension and insert the text in the block below (ensure that it ends with a new line).
  2. Double-click on the file, select ‘Yes’, then restart your computer.
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,03,00,00,00,1d,00,3a,00,3a,00,46,00,00,00,00,00

Backspace

Alternatively to set caps lock to act as backspace while still setting scroll lock to act as caps lock:

  1. Follow the same process as above, but use the following code.
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,03,00,00,00,0E,00,3a,00,3a,00,46,00,00,00,00,00

Undo

To undo either of the above changes by returning your keyboard to the default mapping:

  1. Follow the same process as previously, but use the following code.
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,03,00,00,00,3a,00,3a,00,46,00,46,00,00,00,00,00

VGA video with HDMI audio on the Acer Revo

This information is for a specific setup of the Acer AspireRevo 3700 with XBMC, but it shows that there is no hardware limitation and may provide hints for achieving the same result with other models and configurations.

XBMC

Since 2004 my original Xbox has served as a cheap entertainment centre. Until last week it was running XBMC4Xbox, a branch of XBMC that has been built specifically for the console since the main project began to target newer, beefier hardware. However, the Xbox struggles with most high definition content, so to play such files I was forced to use my PS3 and the excellent PS3 Media Server.

As great as that software is, using the PS3 for any kind of media management after being spoiled my XBMC is a pain. I recently bit the bullet and bought a net-top computer in order to install XBMC and use it to play all of my content. I chose the Revo 3700 because it ticks a number of boxes – particularly its support for VGA-out. Most similar devices have dumped this feature, making them incompatible with my Sony Wega CRT.

XBMC comes in a number of flavours, but a third-party Linux distribution called OpenELEC appealed to me because it is built from scratch specifically for XBMC. It also provides a version compiled for the ION GPU used by the Revo. This means it boots quickly, runs efficiently, and is incredibly easy to install. For most setups this would have been the end of the story, but I had a specific configuration in mind.

Configuration

Although I require VGA for video, my Yamaha receiver takes HDMI and I was keen to take advantage of the benefits it has over optical audio. This is where I ran into trouble – if the HDMI cable was connected the system would assume that it was being used for video, and would switch off the VGA.

OpenELEC allows the user to edit the X.Org configuration by creating an xorg.conf file in a specific config folder. I found that I could specify that the graphics device (an NVIDIA ION2) should use the CRT display, but when I did that the HDMI would be disabled entirely – the HDMI indicator on my receiver would turn off. It seems that in order to use the HDMI audio, X needs to send HDMI video, too.

What I’ve done is simply enable the NVIDIA feature “TwinView”, and set its orientation option to “clone” mode. This means that the same video is being sent to both the VGA and HDMI outputs. I’ve pasted my xorg.config file below; in most setups just the Device section would suffice but I was required to flesh it out a bit more to accommodate my TV.

# X.Org configuration: /storage/.config/xorg.conf

# define the mode required by the TV
Section "Modes"
  Identifier "modes"
  Modeline   "1280x720" 74.250 1280 1330 1370 1650 720 725 730 750 +hsync +vsync
EndSection

# define the device and enable cloning
Section "Device"
  Identifier "device"
  Driver     "nvidia"
  Option     "TwinView" "true"
  Option     "TwinViewOrientation" "Clone"
EndSection

# define a monitor so that we can select the custom modes
Section "Monitor"
  Identifier "monitor"
  UseModes   "modes"
EndSection

# define the screen and select the mode
Section "Screen"
  Identifier "screen"
  Device     "device"
  Monitor    "monitor"
  SubSection "Display"
    Depth      24
    Modes      "1280x720"
  EndSubSection
EndSection

Extras

X could not retrieve the TV’s EDID, so I had to define a modeline manually. I found this by connecting my PC to the TV and using PowerStrip to set my desired resolution and calibration. The software then provided the “Modeline” above.

Finally, in order to use the correct sound device I had to set custom values in XBMC’s audio settings. These values were provided by Therio on the OpenELEC forums:

Audio output:       HDMI
Output device:      custom -> plughw:1,7
Passthrough device: custom -> plughw:1,7

Everything is now working perfectly.