Suppose, not entirely hypothetically, that you have a custom file type which is actually an existing type with a new file extension. Just for the sake of argument, let's say that the file type is Mozilla Archive Format (MAFF). This is actually a standard ZIP file which by its custom extension (.maff) is given a new purpose.

However, at least in Ubuntu 16.04 and earlier, there is no MIME type installed for MAFF. This causes programs like Nautilus that try to be smart about unknown file extensions to treat MAFF files like ZIP files, and suggest that they be opened in File-Roller. There doesn't seem to be a MIME type registered with IANA either. It is fairly easy to add a custom one though.

I'm writing this down mostly for my own reference. Since there is no standard MIME type for MAFF, I'll use application/x-maff which several sites on the internet (none authoritative though) suggests. First, put the following (adapting it to your needs) into a new file in ~/.local/share/mime/packages/something.xml. I don't think the file name is important, as long as it ends in .xml.

<?xml version="1.0" encoding="UTF-8"?>
<mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
  <mime-type type="application/x-maff">
    <glob pattern="*.maff"/>
    <comment>Mozilla Archive Format File</comment>
  </mime-type>
</mime-info>

Then run the following command to update the user specific MIME database.

$ update-mime-database .local/share/mime/

This will install the information above into .local/share/mime//.xml. In this example this is .local/share/mime/application/x-maff.xml. You are done! You can test your changes using the mimetype command:

$ mimetype 'Understanding the Xbox boot process_Flash structures - Hacking the Xbox.maff'
Understanding the Xbox boot process_Flash structures - Hacking the Xbox.maff: application/x-maff

Changes take effect in new Nautilus windows (in my testing), so close the existing windows and open a new one. Note that this is different from restarting Nautilus, as the process is usually still running in the background.

You could probably also do this change system-wide by placing the XML file in /usr/local/share/mime/packages/ (creating the directory if it doesn't exist) and updating the system-wide MIME database using:

$ sudo update-mime-database /usr/local/share/mime/

I prefer to keep such changes in my home directory though. My system is single-user anyway, and I reinstall the base operating system more often than I recreate my home directory.