When Logic Destroys Your Audio Files

This post is about a serious bug in Logic, which causes audio files to be damaged. The symptom of this bug is the following error message when opening a previously perfectly working project: One or more audio files changed in length.

Background

This is how I encountered the bug: we had recording sessions with our band and had completed the first session. In the studio, we could listen to all tracks without any problems. After some time, when our sound engineer opened the project again, he encountered the error message above for some files. After listening through the projects, he discovered that a large number of audio clips could not be played back anymore. We had one backup of the project we had made right after the recording session, but unfortunately it contained the same corrupted files. This meant that we had to repeat the recording session 🙁

We had a few more recording sessions, after which we made 3 or 4 backups right away. After we had recorded a whole album, the issue occurred again. The files were corrupted on all backups again, although we could listen to the tracks in the studio without problems. Consequently, the data was corrupted after the recording and before re-opening the Logic project.

Then, our sound engineer noticed the following: when he opened a project for the first time, logic reported 4 corrupted files. When opening the same project again, suddenly 24 files were corrupted. This leads to the conclusion that Logic itself is responsible for destroying the audio files.

Conditions under Which the Bug Occurs

The exact conditions under which the error occurs are not exactly clear, but one of the following things or a combination of those things seems to be involved:

  • Using hard drives or SSDs that are not formatted with an Apple File system (i.e. Mac OS X Journaled or APFS)
  • Using external (hard) drives
  • Using the Comping Feature in Logic (the one where you see multiple takes stacked on each other and can combine them to an “optimal” take)

The files are damaged when logic is closed, which means that even if the project works perfectly before closing the application, there is no guarantee that it will work when opened again.

We experienced these problems with Logic 9, but internet forum posts suggest that it can also happen with Logic Pro X. If someone can confirm, knows the exact error conditions or has any updates, please feel free to comment.

Error Analysis

Logic destroys the audio files in seemingly random order, e.g. in a sequence of 79 audio files recorded for one track the files with numbers 43, 48, 56, 57, 58, 59, 65, 74 and 79 were corrupted.

For a more thorough analysis, I compared working files with corrupted files using a Hex editor, in which each single byte in the file can be visualized in hexadecimal representation. The first bytes of an intact wave file look like this:

For a detailed description of each byte, refer to this page. In short, these are the contents of the wave file header:

  • Bytes 1-4: RIFF chunk descriptor
  • Bytes 5-8: chuck size (total number of bytes in the file after this block)
  • Bytes 9-12: format (in this case WAVE)
  • Bytes 13-16: fmt-subchunk header (contains fmt )
  • Bytes 17-20: subchunk 1 size (in this case 16 for PCM)
  • Bytes 21-22: audio format (1 = PCM)
  • Bytes 23-24: number of channels (1 = Mono, 2 = Stereo, etc.)
  • Bytes 25-28: sample rate (e.g. 44,100 Hz)
  • Bytes 29-32: byte rate: number of bytes required to store 1 second of audio for all channels (= sample rate * number of channels * bits per sample / 8)
  • Bytes 33-34: block align: number of bytes required to store one sample in all channels (= number of channels * bits per sample / 8)
  • Bytes 34-36: resolution in bits per sample, e.g. 8, 16 or 24 bits
  • Bytes 37-40: data chunk header (contains data)
  • Bytes 41-44: number of bytes representing the raw audio data
  • Bytes 45ff.: raw audio data

Now let’s have a look at a destroyed audio file:

If the bug occurs, Logic fails to write the wave header correctly. Instead, the file contains only zeroes in the first 44 bytes, which is exactly the length of the wave header. The good news: the raw audio data, starting at byte 45, is still intact (note that the hex editor starts counting bytes at index 0).

If such a corrupted wave file is opened, logic can’t read the header and assumes a default 8 bit setting, which leads to a misinterpretation of the audio data. Consequently, the length of the file will also be misinterpreted. Furthermore, the interpretation will be even more off because a wrong sample rate is assumed. Not good.

Repairing the Audio Files

As a preliminary fix, you can restore the destroyed files by copying a wave file header (i.e. the first 44 bytes) from a correct file (with matching sample rate and bit depth) to a corrupted file in a hex editor.

Update August 29, 2019: It was confirmed that this also works for AIFF files. In this case, the first 512 bytes have to be copied. Thank you very much to Sawyer Wildgen for sharing this!

A wave file header specifying a sample rate of 44.100 Hz and 24 bit resolution starts with bytes similar to these (in hexadecimal representation):

1
52 49 46 46 5B 89 3E 00 57 41 56 45 66 6D 74 20 10 00 00 00 01 00 01 00 44 AC 00 00 CC 04 02 00 03 00 18 00 64 61 74 61 6B 5C 3E 00

However, one potential issue now could still be that the (sub)chunk sizes (bytes 5-8 and 41-44) are not correct, but most audio editors don’t check these values. If you want to correct these as well, make sure that you use the correct little endian representation for these byte groups. This means the byte order is reversed. A complete example is given below.

The formulas to calculate the correct values for WAVE files are:

  • chunk size = <file size in bytes> - 8
  • data chunk size = <file size in bytes> - 44

Integer to Little Endian Hex Conversion

Example: Converting the number 44,100 to a little endian hex number:

  1. Convert number to hex using a scientific calculator or an online converter such as this one. The result is: AC 44. Note that this result comprises two bytes and is encoded big endian (most significant byte first).
  2. Make sure the result is padded to the correct byte size. If the field in the header is 4 bytes, we have to add two zero bytes at the beginning: 00 00 AC 44
  3. Reverse the byte order: 44 AC 00 00. The result is now little endian (least significant byte first), as required by the wave header specification.

To confirm, you can open a working wave file with 44,100 Hz sample rate in a hex editor and check bytes 25-28, which will contain 44 AC 00 00.

Using Wave Recovery Tool to Restore the Wave File Headers

Because quite many files were damaged in our case, I did not want to fix all wave headers manually. Therefore, I developed a program which can fix the wave files all at once. Wave Recovery Tool is available on github and is published under the terms of the GNU General Public Licence.

Conclusion

This post reveals a serious bug in Logic, which can potentially destroy hours and weeks of hard work. Fortunately, the data can be restored completely either manually or using a Wave Recovery Tool I developed. I seriously hope that this bug will be fixed soon or is already fixed in recent versions of Logic.

Notes Regarding AIFF Files

In this blog post, I demonstrated the issue by means of wave file header structures. The same can be done for AIFF files, however the header structure is more complex. The good news: I extended Wave Recovery Tool and now it is also possible to restore AIFF files under certain conditions.

Installing Python 3 and music21 using Homebrew and pip3

In this blog post I’m going to explain how the Python library music21 can be installed in conjuction with Python 3 and its dependencies matplotlib, numpy and scipy on Mac OS X. It can also be used as a tutorial for installing any other Python libraries/modules as well.

The Problem

Initially, on my system there were two parallel Python 2 and Pyton 3 installations. The music21 installer chose Python 2 as default installation target. In order to use music21 in conjuction with Python 3, I tried to install it using the command

1
pip3 install music21

which worked fine. However, when I tried to use the plotting capabilities of music21 an error occured due to the missing modules matplotlib, numpy and scipy. When trying to install matplotlib issuing

1
pip3 install matplotlib

the following error occurred:

SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.

Installing Python 3 using Homebrew

My final solution to this problem was to set up a new Python 3 installation using  Homebrew. This is done by installing Homebrew (if you haven’t got it yet):

1
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Python 3 is installed using the command

1
brew install python

Note: Previously, the formula name was python3 but was renamed to python as Python 3 is now the official default version. The old version can be installed using brew install python@2.

If you already have Python 3 installed, Homebrew will not be able to create symlinks to the python binaries since they already exist. To overwrite the existing symlinks (and thus to set the Homebrew Python as default interpreter for your system) you have to execute this command:

1
brew link --overwrite python

Now, symlinks to the new python installation are created under /usr/local/bin.

By the way: python updates can now be installed by simply executing

1
brew upgrade python

Adjusting the $PATH Variable

In some cases it might be required to tweak the settings of the $PATH environment variable, namely if the old Python 3 installation is still preferred by the system because of a $PATH entry with higher priority. To check if this step is necessary, type:

1
which python3

If the output is /usr/local/bin/python3, you can proceed to the next section. Otherwise, check the contents of your $PATH variable with this command:

1
echo $PATH

which might look like this:

/Library/Frameworks/Python.framework/Versions/3.4/bin:/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/opt/subversion/bin:/sw/bin:/sw/sbin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:

As you may notice, the old interpreter entry /Library/Frameworks/Python.framework/Versions/3.4/bin precedes /usr/local/bin. To give priority to the new Python 3 interpreter, change the order of the paths, ensuring that /usr/local/bin precedes other python paths:

1
export PATH=/usr/local/bin:[more path elements here]

This command changes the $PATH settings for the current shell session only. If you want to make the path adjustments persistent, add the command to the file .bash_profile in your user home folder. It is also possible to reuse the current value of the variable:

1
export PATH="/usr/local/bin:$PATH"

After that, our Python 3 installed with Homebrew should now be the default system interpreter. Verify this with

1
which pip3

which should echo /usr/local/bin/pip3, which is in turn a symlink to the Homebrew cellar (the place where Homebrew installs modules/packages).

Installing the Dependencies

Now you should be able to install music21 and the dependencies using pip3:

1
2
3
pip3 install music21
pip3 install matplotlib # this will install numpy automatically
pip3 install scipy

I hope this will help you to install a clean music21 environment. No go have fun with musical analysis and plotting 🙂

Displaying Tracks of your Music Library Filtered by Bit Rate

If you prefer managing your music in the form of audio files on your computer, your collection has probably grown over the past few years and at the same time encoding standards have improved and expectations of sound quality have risen. In most cases, the contained audio files have different sound qualities regarding their bit rates. My motivation was to display a list of files in my music collection which have a bit rate equal to or higher than 256 kbit/s.

To achieve this, I looked for command line tools that display metadata for audio files including their bit rate. For Mac OS X I found the command afinfo which works out of the box:

afinfo myFile.mp3 | grep "bit rate"

The output looks similar to this:

bit rate: 320000 bits per second

If you are using another operating system, you can check for the following commands and/or install them:

1
2
3
4
5
file <fileName> (Ubuntu)
mp3info -r a -p "%f %r\n" <fileName>
mediainfo <fileName> | grep "Bit rate"
exiftool -AudioBitrate <fileName>
mpg123 -t <fileName>

My goal was to develop a program that crawls through my whole audio collection, checks the bit rate for every file and outputs a list containing only files with high bit rate. I wrote a Python script which does exactly that:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
'''
Created on 25.04.2015
 
@author: dave
'''
 
import sys
import os
import subprocess
import re
 
# console command to display properties about your media files
# afinfo works on Mac OS X, change for other operating systems
infoConsoleCommand = 'afinfo'
 
# regular expression to extract the bit rate from the output of the program 
pattern = re.compile('(.*)bit rate: ([0-9]+) bits per second(.*)', re.DOTALL)
 
def filterFile(path):
    '''
    Executes the configured info program to output properties of a media file.
    Grabs the output, filters the bit rate via a regular expression and displays the
    bit rate and the file path in case the bit rate is >= 256k
    Returns True in case the file has a high bit rate, False otherwise
    '''
    process = subprocess.Popen([infoConsoleCommand, path], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    out, err = process.communicate()
    match = pattern.match(str(out))
    if match != None:
        bitRateString = match.group(2)
        bitRate = int(bitRateString)
        if bitRate >= 256000:
            print("bit rate",bitRate,":",path)
            return True
    return False
 
def scanFolder(rootFolder):
    '''
    Recursively crawls through the files of the given root folder
    '''
    numFiles = 0
    numFilesFiltered = 0
    for root, subFolders, files in os.walk(rootFolder):
        for file in files:
            numFiles = numFiles + 1
            path = os.path.join(root,file)
            if filterFile(path):
                numFilesFiltered = numFilesFiltered + 1
    print("Scanned {} files from which {} were filtered.".format(numFiles, numFilesFiltered))
 
# main program
if len(sys.argv) != 2:
    print("Usage: MP3ByBitrateFilter ")
    sys.exit(1)
 
rootFolder = sys.argv[1]
scanFolder(rootFolder)

The root folder of your music library can be given as command line argument. The programs walks through the folder recursively and executes the command line program to display the bit rate in a separate process. It grabs the output and filters the bit rate from the output using a regular expression. The bit rate and the path of the file are displayed in case the bit rate is >= 256 kbit/s. A summary is also displayed showing the total number of files and number of filtered files.

Of course you can extend the filter criteria by adjusting the script to extract other information than the bit rate from the info command.