INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Why is Convert Format so slow?  (Read 993 times)

haggis999

  • Citizen of the Universe
  • *****
  • Posts: 514
Why is Convert Format so slow?
« on: June 22, 2021, 02:58:31 am »

I am currently running a Convert Format process to create MP3s from my music collection for use when driving my car, using VBR encoding at normal target quality. The original files are all in FLAC format. Most of this music is classical, which often has a very wide dynamic range, so I'm using DSP to compress this range so that it is always audible above road and engine noise.

My problem is that it is running at a snail's pace, even though I am using a brand new Windows 10 workstation with a fast processor and 32GB of RAM. I left it running overnight and in 10 hours it has only converted 5347 files.

Is this normal? Have I missed some trick for processing more than 2 files at a time?
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Why is Convert Format so slow?
« Reply #1 on: June 22, 2021, 03:22:45 am »

'5347 in 10 hours' is kind of meaningless without knowing how big each file is, both in minutes and size/bitrate. If each file is a 1h concert then you're doing fine...

Is your collection connected directly to the MC computer (via USB or internal disk), or is it on a NAS connected via Wifi or LAN?

Napkin math: 5000 tracks, 5 minutes per track, ~60 MB FLAC per track, that's 300 GB of data to read; if you're converting to 320Kbps MP3, that's about 100GB to write. So, 400GB in 10 hours, that's about 11 MB/sec of data being transferred, or around 100 Mbit/second. This number rings a bell...

So, you might be limited by your networking. If you're on a 100Mbps cabled network, you can't go much faster. If you're on Wifi, switch to cabled or try moving things closer to the Router or Access Point.

If you're using USB or an internal disk, then the issue is something else, as 10 MB/sec is not that much for a directly-attached disk. Check the CPU usage to see if it's pegged at 100%.
Logged

haggis999

  • Citizen of the Universe
  • *****
  • Posts: 514
Re: Why is Convert Format so slow?
« Reply #2 on: June 22, 2021, 03:49:02 am »

'5347 in 10 hours' is kind of meaningless without knowing how big each file is, both in minutes and size/bitrate. If each file is a 1h concert then you're doing fine...

Is your collection connected directly to the MC computer (via USB or internal disk), or is it on a NAS connected via Wifi or LAN?

Napkin math: 5000 tracks, 5 minutes per track, ~60 MB FLAC per track, that's 300 GB of data to read; if you're converting to 320Kbps MP3, that's about 100GB to write. So, 400GB in 10 hours, that's about 11 MB/sec of data being transferred, or around 100 Mbit/second. This number rings a bell...

So, you might be limited by your networking. If you're on Wifi or on 100Mbps cabled network, you can't go much faster.
If you're using USB or an internal disk, then the issue is something else, as 10 MB/sec is not that much for a directly-attached disk.

These music files are all stored on a Synology NAS with access via what should be a Gigabit wired Ethernet link. They were all ripped from my CD collection. Track durations vary widely with classical music, though very few would last much longer than 30 mins. I've no idea what the average duration might be, but 7 minutes would be my best guess.

I will check my network later today to see if the relevant connection is Gigabit or 100Mbps.
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Why is Convert Format so slow?
« Reply #3 on: June 22, 2021, 03:50:58 am »

You can select those 5000 files in MC and then check the status bar - it tells you the total runtime and total size.
All connections in the path would need to be Gigabit, all it takes is one slow segment.
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Why is Convert Format so slow?
« Reply #4 on: June 22, 2021, 04:00:39 am »

Also, how long is your commute?  ;D
5000 files x 7 minutes = 24 days of continuous play, or about 10 months of listening to 2 hours of music per day in your car... I understand wanting to carry your entire collection for freedom of choice, but sometimes it takes longer to pick want you want to hear next instead of just having some curated playlists.
Logged

Hendrik

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 10941
Re: Why is Convert Format so slow?
« Reply #5 on: June 22, 2021, 05:20:30 am »

10 hours are 600 minutes, thats about 9 files per minute, or one file every ~7 seconds. That seems like a rather decent speed to me, factoring in overhead of reading new files, decoding and encoding them, writing them, tagging them, etc.
MP3 in particular uses an external encoder, which also has some added overhead executing it, which isn't huge, but it all adds up when you process thousands of files.
Logged
~ nevcairiel
~ Author of LAV Filters

haggis999

  • Citizen of the Universe
  • *****
  • Posts: 514
Re: Why is Convert Format so slow?
« Reply #6 on: June 22, 2021, 06:49:23 am »

Also, how long is your commute?  ;D
5000 files x 7 minutes = 24 days of continuous play, or about 10 months of listening to 2 hours of music per day in your car... I understand wanting to carry your entire collection for freedom of choice, but sometimes it takes longer to pick want you want to hear next instead of just having some curated playlists.

I think playlists work better for non-classical music. In my case, I usually know what I want to play and many of my potential choices can last much longer than my typical journey time, so having my full collection available is an advantage. The USB memory stick in my car was last loaded several years ago, so an update is long overdue.
Logged

haggis999

  • Citizen of the Universe
  • *****
  • Posts: 514
Re: Why is Convert Format so slow?
« Reply #7 on: June 22, 2021, 06:57:28 am »

10 hours are 600 minutes, thats about 9 files per minute, or one file every ~7 seconds. That seems like a rather decent speed to me, factoring in overhead of reading new files, decoding and encoding them, writing them, tagging them, etc.
MP3 in particular uses an external encoder, which also has some added overhead executing it, which isn't huge, but it all adds up when you process thousands of files.

Given that modern CPUs can process a gazillion instructions every second, taking 7 seconds to convert a FLAC to an MP3 seems very slow to me, though I realise that the CPU is not the only factor in this operation. 

I last did this several years ago and it's possible I attached my NAS locally via USB at that time. EDIT: My NAS can't attach to a PC via USB, so that recollection was wrong.

I also seem to remember that it processed about 10 files at a time rather than the 2 it's doing at the moment. Is this something you can control in the settings? I couldn't find anything relevant in Tools/Options.


EDIT:   I've now found the setting for changing the number of files processed in parallel and have upped it from 2 to 16 (it was in the Audio Conversion Options dialogue box). Things have now noticeably speeded up.

BTW, I have confirmed that both my PC and my NAS have Gigabit connections to the network.

 
Logged

haggis999

  • Citizen of the Universe
  • *****
  • Posts: 514
Re: Why is Convert Format so slow?
« Reply #8 on: June 22, 2021, 09:32:20 am »

Stopping the conversion and then restarting it after setting the ‘Number of files to convert at the same time’ to the maximum value of 16 has made a huge difference.

The job soon completed, but I then realised that my choice of a VBR Target Quality of 'Normal' was not what I used in the past, as I had the space to go for a Target Quality of 'Extreme'. I then ran a new conversion using this setting, which has taken just one hour to process 6000 files.

My original problem is thus resolved, and I have updated my file conversion notes to make sure I will remember all my optimum settings for this process :)
Logged
Pages: [1]   Go Up