Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[blgentry]

I've got a fairly common situation:  My library is much larger than my handheld can hold.  I've been manually managing it, but it's pretty painful to do so.  I decided that what I'd really like is the following:

1.  A list of files to always be included on the handheld.
2.  A list of files to always be Excluded from the handheld.
3.  The rest of the space filled up with songs that don't meet the above *and* limited to exclude some genres.
4.  All of the "filler songs" need to be full albums.

With some research and experimentation I think I have this done.  It was tricky and requires several lists both smart and regular.  I really only have one sticking point:  I have to manually calculate the amount of space for the "filler songs".  What I mean is this:  My current device holds 64GB.  My include list, right this second, is 15GB.  That's a difference of 49GB.  I have to manually specify that 49GB in my filler smart list.  If there was a function or expression for calculating the total size of a Playlist, I could use that in an expression for the size limit instead.  But I don't think there is.

The reason this is necessary is I have to ADD IN my include list at the very end.  If it's part of a limit expression, it's necessarily going to remove some of the songs from my include list as part of the limit.  I can't see a way around that.  So my final list that I use is a sum of two lists: 

1.  A smart list of the random stuff minus the exclude list.
2.  The include list.

#1 is currently limited to 49000 MB.  If I could calculate that instead, it would require less manual work.

Coming up with a way to get a random selection of FULL albums was pretty tricky.  If you just choose random albums and then try to limit by size, it removes random tracks from each album.  If you get clever, you select random tracks and then expand to full albums, then do the limit, you'll get mostly full albums.  But you'll only get albums from the top of the alphabet, because the limit code chops off the bottom selections.  If they are sorted alphabetically, you lose the lower letters.  This becomes very apparent when you are selecting a very small amount of your library with this technique.  You end up with only albums starting with A, B, and C (for example).

The trick is to choose random selection of songs and make the limit way lower than what you want.  Then expand *that* into full albums.  Then limit again, but this time the real limit.  This has the effect of spreading the random pick over all of the album letters, picking a few out and then limiting it down.  A ratio of 7:1 or 8:1 seems to work perfectly.

For example:

1.  Limit size to 6000 MB
2.  Expand to full albums
3.  Sort by Album
4.  Limit size to 49000 MB

Works like a champ.  You get full albums, spread amongst all of them, and limited to a total size of 49000 MB.

I'm mostly posting this as a sanity check to see if anyone sees any ways around the manual calculation.  I'm also open to simplifying this process.

Brian.

[blgentry]

Anyone have any thoughts on this?  Am I approaching this in a sane manner?  Has anyone solved this problem, or similar, in a better way?

Thanks,

Brian.

[blgentry]

Ok, let's rewind.  Here's what I'm trying to do:

I've got a 64 GB card for my handheld (Fiio X1).  My music library is quite a bit bigger, so I want to control what goes on it.  I've got a medium sized list of albums and songs that I want to guarantee will be on the handheld.  I've also got a small list of stuff that I could care less if it ends up on the handheld.  It would mostly be wasted space because I hardly listen to it.  Everything else I'd like a random sampling of.  I prefer full albums, but it wouldn't be a 100% requirement for my project. 

How would any of you who are good at this go about doing my project?

Thanks!

Brian.

[glynor]

That helps, Brian. Thanks. I'll try to reply more later (got Daddy duty right now). But if I'm not too tired after putting her to bed, I'll try to write up some (hopefully not too jumbled) thoughts.

[glynor]

Here's what I'd do, to accomplish that.  You need (at least) four Smartlists:

1. Handheld Files to Include:

• This contains all of the files you want constantly synced over to the device.
• You probably already have this, but a good way to do this is a combination of two "methods".
  
  • One or more "manual" Playlists, which you can just add files to when you want them always manually included. This is handy for stuff you might add and then take back off again.
  • A list that is auto-populated based on a combination of Ratings and other playback stats (like most played, recently played, etc).

2. Handheld Files to Exclude:

• This contains all of the files you want blocked from syncing over to the device in the random shuffle.  I'd make it its own list to keep things simpler.

Example:

Code: [Select]

[Media Type]=[Audio] ([Rating]=1 or [Keywords]=[Hide From Views] or [Bitrate]=<96) ~sort=[Artist],[Album],[Disc #],[Track #],[Name]
3. Handheld Shuffler:

• First, add a Playlist Is Not Any "Handheld Files to Exclude" rule.
• Add [Media Sub Type]=[],[Music] to keep it from including any Audiobooks or Podcasts or whatever (and just pre-filter the subsequent list more).
  
• Add your [Genre] is not any rule, and any other rules you want to in order to limit the scope of the resulting list.
• Next, add a ~limit like this:
  
  • ~limit=-1,1,[Album]
    
  • That will spit out a list of individual songs (limited to one per album, but otherwise every single song in your library)
• Add ~sort=Random
  
  • will make sure these songs are good and shuffled.
• Add ~expand=Album
  
  • will expand them to their full albums, but in a random shuffled order, which changes each time the list is opened.
• Add ~s=64000
  
  • will limit the total list size to less than 64GB (you don't need a special calculated value here, just the maximum possible size of the sync).

Example:

Code: [Select]

[Media Type]=[Audio] [Media Sub Type]=[],[Music] -playlistid==706186927 -[Genre]=[Classical],[Electronic\Experimental],[Experimental],[Hair Rock],[Kids] ~limit=-1,1,[Album] ~sort=Random ~expand=Album ~s=64000
This will return a ~64GB shuffled list of albums.

The -playlistid in my example above points to the Handheld Files to Exclude list.

Because, you created a shuffled list of individual songs (with no two songs from a single album in the list), and then expanded it.  This results in a massive list of basically all of the possible albums, but their order is shuffled (because the "single-track seed list" it used was shuffled before the ~expand).  Then, if desired, you can just limit it to a particular size with the ~s limit.  You don't really even need this limit, because the next step will actually take care of it, but it is probably a good practice to just limit it here too (since you'll never need more than 64GB total).

Order of the modifiers matters!  You can do one, then another, to get a list how you want before limiting it.  It reads the Search Language from left to right (which correlates to from top to bottom in the Wizard interface).

4. Sync List

• Add Playlist is any Handheld Files to Include list.
• Add separately Playlist is any Handheld Shuffler list.
• Open Import/Export and add parenthesis around the whole thing and add "or" to the middle, so it looks like:
  
  • (playlistid==218414500 or playlistid==407790842)
    
  • This will add the two lists separately to the results (triggering a "match" on all of them in the Including list first, and then afterwards the ones in the Shuffler list you made.  This is key, because it allows the next step to work right.
• While you're still in the Import/Export, add this to the end:
  
  • ~seq ~sort=[Sequence] ~s=64000

Example:

Code: [Select]

(playlistid==218414500 or playlistid==407790842) ~seq ~sort=[Sequence] ~s=64000
Playlist ID 218414500 in my example above points to my example Files to Include smartlist, and 407790842 points to the Handheld Shuffler example list above.

This will generate a 64GB list, with all of them in the "forced included" list at the top, and then all of those from the "randomized shuffle" list further down.  The "magic" is in the adding those two lists separately, and then the ~seq modifier.  It adds a sequence to the files as they're added to the Smartlist. You can then sort by this Sequence, which puts the files in order with those from List A on top, and those from List B on the bottom.

Then, the limit happens after the Sequence ordering magic, so the ones "lopped off" will always be a random assortment of those at the "bottom" of the list. So you will get one "partial" album at the bottom (where it has to cut them off to keep the whole thing under 64GB), but only that one, because it will "randomly" cut off wherever it gets to 64GB.

Then, sync only that final Sync list to your device in MC's Handheld setup.

It works perfectly. I tested as I was writing this up.

Be aware that you CANNOT CREATE OR MODIFY these lists (since they contain references to other Playlists) while MC is serving a Library and has any other copy of MC connected (even if the client is idle). Otherwise, it breaks those lists.  Once you've made them, it is fine to use them with a client connected (or even from the client, though I don't know why you'd want to) but if you need to tweak them, you have to kill any connected clients.  Because of the bug.

Bonus Points:

In the Handheld Shuffler list you could replace the ~sort=Random with a ~mix rule to "weight" the results to higher-quality songs.  Say, for example, you include all 5-star rated tracks automatically in your "Files to Include" list.  But, you would like a higher-percentage (than pure chance) of the random albums that come out of the shuffler to be "expanded" from a track rated 3 stars or greater?

Then you could do ~mix and set it to something like 50% equal to 4 (or maybe greater than 3) and 50% less than or equal to 3.  Example:

Code: [Select]

[Media Type]=[Audio] [Media Sub Type]=[],[Music] -playlistid==706186927 -[Genre]=[Classical],[Electronic\Experimental],[Experimental],[Hair Rock],[Kid's] ~limit=-1,1,[Album] ~mix=10000,50%,{[Rating]=4},50%{[Rating]=<=3} ~expand=Album ~s=64000
That way, the shuffle list you make will be "weighted" to be more likely to include albums with highly rated tracks. You can play with the percentages until you get it how you like it.

[blgentry]

Wow, what a fantastic reply!  Thanks Glynor, this has several bits of JRiver magic that I was looking for.

I've just tested this and it seems to work exactly as described.  I have some questions that I'll put in my next reply.  I just wanted to isolate this reply to say THANK YOU! 

Brian.

[blgentry]

Ok, I understand most of this.  The shuffler list in particular was something I was missing.  It's exactly what I was looking for without the weird stuff I was doing to try to emulate it.  Cool.

The one that's got me right now is the Sync List.  It doesn't seem to work as I first thought it did.  In fact, it seems sorta broken.  My first clue about this was as I was building the list.  I built the rules in the order you specified, and applied the rules at each stage to see how it was working so I'd understand each step.  What got me was that applying a Sequence number seemed to change the order so it wasn't in album order any more.  It seemed to work though (overall), so I just kept going.

But the sequence numbers are definitely not in album order.  I got my first true indication of this when I was testing syncing and a new album I got is cut down the middle.  It's missing the middle few songs.  I did a bunch of testing and found out this wasn't the only one.

Let me paste in my exact list here so we can be on the same page:

Code: [Select]

(playlistid==937102607 or playlistid==1867788440) ~seq ~sort=[Sequence] ~s=64000
The first playlist ID is my include list.  The second is my Shuffler list.  Looking at the sequenced output, it's not quite random, but it's certainly not ordered in the same order as the playlists are.  For the same album, the Shuffler list shows the songs in album/track order.  But for that same album, the Sync list shows the songs in a strange order.  This is when sorting by sequence in both lists.  I added sequence at the end of the Shuffler list as a troubleshooting step.  It didn't seem to change anything; it's just for troubleshooting.

So at the moment I'm stuck because the output of the Sync list is cutting up some of my albums.  This is very unexpected behavior, but I've verified it.  The list is ordered by album until the Sequence is added to the Sync list.

Am I doing something wrong, or have I found an unusual behavior?

Brian.

[blgentry]

In case what I'm saying isn't clear:

Something about the sequence number and the Limit is causing it to remove songs not in album order.  To explicitly test this, I made another smartlist.  It's job is to show me all of the songs that are in Shuffler, that are NOT in Sync.  If it's operating properly, I should see several (or many) complete albums.  Then I should see a group of songs from *one* album that are incomplete.  Because the Sync limit should cut the Shuffler list off at a certain point, and it's sorted by albums.  So it should only make one incomplete album.

Here's my list that shows me the removed songs:

Code: [Select]

[Media Type]=[Audio] playlistid==1867788440 -playlistid==1062206144 ~sort=[Artist],[Album],[Track #]
Scrolling through this, I see something like 10 complete albums and something like 4 to 6 incomplete albums.  Every time I refresh it, it changes (which is expected).

I'm confused by this behavior.  Anyone have any insights?

Brian.

[glynor]

I did notice some ordering issues, though it definitely ended up with List A on top and List B on the bottom.

I suspect it is because ~seq adds sequence numbers, as I mentioned, as the files are added to the resulting Sync List.  So, it is going through your entire Library, less any files excluded by previous filters, to see if they qualify and giving each file a Yes or No.  It does NOT open up Playlist A, and then add those, and then open Playlist B, and add those.  It goes through every file in your Library and checks them all.  It does this in a particular order, but I don't know what that order is. Perhaps by File Key in numerical order?

It does work to use the OR between the lists to keep those in order, but when filtering one list against another, I don't know if you can keep the order of the lists themselves precisely in order.

In my tests, though, the stuff at the bottom wasn't wildly out of order (tracks on albums were still, generally, grouped together, I think).  The stuff at the top (from the Files to Include) list, I noted, were out of order. All of my Music was on-top of my Audiobooks in my test list, whereas in my source "Files to Include" list I had them sorted by [Media Sub Type] and Audiobooks were up top.  Though I think the different books and albums were still generally together (perhaps tracks out of order, but who cares if they're all there).

I didn't check it extremely thoroughly, as I thought it was "within the margin of error" for your requirements.  And, unfortunately, my test lists are all broken because I didn't follow my own advice and:

Be aware that you CANNOT CREATE OR MODIFY these lists (since they contain references to other Playlists) while MC is serving a Library and has any other copy of MC connected (even if the client is idle). Otherwise, it breaks those lists.  Once you've made them, it is fine to use them with a client connected (or even from the client, though I don't know why you'd want to) but if you need to tweak them, you have to kill any connected clients.  Because of the bug.

Yeah, I had the HTPC copy running upstairs when I did my testing for you, so those test lists I made when testing for you are all obliterated.  I guess I could copypasta them back and verify. Maybe later.

But, I don't think that is solvable. If you have two lists, and you want to filter one against the other and have them be in precisely the same order, I think you're out of luck. But getting List A to always be on top of List B, you can do.  Since List B is clearly of "lesser" importance (and randomly generated) I figured any shuffling or disorder is of limited consequence.  A few of the albums that otherwise wouldn't have been might end up "partial", but when I looked at it, it wasn't a TON of them.

Perhaps this, though, depends on your original Library's data set?  If it is by File Key, then maybe mine generally group simply because my Library is OLD.  So, albums ripped together or downloaded together have sequential file keys, roughly.  Where you imported your whole Library at once only a year or two ago, so it got File Keys based on whatever crazy optimized filesystem reading logic Auto-Import uses?

I don't know.  Were you seeing a complete and utter shuffle at the bottom of the Sync List? I certainly wasn't for either "Portion A" or "Portion B".

[glynor]

Yeah. I fixed the lists and verified. I don't know for sure without getting crafty and looking up File Keys for tracks, but it looks to me like it is checking through the Library by sequential File Key.

The vast majority of mine are still grouped by album within the Sync List, even down in "section B" for the randomly added albums (and even, usually, in sequential order by [Track #]).  I do have a bit of this kind of thing in every "run" of the lists:



But far more of them are like this:



I can tell by the groupings that there is some kind of logic.  Probably by File Key. Looked at roughly by eye with a few runs of my lists, that things that were imported together are grouped together in the "section B" of the Sync List.  I do see large swaths of files "together" by Artist in my Sync List. I assume this is because I'll often rip or buy more than one album by a particular artist at a time (I'll rip a whole discography at once, and then move onto the next Artist, or whatever).

In any case, it isn't shuffled anymore at the bottom of the list, but that doesn't matter. The selection that created section B of the list is still shuffled, so it is still a random assortment of albums (and not "all of the As, and then Bs, and then Cs, and no Ds, etc).

This is, however, more of a reason to ensure you add that 64GB limit to the Shuffler, though (and so not just for good practice, you need it). If you omit this part, then every possible track will be "included" as part of the shuffler in section B of the sync list, and you will probably always get As, Bs, Cs, and not Ds, etc.

So make sure you don't skip that.

[glynor]

But in Section A of the Sync List (which is generated by my Files to Include source list) the files are always in exactly the same order.  They aren't in the same order as the Files to Include list, but it doesn't vary from run-to-run of the Sync List (as the Section B at the bottom does).

So, in my case, I have all of my Avett Brothers songs added to my Files to Include list (which is pulled from my real files to include list used by my iTunes syncing system that I have). For whatever reason, the first track off of Four Thieves Gone is always stuck in the middle of the two Gleam albums at position 29, and the rest of that album is down after Brothers at position 53 and on:


I don't know why. Maybe I got just that track originally and then later added the rest of that album?  That could be, though I'm not sure. In any case, there clearly is an order it uses to evaluate the tracks, because the Section A portion of the list is always in precisely the same order.

Looking at it further, one extra thing you should do to the Handheld Shuffler not mentioned above is filter out any tracks where [Album] is empty.  Because otherwise, those will always expand "together" (all tracks in your Library where [Album] is empty are "part of" one huge album, as far as the ~expand logic is concerned).  If you have a lot of tracks with no [Album] tag at all, then those could completely fill your Handheld Shuffler list if one of them happens to get randomly selected close enough to the top of the seed list to end up in the final resulting list.

So, modify the shuffler to something like:

Code: [Select]

[Media Type]=[Audio] [Media Sub Type]=[],[Music] -[Album]=[] -playlistid==706186927 -[Genre]=[Classical],[Electronic\Experimental],[Experimental],[Hair Rock],[Kids] ~limit=-1,1,[Album] ~sort=Random ~expand=Album ~s=64000

[glynor]

Two last things worth mentioning, probably, since we're being precise:

Files to Exclude and the Shuffler:

The Handheld Shuffler list I gave above can include tracks that are otherwise forbidden in certain circumstances. That's because the Files to Exclude list check only applies to the seed for the list, and the ~expand happens afterwards.

So, if an Album in your Library contains files that are both "okay" and those "excluded", then it will expand those because it is trying to include the entire Albums.  Those excluded files are only excluded from the "seed" it makes originally before it expands them. This happens with my Library in two instances:

• One star files on an album with other tracks that aren't one-star.
• Alternate "versions" of albums (which have an identical [Album] tag) but which are excluded by my "Hide from Views" magic keyword.

Those latter ones are usually, in my case, tracks I've gotten from sources I'm unsure about and haven't sorted through yet. So, I might have two versions of the same album in my Library, but I'm not sure which version was ripped with the best quality, and I just have one "set" excluded with my Hide from Views magic keyword until I get around to sorting through them.

If you want to make sure that it never includes those files in the Section B of the Sync List, regardless if they are on the same album as "permissible tracks", then you need to filter that list again in the Sync List.  You can do this with another AND check "inside" the OR check you already have.

Code: [Select]

(playlistid==218414500 or (playlistid==407790842 -playlistid==706186927)) ~seq ~sort=[Sequence] ~s=64000
In the example above, the Playlist IDs correspond to:

• 218414500 - Files to Include list
• 407790842 - Handheld Shuffler
• 706186927 - Files to Exclude list

So is logically:
(Files to Include OR (Handheld Shuffler AND NOT Files to Exclude))

That way, it will include ALL files (regardless of exclude status) on the Files to Include list. But it will only include files on the Handheld Shuffler list if they still pass the Files to Exclude list filter.

Duplicates:

You could, especially if you have a LOT in your Files to Include list, end up with a lot of files that are in both the Files to Include list and which (randomly) end up in the Handheld Shuffler list.  This will make them show up once in Section A and once in Section B of the final Sync List, which wastes a little space if there are a lot of them, because they won't actually get synced twice (the Handheld Sync function will filter them out).

So, you should filter out duplicates in the Sync List before you do that last limit to 64GB step. So, here's a better example that will filter out exact track matches, as well as re-apply the Files to Exclude list (as described above), before limiting the whole thing to 64GB.

Code: [Select]

(playlistid==218414500 or (playlistid==407790842 -playlistid==706186927)) ~seq ~sort=[Sequence] ~nodup=[Album],[Artist],[Disc #],[Name],[Track #] ~s=64000

[blgentry]

I don't know.  Were you seeing a complete and utter shuffle at the bottom of the Sync List? I certainly wasn't for either "Portion A" or "Portion B".

Not a significant misordering.  Just enough that it's breaking up somewhere between 4 and 8 albums each time.  By "breaking up" I mean a random selection of tracks from the affected albums.  Which isn't a huge big deal, but my old solution doesn't have this issue.  Just for completeness, here's a shot of the bottom of my Sync list.

Brian.

[blgentry]

Answering a few questions without relevant quotes:

1.  My library is pretty clean.  I just checked and I have ZERO songs that have an empty [Album] tag.  Yay! 
2.  With my earlier experiments I had already figured out that you need to exclude the include list when you generate the Shuffler list.  Makes sense, as you said:   Because it will end up doubling those files if the Shuffler picks files that are already in the Include list.
3.  Your Exclude trick is nice.  I had figured out that if I wanted to exclude some tracks on an album, but not others that the Album expander was going to break that.  Doing it again at the VERY end fixes that.  Nice!

Thanks for all your help with this.  It's kind of a beast of a problem.

Let me take a different tack on this for a moment.  I'm using my handheld in ways that I hadn't anticipated when I bought it.  Namely, I tend to play a big playlist *most* of the time (maybe 500 songs).  Every now and then I play albums not in the Big List.  But not often.  So I wonder if I'm making this overly complex for no good reason.

What is your preferred approach to Handheld sync?  Do you have any handhelds that have the kind of storage capacity mine does?

Thanks again.

Brian.

[glynor]

Yeah. I'm not positive it is by File Key, but it is evaluating tracks in some kind of consistent order.  Probably by File Key, but who knows (well, Matt probably does).

In any case, I think for your purposes, you can ignore it unless you are incredibly anal. Most of the albums in the Shuffler will be full albums. You might get a few that are partial or missing tracks, but... Meh, they're the "slop" anyway.

Better to accept that then have to manually adjust your Sync Lists each time you add a few tracks to the Files to Include list, IMHO.

[glynor]

2.  With my earlier experiments I had already figured out that you need to exclude the include list when you generate the Shuffler list.  Makes sense, as you said:   Because it will end up doubling those files if the Shuffler picks files that are already in the Include list.

If your Files to Include list is static (all manually added) then this will work fine. However, in my case my Files to Include list consists, itself, of a scheme kind of like this whole thing.  It is:

• Files I've added to my manual Always Sync list.
• Listened to in the Past Two Weeks smartlist
• 500 Favorites smartlist (a random assortment of 4 and 5 star tracks)
• 100 Most Played (Recent Only) smartlist (the 100 most played tracks, filtered to exclude tracks that haven't been played at least once in the past year).
• Some New, Some Popular smartlist (a fancy ~mix based smartlist that has a mix of 3-star and better tracks along with those imported in the past 4 months or so).

And so on and so forth. Since there are a few "randomized" mixes in these lists, then the contents of my Files to Include master list changes (slightly) each time it is run. So, you can't exclude it directly in the Shuffler, because that won't "catch" them all.  It is safer to do a ~nodup modifier before you do the final size limit.

When I did my tests, I actually just pointed the Files to Include at my real "manual" Always Sync playlist (which is why my Section A never changes at all, even the Sequence numbers).  But, if I was doing this for real, it would use my current Sync List system which includes all of those other lists, and even the Files to Include list would "shift around" a bit.

In my case, I use iOS devices, so my scheme syncs over to iTunes, and then iTunes syncs those lists to my iOS devices. It works similarly, except I have lists in MC set to sync EVERYTHING that could possibly matter (basically my whole music Library) over to iTunes, and then MCiS syncs individual lists to iTunes which are used to sync my devices.

[blgentry]

Thanks for all the effort explaining this.  It's a lot to take in at once.

I'm going to think about this over the next hours and days and see where I want to go from here.  I suspect I'll be changing over to a more static playlist driven system, or maybe modifying my old method.  I'm not super duper anal about this stuff, but having broken albums definitely bothers me.  I noticed it broke one of my newest (and awesome!) albums.  I guess if I really cared, I could just add that album to the include list.  Duh. 

BTW:  Daft Punk, Random Access Memories.  Just WOW. 

Brian.

[glynor]

BTW:  Daft Punk, Random Access Memories.  Just WOW. 

That is a great one!

You may want to consider making your Files to Include list a more complex beast like I described above. My system isn't directly applicable to your situation because I can't use MC to sync directly to my devices (they're all iOS devices, so have to sync through iTunes).

But, I did face similar issues.  I didn't include the titles of all of my "always sync to my devices" lists above, because it varies a bit depending on the device (there are separate lists for my phone vs my iPad vs my wife's phone, and so on).

One of the things I do, though, is have a Smartlist included that:
* Includes all tracks imported in the last X weeks (I keep changing how many weeks so I'd have to look it up to be sure what it is set to now).
* Expands those to full albums (in case I downloaded a particular track and then went back for the whole album later)

That stops the "ugh, the new album I want isn't included fully" issue.

Of course, I almost never actually sync stuff to my iOS devices anymore except when we're going on trips where I know there will be no cell service (as happened recently with our camping trips).  But otherwise, I just use JRemote, and so I have the whole thing with me everywhere.