INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: [Feature Request] Improved Support for Files with Media Type "Data"  (Read 19758 times)

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient

So I've recently taken a stab at integrating a number of "Data" type files into media center to take advantage of Media Center's excellent database and library server functionality.  But there are a lot of little roadblocks and inconveniences with Data files that make them hard to handle like other files, and I think Data support could benefit from a few tweaks (most of which are probably pretty incremental).

Basic functions:

1) Basic Cover Art Support: Currently no cover art operations work on Data type files.  One can "trick" Media Center into allowing cover art to display by changing the media type of the files to "Video" performing the cover art addition, and then changing the Media type back to "Data."  At that point the cover art will "stick."  So it doesn't appear to be a limitation of the file type, those functions are just disabled for "Data" files. The use case: adding E-books (which are supported by default), or any other data files with cover art (Roms, Maps, etc.) to JRiver.

2) Sidecars: Currently there's no way to get JRiver to write a sidecar file with metadata for a Data file (the "write sidecars" option only allows Video to be chosen).  It would be nice to have Data sidecars for the same reasons we have them for video files, metadata backup, library migration, etc.  It would also be nice as one could then edit them to add xml metadata from scraping programs (as MC lacks scraping capability for data files).  The use case: the same as sidecar files for Videos, but also programs like Calibre offer automated metadata scraping for e-books and store that metadata as .xml files next to the files; if JRiver would write (and read) sidecar files for data files I could easily reformat Calibre's .xml metadata into a JRiver sidecar and programmatically migrate that data.

3) The "Launch with External Program (default)" option does not appear to work correctly with data files.  On a client machine, when I attempt to open a data file that I have local access to using the launch with external program (default) option, I would expect that function to pass the local file to the system default program for processing those files.  Instead MC opens a web browser and attempts to download the file, requesting the authentication password everytime as though the machine didn't have local access to the file and had never connected to the server before.  However, when I use "Launch with External Program (custom)" and pass the filename as an argument, it works correctly (showing that I do have local access to the files).  It's possible that it just doesn't respect the "Use Local File if Available" switch under media network (which is checked in my case).  Whatever is causing it, this makes using data files in a client/server environment much more challenging as one needs to specify a custom launcher command line for each file type.

More advanced functions that could be helpful, but would probably require more work:

1) Advanced Cover Art Support: It would be nice if JRiver could create a thumbnail for certain common data file types the same way it does for images or videos.  For example, pulling the first page of a .pdf file as a thumbnail would be much better than having no cover art at all.  I recognize that's would require a good bit more work than basic cover art support, but it would be nice to have auto-thumbnailing support for common data file types.  

2) In-File Metadata Support: Another nice to have would be to write and read metadata to/from common data files that support it.  For example, .epubs and .pdfs have some in-file metadata support and being able to read that and/or write it in JRiver would be nice (and would provide another avenue for migrating metadata from programs that auto-scrape). I expect that would be a good bit more work than just enabling sidecar writing though  ;D.

These obviously aren't exhaustive, but the first three in particular seem like small things that could be done to dramatically improve usability/user experience with data files in MC.  All feedback is welcome.

Some neat uses of data files that folks (shoutout to syndromeofadown for pioneering them) have done in JRiver that would be made easier by these changes (some of these issues are discussed in those threads):
Video Games: https://yabb.jriver.com/interact/index.php?topic=78049.0
Ebooks and Comics: https://yabb.jriver.com/interact/index.php?topic=87545.0

Logged

glynor

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 19608
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #1 on: February 16, 2016, 02:07:40 pm »

One note: For all Data file types, when you leave the Play Method set to Automatic, it is equivalent to Launch with External Program (default). Does it work right from a client when you just leave it alone and use Automatic?

I haven't tried. The Library I use for most "data" file management is used locally, and not frequently over the network, so I haven't noticed this.
Logged
"Some cultures are defined by their relationship to cheese."

Visit me on the Interweb Thingie: http://glynor.com/

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #2 on: February 16, 2016, 02:56:30 pm »

One note: For all Data file types, when you leave the Play Method set to Automatic, it is equivalent to Launch with External Program (default). Does it work right from a client when you just leave it alone and use Automatic?

I haven't tried. The Library I use for most "data" file management is used locally, and not frequently over the network, so I haven't noticed this.

No "Automatic" doesn't work in that context either, it opens a browser and presents the authentication dialog, same as "Launch with External Program (default)."  It's frustrating because "Launch with External Program (custom)" works passing nothing but the [filename], so it's obvious that something isn't working quite right with the other settings.  MC appears to be missing/ignoring the fact that the file is locally available and trying to "getfile" from the server.

I'm trying to use MC as an E-book and nes/snes ROM manager, and it's mostly working well with the workarounds, but making changes to cover art is fiddly, and getting detailed metadata into MC is proving daunting.  I've got piles of nicely scraped metadata, just waiting to go  ;D
Logged

syndromeofadown

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 805
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #3 on: February 16, 2016, 06:38:44 pm »

Your basic requests are all great. I currently use MC for video games, books, and comics.

Quote
1) Basic Cover Art Support
Though I have gotten used to the work around on my desktop, it is very annoying to do on a touchscreen tablet every time a file is added.

Quote
2) Sidecars
I would like them for marking books and comics as read.
Also when transferring files to another PC I wouldn't have to worry about naming conventions for Tag on Import.

Quote
3) The "Launch with External Program (default)"
This doesn't affect me personally because I'm not currently using a network, but there have been reports of the problem you noted.

I believe 1 and 2 would benefit users of SACD iso's too.

I would be extremely happy if just 1-3 were added to MC.


A couple notes about the advanced requests.
Quote
In-File Metadata Support
Calibre can scrape. It stores a opf file either inside or beside epubs(it's up to you).
Comic Rack can scrape. It stores metadata in "cbz" only by putting an xml file inside.
Is there currently any way to store scraped video game info outside of a database?

An issue is the number of file types users may have.
For comics:
cbz/zip/cbr/rar/cbt/tar/cb7/7z/pdf/djvu

For books:
AZW, AZW3, AZW4, CBZ, CBR, CBC, CHM, DJVU, DOCX, EPUB, FB2, HTML, HTMLZ, LIT, LRF, MOBI, ODT, PDF, PRC, PDB, PML, RB, RTF, SNB, TCR, TXT, TXTZ

For comics I only use cbz and for books I only use epub, but it's unlikely that's typical.
Logged

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #4 on: February 16, 2016, 08:45:06 pm »

Is there currently any way to store scraped video game info outside of a database?

There are third party scrapers that leverage the work emulation station has done. For example, this script (https://github.com/sselph/scraper) will scrape a directory full of roms, pull art for them, and pull metadata into a single .xml file in just a few minutes.  With a little ingenuity it can be made to produce separate files for each rom (putting each one in a separate directory, etc.), or the single file can be parsed and split into chunks for each rom.  I know of no program that will store anything in the ROM files, but that's why I'd like to have sidecar support  ;D

Quote
An issue is the number of file types users may have.
For comics:
cbz/zip/cbr/rar/cbt/tar/cb7/7z/pdf/djvu

For books:
AZW, AZW3, AZW4, CBZ, CBR, CBC, CHM, DJVU, DOCX, EPUB, FB2, HTML, HTMLZ, LIT, LRF, MOBI, ODT, PDF, PRC, PDB, PML, RB, RTF, SNB, TCR, TXT, TXTZ

For comics I only use cbz and for books I only use epub, but it's unlikely that's typical.

I agree it's potentially an issue, but I think it's less of an issue than it seems for a few reasons.  Just taking the e-books (as I don't know much about comics):
1) Some of the listed formats are not particularly common as e-book formats (e.g. DJVU).
2) Many of those formats are not supported by common e-readers, which is why they're not common for e-books (e.g. several of the "office" file formats aren't well supported on most e-readers)
3) Several of the formats are freely convertible to other formats, and
4) Some of those formats don't support embedded metadata in any meaningful way (i.e. txt)

So it seems like, if in-file tagging support were to be added, it would make sense to go after a handful of the most common formats first. The only e-book specific file types MC currently supports by default (as opposed to basic office document types, like doc, docx, odt, rtf, or txt) are cbz, epub, pdf, and mobi.  I think those four would be a reasonable place to start as they're fairly common formats.  The only very common format not on that list are the various flavors of AZW, but most of the AZW files out there are DRMed IME, so would be tough to manage in MC in any case (unless one wants to break the DRM, in which case they can be easily converted to epub or mobi anyway).

So I think hitting those four would cover a lot of bases.  But I'd be happy just to get the basics sorted out.
Logged

ferday

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1732
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #5 on: February 17, 2016, 06:44:11 am »

I will +1 for sure

I actually (try to) use MC for documents and other hard data.  Because I do project work that may have 1000's of reports in every conceivable format, and 10,000's of pictures, split over dozens of job numbers and stuff, MC is super useful especially with custom fields and the mighty RMC tool

But it's just not quite there (not that I expect it, it's a media program after all) but any improvements would be welcome.  Easier handling of file types is one I'd like (rather than manually changing the config to include office formats).  It's such a powerful database setup, I'd like to leverage it more...and I'm already using it for tunes while I do the paperwork anyways :)
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #6 on: February 18, 2016, 01:37:28 pm »

Coming next build:
Changed: If a data file is played over library server and it's less than one megabyte, it will be downloaded locally and then played.
Logged
Matt Ashland, JRiver Media Center

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #7 on: February 18, 2016, 01:47:09 pm »

Coming next build:
Changed: If a data file is played over library server and it's less than one megabyte, it will be downloaded locally and then played.

I'm not sure I fully understand the change, but is that aimed at resolving issue 3) above?  I'll test regardless once the new build arrives, thanks for the attention!
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #8 on: February 18, 2016, 01:49:17 pm »

I'm not sure I fully understand the change, but is that aimed at resolving issue 3) above?  I'll test regardless once the new build arrives, thanks for the attention!

Yes, it addresses issue 3 from above.
Logged
Matt Ashland, JRiver Media Center

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #9 on: February 18, 2016, 01:51:44 pm »

Yes, it addresses issue 3 from above.

Awesome, I'll report back once the build is up  ;D
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #10 on: February 18, 2016, 02:13:03 pm »

1) Basic Cover Art Support: Currently no cover art operations work on Data type files.  One can "trick" Media Center into allowing cover art to display by changing the media type of the files to "Video" performing the cover art addition, and then changing the Media type back to "Data."  At that point the cover art will "stick."  So it doesn't appear to be a limitation of the file type, those functions are just disabled for "Data" files. The use case: adding E-books (which are supported by default), or any other data files with cover art (Roms, Maps, etc.) to JRiver.

I'm confused, because I just added art with Cover Art > Add From File... on some data files and it worked great.

Thoughts?
Logged
Matt Ashland, JRiver Media Center

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #11 on: February 18, 2016, 02:18:48 pm »

I'm confused, because I just added art with Cover Art > Add From File... on some data files and it worked great.

Thoughts?

Quick Find in Directory definitely doesn't work (but works fine if you change the file type to Video first); can you try that with an image with the same name as the data file in the same directory? I thought I had also tried add from file and copy/paste as well, without success, but I'll test again when I get home.
Logged

rudyrednose

  • Regular Member
  • Galactic Citizen
  • ****
  • Posts: 344
  • nothing more to say...
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #12 on: February 18, 2016, 02:46:38 pm »

+1
Thank you for this thread mwillems.
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #13 on: February 18, 2016, 02:48:49 pm »

Next build:
Changed: Doing a Cover Art > Quick Find in File / Cover Art Directory on data files now works.
Logged
Matt Ashland, JRiver Media Center

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #14 on: February 18, 2016, 05:50:51 pm »

Next build:
Changed: Doing a Cover Art > Quick Find in File / Cover Art Directory on data files now works.

Aces!  Will test when the build's up.  I checked and adding from a file does currently work, but copying and pasting does not (for me).  Somebody else might want to test that to confirm, though, as I'm remoting into my server and clipboards can be a little flaky with remote desktop software.
Logged

syndromeofadown

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 805
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #15 on: February 27, 2016, 04:55:53 pm »

Quote
Changed: Doing a Cover Art > Quick Find in File / Cover Art Directory on data files now works
Thanks for this.
I just transferred some new books to my tablet. Auto import picked them up immediately and art showed up automatically.
It was an absolute pleasure.

Logged

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #16 on: February 27, 2016, 05:47:19 pm »

Thanks for this.
I just transferred some new books to my tablet. Auto import picked them up immediately and art showed up automatically.
It was an absolute pleasure.



It's working well here too, and I noticed the beneficial side effects on auto-import earlier today.  It's quite nice to have them come in like any other file.  I'm having some difficulty with the other change because about half of my files are over 1MB, but for the smaller files it works well.
Logged

syndromeofadown

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 805
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #17 on: February 27, 2016, 06:10:37 pm »

Quote
I'm having some difficulty with the other change because about half of my files are over 1MB, but for the smaller files it works well
I'm not using a network but here are my typical file sizes:
epubs are between 0.5MB and 10MB
cbz are between 1MB and 1GB with the majority between 100MB and 400MB.
pdf are between 27MB and 1.4GB
roms are between 0.5MB and 32MB until you get into disc images the size of CD, DVD, and DL-DVD.
Logged

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #18 on: February 27, 2016, 06:18:29 pm »

I'm not using a network but here are my typical file sizes:
epubs are between 0.5MB and 10MB
cbz are between 1MB and 1GB with the majority between 100MB and 400MB.
pdf are between 27MB and 1.4GB
roms are between 0.5MB and 32MB until you get into disc images the size of CD, DVD, and DL-DVD.

That's mostly my experience too.  A narrow majority of .epubs are under 1MB, but none of my pdfs are that small, and most of my ROMs are bigger than that too.

Really I'd be happy if local access worked correctly without asking the server to serve it when it doesn't need to.  That would solve files of any size in my use case, although a more robust server solution would be handy in the long term.
Logged

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #19 on: May 19, 2016, 01:54:31 pm »

So I wanted to gently bump this for an issue related to item three above which was partially implemented (Thanks Matt  ;D).   Matt made the following change: "If a data file is played over library server and it's less than one megabyte, it will be downloaded locally and then played."  This works great for data files that are less than 1MB, but doesn't address the issue for data files that are larger than 1MB, which are almost all of the pdfs and Roms in my collection.  

To restate the case for why this is important:

1) MC can't generally play data files itself, and needs to rely on an external program to do so.  Most of those programs are expecting to be handed a complete file, so the server's default "streaming behavior" doesn't work.
2) If the client has direct access to the file, you can work around the issue a little bit by specifying the external program and passing the path to the file as a parameter.  The default external program options don't work though, so it's a bit of work, but does allow for a usable setup with files of all sizes.  
3) However, if the client doesn't have direct access to the file (i.e. needs to rely on the server) there is currently no way to play data files over 1MB in size on the client. This has implications for cross-platform server client setups: because, for example, a windows server and Linux client can never have the same path to a file, the "play local file if one can be found" option will never work, and the JRiver [Filename] tag will be "wrong" for the client.  So Linux clients of a windows server can't play any data files over 1MB, no matter what, and vice versa with a Linux server and Windows client, etc.  

So while the 1MB download was a good first step (it allowed for many .epubs to be used), I'd like to ask if there's a chance we could get the same "download-then-play" behavior for data files larger than 1MB?  Ideally there would be no upper limit, but if there needs to be a "cap" making it 1.5G or 2G would cover the vast majority of cases.  That may sound extreme, but even quite small .pdfs can be 30MB or 50MB, and there really is no upper limit on .pdf size.  Syndromeofadown notes that his largest .pdf is 1.4G, my largest is about 700MB, for a reference.

I understand if there are barriers, but I wanted to ask again to see if it was feasible.
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #20 on: May 21, 2016, 05:54:50 pm »

What if we just made the limit 100 MB for PDF files?  Good or bad idea?
Logged
Matt Ashland, JRiver Media Center

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 71465
  • Where did I put my teeth?
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #21 on: May 21, 2016, 06:14:39 pm »

I can see the logic of having no limit on certain types of files.  If a PDF is 10GB, why not download it first?
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #22 on: May 21, 2016, 06:19:30 pm »

I can see the logic of having no limit on certain types of files.  If a PDF is 10GB, why not download it first?

Well because MC would just hang forever while it's downloading.  We're not displaying any sort of status with a cancel button.
Logged
Matt Ashland, JRiver Media Center

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #23 on: May 21, 2016, 06:30:11 pm »

What if we just made the limit 100 MB for PDF files?  Good or bad idea?

I wouldn't want it to be restricted to .pdfs, because the current 1MB limit is a problem for the other formats too.  Almost half of of my .epubs are bigger than 1MB (the largest is 15MB), most of my comic book files (.cbz) are larger than 1MB, and 90% of my ROMs are larger than 1MB.  So ideally the limit could be raised across the board.

If you're willing to consider 100MB as an across the board limit, that would present fewer problems than a 1MB limit, but it would leave some files out.  About 5% of my .pdfs are larger than 100MB, but many ROMS and comic book files (.cbz) are also over 100MB.  That said 100MB across the board would allow almost any conceivable .epub, any ROMs from before the CD era, and the majority of .pdf files.  So much better for sure, but not perfect.

The 100% solution is to have no cap or a cap in the gigabyte range.  I guess I'm just used to seeing my MC server stream 30GB Blu Ray quality movies without batting an eye, so I instinctively want it to be able to manage these much smaller files (but I know full well the technical issues presented aren't the same  ;D )


Well because MC would just hang forever while it's downloading.  We're not displaying any sort of status with a cancel button.

Maybe the answer is to show a dialog or warning when the file size is over a certain level (i.e. "You've chosen to open a data file larger than 100MB, this might take a while")?   Maybe the download could leverage the built in MC downloader so it can download in the background?  Just spitballing.

As it is, if someone had to wait a bit, it's not really any different than what happens with giant images in the client-server context (i.e. I have a 37MB .png that takes about 10 full seconds to load on a client while I'm staring at a blank screen).

For my part I'd rather sit and wait for it to work, than not have it work at all.
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #24 on: May 21, 2016, 06:39:53 pm »

I think asking any time it's over 1 MB might work.
Logged
Matt Ashland, JRiver Media Center

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #25 on: May 21, 2016, 06:43:32 pm »

I think asking any time it's over 1 MB might work.

So MC would be able to download any size data file, but it would pop up an "are you sure?" dialog if the data file is over 1MB?  That sounds great to me, I could use MC with all my files then!

Thanks for hearing me out  ;D
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 71465
  • Where did I put my teeth?
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #26 on: May 21, 2016, 06:57:50 pm »

I think asking any time it's over 1 MB might work.
Why not raise that?  An MP3 track is 5MB. 

A 100 Mb network is nominally 10MB per second, or realistically maybe 5MB/s.  A 25MB file should take about 5 seconds.
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #27 on: May 21, 2016, 07:21:55 pm »

Well as an example I sometime play from my home library when I'm at the office.

I have probably 5 megabit per second upload speeds (not really sure).

But it's just not that crazy fast.

Maybe I'm a weirdo running Library Server when you're not sitting on the same LAN, but that is partly the idea isn't it?
Logged
Matt Ashland, JRiver Media Center

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #28 on: May 21, 2016, 07:29:36 pm »

Well as an example I sometime play from my home library when I'm at the office.

I have probably 5 megabit per second upload speeds (not really sure).

But it's just not that crazy fast.

Maybe I'm a weirdo running Library Server when you're not sitting on the same LAN, but that is partly the idea isn't it?

That is definitely part of the idea: being able to access the files whether on the LAN, or across platform, or wherever you want them.  I'll admit, my main use case was on my LAN because the current state prevents me from even accessing the files if I'm on a Linux client with a windows server, but being able to get to them away from home is a big advantage too.

I've got wired Gb connections at home, and away from home, my ISP gives me about 50Mb upspeeds, but I recognize that not everyone has that kind of bandwidth.  So, for my part, I have no problem clicking through a confirmation dialog, so long as I can actually get the file served correctly.
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #29 on: May 23, 2016, 10:16:00 am »

Next build will have this:
Changed: Switched the limit to download a data file at playback time to 5 MB from 1 MB.

I'm still thinking about showing a popup if the file is even bigger.  I haven't decided yet.
Logged
Matt Ashland, JRiver Media Center

RD James

  • Citizen of the Universe
  • *****
  • Posts: 1871
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #30 on: May 23, 2016, 07:57:17 pm »

Does the program know whether the file is being transferred over a local/remote connection?
Perhaps it could only prompt on remote connections.
Logged

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #31 on: May 26, 2016, 08:47:48 am »

Just wanted to weigh in on the change: I did some stress testing, and the change to 5MB is working as expected; it makes the crossplatform scenario (Linux server with Windows clients) a lot more usable for data files (at least the ones smaller than 5MB).  Retro Gaming just got a lot easier in my house!

I appreciate the sustained attention here, and I live in hope for the larger files  ;D
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 71465
  • Where did I put my teeth?
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #32 on: May 26, 2016, 08:52:37 am »

What size would cover most of your needs?
Logged

Awesome Donkey

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 7400
  • The color of Spring...
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #33 on: May 26, 2016, 09:23:20 am »

If he's doing what I think he's doing (retro gaming using cartridge game ROMs, e.g. up to the Nintendo 64 and Game Boy Advance), then likely between 75MB to 100MB would be the upper limit requirement, since I know the largest Nintendo 64 game ROM files is around 65MB.
Logged
I don't work for JRiver... I help keep the forums safe from Viagra and other sources of sketchy pharmaceuticals.

Windows 11 2023 Update (23H2) 64-bit + Ubuntu 24.04 LTS Noble Numbat 64-bit | Windows 11 2023 Update (23H2) 64-bit (Intel N305 Fanless NUC 16GB RAM/256GB NVMe SSD)
JRiver Media Center 32 (Windows + Linux) | iFi ZEN DAC 3 | Edifier R2000DB Bookshelf Speakers | Audio-Technica ATH-M50x Headphones

Hendrik

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 10727
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #34 on: May 26, 2016, 09:31:25 am »

Asking if it should Download such files is definitely sensible, blindly downloading sounds a bit iffy.
Logged
~ nevcairiel
~ Author of LAV Filters

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #35 on: May 26, 2016, 09:35:11 am »

Another approach would be to set the file loader to show a progress UI after 1 second with a cancel button.

Then a user would see what's happening and be offered the choice of cancel.

Thoughts?
Logged
Matt Ashland, JRiver Media Center

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 71465
  • Where did I put my teeth?
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #36 on: May 26, 2016, 09:48:21 am »

Asking if it should Download such files is definitely sensible, blindly downloading sounds a bit iffy.

How about just asking when the file is >5MB.  "This file is 50MB.  Are you sure?"
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #37 on: May 26, 2016, 09:53:26 am »

I think my idea is even a little better because then they wouldn't have to click anything for the "yes" answer, but instead only the "no" answer.

And users that have a crazy fast network so the limit is way too small wouldn't see anything.
Logged
Matt Ashland, JRiver Media Center

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #38 on: May 26, 2016, 11:00:55 am »

What size would cover most of your needs?

I'm currently using JRiver for two data related tasks: Organizing and reading ebooks and .pdfs, and as a retro gaming front end.  As noted above, to get almost full coverage I'd need a cap in the low Gigabytes (1 or 2). My largest .pdf is 700MB, and I have a few dozen that are close to that.  Additionally, ROMs from the CD era are typically 500-700MB.  I have a handful of data files that are over 1GB (DVD era ROMs), but I can count them on two hands so I could live with leaving them out.

That said, I think Matt's proposed solution of a download dialog would really be the way to go, or a user prompt could be good too. Having a fixed cap sort of builds in scalability and support problems (people will be confused about why some files work and some don't, etc.).  Two of MC's greatest strengths are its universality (can handle a huge range of files) and it's scalability with large libraries and large files.  It already does a great job with local data files;  I'm just trying to get comparable performance in the client-server scenario.

Asking if it should Download such files is definitely sensible, blindly downloading sounds a bit iffy.
Another approach would be to set the file loader to show a progress UI after 1 second with a cancel button.

Then a user would see what's happening and be offered the choice of cancel.

Thoughts?
I think my idea is even a little better because then they wouldn't have to click anything for the "yes" answer, but instead only the "no" answer.

And users that have a crazy fast network so the limit is way too small wouldn't see anything.


I think Matt's downloader solution is the best solution I've heard, but an "are you sure?" prompt would work too.  One thing to consider about a confirmation prompt is how such a dialog would be handled in Theater View (one of my main use cases is to use theater view as a retro gaming front end in a client-server environment, so I would need a way to accept the confirmation with a controller or remote), but other than that caveat either approach would work.   My goal is just to be able to get to the files (somehow) in MC.

Matt's proposal would be my preference FWIW.  Thanks again for staying with this.
Logged

Hendrik

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 10727
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #39 on: May 26, 2016, 11:39:35 am »

How about just asking when the file is >5MB.  "This file is 50MB.  Are you sure?"

Of course, keeping it for small files is fine.
I feel like all downloads should show a cancelable progress Dialog though, hanging MC while it downloads is never good.


I also think however that for a truly transparent behavior, we should always recommend to setup local file access through shares etc.
Logged
~ nevcairiel
~ Author of LAV Filters

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #40 on: May 26, 2016, 11:59:33 am »

I also think however that for a truly transparent behavior, we should always recommend to setup local file access through shares etc.

Hendrik, that would be the best solution, but setting up local file access through shares (that MC will utilize) is currently not possible in cross-platform scenarios due to file path issues.  I have a mixed network of Windows clients and a Linux server and the linux filepaths don't allow the windows clients to have direct access through MC.  Fixing the cross-platform file path issue would (mostly) fix the data file downloading issue too, but that seemed like a bigger task to take on.  

But don't get me wrong, I'd be thrilled if MC would allow for a root path substitution for cross-platform file paths.  That would mostly solve this issue and a bunch of other video-related issues too.
Logged

Hendrik

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 10727
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #41 on: May 27, 2016, 02:13:44 am »

Hendrik, that would be the best solution, but setting up local file access through shares (that MC will utilize) is currently not possible in cross-platform scenarios due to file path issues.  I have a mixed network of Windows clients and a Linux server and the linux filepaths don't allow the windows clients to have direct access through MC.  Fixing the cross-platform file path issue would (mostly) fix the data file downloading issue too, but that seemed like a bigger task to take on.  

I would also recommend to not mix platforms then, at least for the time being! :)
Logged
~ nevcairiel
~ Author of LAV Filters

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 41971
  • Shoes gone again!
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #42 on: May 31, 2016, 07:45:14 am »

Next build we'll try this little tweak:
Changed: Media Center downloads _all_ data files on library server, but shows a progress dialog while it works so that you can cancel if you don't want that.
Logged
Matt Ashland, JRiver Media Center

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #43 on: May 31, 2016, 07:09:57 pm »

Awesome, looking forward to testing.
Logged

glynor

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 19608
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #44 on: May 31, 2016, 10:30:52 pm »

Of course, keeping it for small files is fine.
I feel like all downloads should show a cancelable progress Dialog though, hanging MC while it downloads is never good.

Agreed.

Next build we'll try this little tweak:
Changed: Media Center downloads _all_ data files on library server, but shows a progress dialog while it works so that you can cancel if you don't want that.

I'd actually like the same thing as your original idea for other file types, when streaming isn't working adequately. Video, for example.

I've learned, the hard way, these weeks, that streaming from my DSL connection just isn't cutting it (even for MP3 audio). I know you can't design for quite this low of a performance level. I'm not asking for that. But I've seen "frozen" MC behavior many times before I moved when I had "decent" (by US standard) upstream bandwidth. When I tried to play a video, or a particularly large image file, for example. Sometimes MC would freeze for a 30 seconds, or a minute or more, while it got enough data to display something.

No way to stop it, other than wait, or kill the process.

I think the most sensible solution is something like your original idea, Matt:

Another approach would be to set the file loader to show a progress UI after 1 second with a cancel button.

Then a user would see what's happening and be offered the choice of cancel.

[Media Type] doesn't matter.

One second? Two? Not sure, but I think this basically gets down to it. If MC is going to "feel frozen" to the user (latency high enough that it is noticeable and unexpected), then it should always go to a progress dialog with a cancel button. At least in any operation that you can guess might take a long time, particularly over a WAN connection.

In an idea world, it would happen in the Action Window and you could do other things with your MC in the meanwhile. But that might be asking too much and a modal dialog is acceptable.  ;D

You don't want to show it when things are going to stream quickly enough that the user wouldn't be bothered. But if MC is going to otherwise look stuck, for any file type, it should react, not freeze. Where exactly you draw that line? 1 second, 2, 500ms? Not sure. You'll probably have to play with it.
Logged
"Some cultures are defined by their relationship to cheese."

Visit me on the Interweb Thingie: http://glynor.com/

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5177
  • "Linux Merit Badge" Recipient
Re: [Feature Request] Improved Support for Files with Media Type "Data"
« Reply #45 on: June 01, 2016, 08:03:33 am »

Agreed.

I'd actually like the same thing as your original idea for other file types, when streaming isn't working adequately. Video, for example.

I've learned, the hard way, these weeks, that streaming from my DSL connection just isn't cutting it (even for MP3 audio). I know you can't design for quite this low of a performance level. I'm not asking for that. But I've seen "frozen" MC behavior many times before I moved when I had "decent" (by US standard) upstream bandwidth. When I tried to play a video, or a particularly large image file, for example. Sometimes MC would freeze for a 30 seconds, or a minute or more, while it got enough data to display something.

Large images are a good example of similar behavior because they cannot be streamed, and I've definitely seen "freezes" while waiting for big images (mentioned above).  The difficulty with extending the logic to Audio or Video is that they are streamable, so how does the program decide when to give up streaming and start pre-downloading?  Or do you just mean that some kind of progress bar should be shown if things are going to take a second no matter what?
Logged
Pages: [1]   Go Up